I think what's happening is that the unit you're trying to add magazines to doesn't have enough inventory space.
This might have a couple errors because I can't test it right now, but try this:
EDIT: This worked in the editor for an AAF sniper
this removeMagazines "5Rnd_127x108_Mag";
this addMagazines ["5Rnd_127x108_APDS_Mag", 1]
You might want to increase the magazines added like maquez said though. Also, keep in mind this will remove all regular 12.7mm ammo from the unit.
Let me know if you need any more help.
Good luck.