To my knowledge this is very much possible. The reason is that Arma 3 simulation is not deterministic. Each decision that units do have certain amount of randomness. Therefore if one wants to do as comparable benchmark as possible, you cannot run simulation on any unit as the course of simulation is different on different machines, even on the same machine based on the time you run it. With some planning you can get reasonably deterministic behaviour even with units but really cannot guarantee it.
Just open editor (or script the whole thing if you will) and add bunch of units from two factions close to each other and you'll see how they battle each other. However, they way how they battle and the outcome too may wary on each run.
I'd guess that most accurate benchmark would not involve units doing their own decisions, or for each unit call "enableSimulation false;", that makes unit static but CPU and GPU work as identical as possible.
