What I found about performance and I what I want to help fix it

[brightcandle 114]

My machine specs:

3930k @ 4.3 Ghz

16 GB DXDR3 2133

2x GTX 680's

m4 SSD

Benq XL2411T 144 hz monitor

The basic performance metrics I know about this system:

Altis airport (at our starting location) - 80-90 fps in all directions GPU usage around 75%.

Pythos town NE corner - 70-80 fps - GPU usage 60%

Pythos town with 40 AI in the town - 45-60 fps - GPU usage 55%

Multiplayer with 35 players and 70 AI before the fight - 50 fps - GPU usage 50%

Multiplayer with 35 players, 70AI most of them dead - 25fps - GPU usage 30%

Multiplayer with 35 players, 70 AI most dead and running a deletion script to remove them - 35-40 fps - GPU usage 40%

Singleplayer campaign first mission at start - 25-30 fps - GPU usage around 30-35%

Showcase infantry at start - 70 fps - GPU usage 75%

Bare in mind during our multiplayer sessions our server FPS runs between 40 and 48 fps, we don't believe server FPS is an issue with so few players and AI. Our tested limit for our server to maintain 40 is 150 AI and 50 players.

So I am pretty confident that performance is entirely CPU dominated on the client, at least once you have sufficient GPU performance. It wouldn't make any difference if I went to 3x 780 ti graphics cards, my frame rate would be identical by the looks of things. I see no difference running the game on low settings for example, just lower GPU usage (as measured by GPU-Z). The big problem is the multiplayer performance which I really needed to get to well above 30, ideally more like 60 fps. So initially what I did was create a multiplayer performance mission with 50 v 50 AI battle in a valley and watched from a vantage point. I tested it with our mods, no mods or map scripts +mods and with our full setup including MCC spawned units. I wanted to make sure our modset wasn't to blame. What I determined was less than a 2% difference between the runs, so the scripts+mods didn't seem to be the cause. FPS did drop once the firefight started and map modifications did elongate the performance drop but it also elongated the firefight as well by reducing the AI accuracy but the drop in performance wasn't any worse than the others. In all cases FPS never fully returned after the firefight once they were all dead and stopped moving, but it did recover about 50% of the performance. I saw 80fps before the firefight, dropped progressively to a minimum of 45 during the fight and then recovered to 60 fps.

As a test showed the problem to some extent but it never got down to 25 FPS despite all that was missing was 30 odd other players. I tried changing the test in various ways and I couldn't reproduce the 25 FPS, the results didn't really change too much with more or less AI or vehicles or changing the location to a town. This suggested to me maybe it was due to the players and not the AI or the mission. However the performance test did show that dead bodies consumed a lot of performance, so we tried deletion dead bodies mid mission and it certainly helped a bit (as you can see above) but its only about 60% of the drop that happens after a firefight is finished. Whatever is dropping performance through missions is mostly dead bodies, and our best guess is probably PhysX (since it doesn't seem to run on the GPU at all) but there is still a third that never comes back.

So create a firefight with 100 AI or so, they all die and then I delete the corpses and the live ones remaining. Even if the firefight and the AI caused a drop in the FPS we would expect once they were removed the FPS would return to what it was before they even existed. There should be nothing consuming CPU or GPU performance left from the firefight and yet that isn't the case. You will get about 60% of the FPS drop back but it will never return. If you now do another firefight of a similar size it will drop even further than it did the first time and deleting it all wont return to the previous levels either. This leaves me with one of two conclusions - there is something in the engine that over time is just dropping performance after stuff happens (because just sitting on the map doesn't show constant drops), or something else is being left behind after a firefight that has quite a dramatic impact on performance.

The conclusion is I can't reproduce it outside of our multiplayer game. I can achieve similar FPS in the singleplayer campaign but the cause there is possibly different. One thing BIS could do to improve performance is focus on that single player campaign initial mission with an excess of GPU performance and you can reproduce reliably a low FPS on your own for profiling and fix it. Fixing that will almost certainly translate into real FPS in multiplayer as well, its the only scenario with a single player that has similar FPS that I have found but I don't know if its related to the multiplayer performance issues.

For Multiplayer what I think we need is some performance stats logged out (via a switch or whatever) telling us where the time is going on the CPU. Then we can provide results back to the devs to further refine the logging in the area where the problem manifests and over a couple of iterations we should get down to what is causing the problem and the devs know what section of code is going slow and produce reliable tests that show the problem. The alternative is if a dev with a client and a profiler would like to join our weekly game and capture some data and see it for themselves we can organise that (Sunday 20:15 CET). Unfortunately our communities conclusions is that there is no hardware you can buy today that can fix it, client FPS is limited to below the playable threshold on the fastest most expensive CPUs today when overclocked to their maximum. In order for the game to run reasonable on more moderate hardware the CPU time per frame needs to halve so that a more moderate 2.5 Ghz quad core can achieve 30 fps (they get 15 fps today).

Tell us how we can help you find the cause. We have quite a few developers in our community so we know how to use profilers and such, just let us know how we can help and fix this.

[giorgygr 61]

Respect!

I really hope this will draw the attention of developers.

[dezkit 28]

All I ask for is to play full multiplayer servers with at least 30 fps. I can't stand playing 10-20 fps.

Bohemia, if you fix your optimization, your sales will go through the roof. Pls.

[nomisum 129]

interesting. indeed MP performance is so much worse than SP. hope the devs find the glitches.

[ratszo 17]

Netcode? Makes sense if your 'puter is waiting for packets, performance will suffer.

Syncing issues come into play as in 'who sees what, when, where.

Interested how dayz sa will optimize netcode for 120 slots & gobs of ai.

[roshnak 41]

All I ask for is to play full multiplayer servers with at least 30 fps. I can't stand playing 10-20 fps.

Bohemia, if you fix your optimization, your sales will go through the roof. Pls.

30 FPS is not an acceptable framerate. 60 is the standard that that most developers generally shoot for.

[giorgygr 61]

Is it possible we'll see a developers comment please?

[all3n 1]

I support this 100% give us a way to help you find the problem, let us log what is going on and report it to you BI

[maturin 12]

30 FPS is not an acceptable framerate. 60 is the standard that that most developers generally shoot for.

There's no such thing as a standard. Everyon has a different computer.

No game as large and detailed as ArmA is ever going to run at 60 reliably. Not even BF3 did on consoles, ffs, and BF3 is like ArmA in miniature, after a massive, wealthy dev team has tested and optimized it out the ass.

[all3n 1]

Let's try to keep this on topic, seeing as it's the only topic I've seen in a while that doesn't just say "the code is shit" "fix fps".

The proposal of enabling the users to log and debug the running scripts themselves would greatly speed up and benefit The process of coming to an overall more stable platform

[best2nd 10]

So create a firefight with 100 AI or so, they all die and then I delete the corpses and the live ones remaining. Even if the firefight and the AI caused a drop in the FPS we would expect once they were removed the FPS would return to what it was before they even existed. There should be nothing consuming CPU or GPU performance left from the firefight and yet that isn't the case.

Do the repo again but this time delete the groups. Groups will continue to exist even if nobody is in them possibly causing a performance hog.

[dr. hladik 231]

rev 113205

http://community.bistudio.com/wiki/Performance_Profiling - was enabled for dev branch. DS writes results to RPT.

> Pythos town with 40 AI in the town - 45-60 fps - GPU usage 55%

> Multiplayer with 35 players and 70 AI before the fight - 50 fps - GPU usage 50%

> Multiplayer with 35 players, 70AI most of them dead - 25fps - GPU usage 30%

Could you please send me profiler results? (Copy all button will export it clipboard)

[maturin 12]

This is an old problem, by the way. Even in ArmA 1.0, dead AI would take as much resources as alive ones (in combat). IIRC, the same goes for SP.

[nicolasroger 11]

Even if the firefight and the AI caused a drop in the FPS we would expect once they were removed the FPS would return to what it was before they even existed. There should be nothing consuming CPU or GPU performance left from the firefight and yet that isn't the case. You will get about 60% of the FPS drop back but it will never return.

IMHO the core problem is right there. It would explain why we always have a nice FPS when we start the multiplayer server, and then overtime it gets worst for everyone (even though you can't see a dead body anywhere).

Most constructive post I've seen related to performance in a while!

[Sealife 22]

Same tests on a terrain other than Altis or Stratis would be very much appreicated also , at this moment in time all things point to Altis and its number of objects and cell size negating any progress with ancillary fixes on the face of it .

This would benefit those of us who wish to build arma3 terrain but not quite sure if the price for smaller cell size and thus subsequent data sets appplied to thos cells is a price worth paying .

There are limited terrains to try but on Armaholic they have a few in a&m complete for A3 if you get a chance.

cheers for your input so far its great to see some numbers put to the problem :)

[iceman77 18]

And would simply deleting the bodies help? :p

Edited December 4, 2013 by Iceman77

[Tyl3r99 41]

this sounds very promising... very interested here

it sounds like when the AI dies its like the game engine does not recognise they're actually dead??? or am i confused again?

[daimyo21 2]

First off to the OP, I'd like to thank you and your community for taking the time to test this out for the devs. I think its been one of the most plaguing things about Arma 3's multiplayer experience and I'm sure even mission scripters take a breath of relief reading this.

I am completely aware of server cycles correlated with client FPS drops from Arma 2 and thought the same was happening for Arma 3. I would go on domination servers with optimization tricks and tips and just pull my hair out. However, one message in side chat of "Whats everyones FPS?" and the replies of 15-25 FPS would let me know that this seemed like a server-side thing with two much AI bogging it down like in Arma 2, however, I began noticing this on almost every server, even ones with little to no AI like King of the hill mod and PvP servers alike.

This post clarifies many things and I am glad I stumbled upon it as Arma 3's performance I'm sure has been demoralizing for multiplayer mission scripters who came in from Arma 2 with higher ambitions (I know I did).

Here's to hoping BIS squashes these problems sometime soon!

[dr. hladik 231]

this sounds very promising... very interested here

it sounds like when the AI dies its like the game engine does not recognise they're actually dead??? or am i confused again?

It does.

[Greenfist 1863]

It does.

As in "engine does in fact recognise the dead" or "it does sound promising" ?

[nomisum 129]

i think it means it does recognise the dead as dead, otherwise they would keep on fighting :D

[TSAndrey 1]

As in "engine does in fact recognise the dead" or "it does sound promising" ?

It does recognize they are dead. It's just that dead soldiers still need a lot of resources, because they have gear that can be picked up and etc.

[Greenfist 1863]

Of course the game knows they're dead, I just wondered if Dr. Hladik was referring to some newly discovered issue of not clearing all the unnecessary data of the dead. Especially if the OP is right and removing the dead does not recover performance.

Edited December 5, 2013 by Greenfist

[mt 10]

Apparently there are still some left-overs from the AI fight, even though all bodies are removed (and there are 70AI less to process).

In my imagination, it shouldn't be that hard to find the place where the brains for those missing 25fps are spent here.

I'd love to see some profiler results of that (in this thread).

[iceman77 18]

I believe he's referring to both lol.