What I found about performance and I what I want to help fix it

[Drakenof 11]

I can live with lower visibility settings but we do need a real game (15 players is not a big MP mission) to have playable frame rates.

Truth. And that's why i don't play ArmA 3 anymore.

[arotished 10]

I understood everything he wrote, no need to turn this into a grammar contest :)

Thank you.

[dnk 13]

If you turn up the visibility so you can see the whole map then chances are, based on the traces I took, the game is struggling to make enough draw calls and is CPU limited in its main thread resulting in the FPS drop. However if you don't turn up the visibility and instead run it at something reasonable like 2000 and play a real multiplayer game what you find is FPS still drops to 25 fps, and it seems to do so because the simulation of the game world itself is taking a huge amount of time, not particularly because of draw calls. On that main thread are calls out to Input, PhysX, AI, Sound, visualUA, rendering and the game world. With nothing in the world the rendering part takes at least 10ms from anywhere on Altis, more often its 16ms and my traces show it up to 22ms or more. So its very hard for the game to exceed 60 fps just due to the render calls on the main thread. But when you start putting AI and such into the game then game world updates take over as the dominate problem.

The issue is its not one thing growing, its basically everything as far as I can see but with the game world updates being the biggest climber throughout the mission. It keeps getting worse despite the AI dying off, the last trace is just us surrounded by dead bodies in a town, with depleted players and everything else. What is taking the simulation so long in those circumstances? Because its got considerably less to do than at mission start which had AI moving around, players moving around etc etc. At the top level lots of the game sim elements grow between my traces, the growth is in basically all the top elements of game world updates, but the names are too cryptic to work out what they mean and do. And because so much shares the main thread the offloaded AI, the doubling of the time processing sound and extra render calls all push it into unplayable FPS. Its not as simple as to say its geometry, so while its true if you have the visibility so high the problem I am looking into is wider than that. I can live with lower visibility settings but we do need a real game (15 players is not a big MP mission) to have playable frame rates.

Excellent response. Sums it up as perfectly as I could.

The CPU is limited in its usage (60-70% on core 0 usually, even on a 3.3GHz) likely due to having to wait around regularly for new data as it calculates all the geometry or AI routines and such. Even with 0 latency RAM and infinite bandwidth, there are laws of physics and clearly the engine isn't able to cache everything it needs in a frame to keep the usage near 100%. That's my take on it, but I'm not 100% sure either.

note that VD, object quality, and terrain quality will all heavily affect your geometry-based CPU bottlenecking. Especially with higher object quality, where LOD switch distances are extended much further and the %age of objects drawn is much higher at each LOD level, the geometry on screen can increase exponentially with increases in VD (since we're squaring the distance). Note that 12000VD is ridiculous, and setting it under 3500 is advisable unless you're not having FPS issues.

Edited December 14, 2013 by DNK

[kremator 1064]

There has already been a huge improvement in using the new dedi standalone server. Perhaps you guys need to try that?

[brightcandle 114]

Did a similar assault with 14 people to towns to the NE of altis, this time using the new performance binary. Frame rate in the middle of the game dropped down to 30 fps. Thus the performance binary makes no difference to the performance of the game. Our problem is not server side and the recent work there hasn't helped.

Profiles from previous page are still the best indication of the performance we are seeing. Dr. Hladik do let me know if you need any other tests doing, otherwise hoping to hear from you soon on what is going wrong to drag the game FPS consistently low.

[brightcandle 114]

I have made a short program that can summarise the profiler output and sort the main contributors to performance. The beginning of the mission the top 20 elements on the main thread are as follows:

total times=1 totalTime=16.42797
wSimu times=1 totalTime=16.35929
rendr times=1 totalTime=11.55092
hudDr times=1 totalTime=5.0409
oPrep times=1 totalTime=1.67009
wDraw times=1 totalTime=1.50756
wPrep times=1 totalTime=1.50044
wSimA times=1 totalTime=1.46107
recCB times=16 totalTime=1.36057
wsSet times=1 totalTime=1.33
preLd times=1 totalTime=1.24929
pdDrw times=2 totalTime=1.02054
sceAC times=1 totalTime=0.94528
o1Drw times=6 totalTime=0.8402000000000001
wSimR times=1 totalTime=0.77905
sceACPrx times=1 totalTime=0.77576
wSimSV times=1 totalTime=0.72938
xim times=41 totalTime=0.7221199999999993
ximO times=8 totalTime=0.71714
simSo times=5 totalTime=0.66273

The rest of that summary can be found here: http://pastebin.com/W1iQgMvL

By the end of the mission that top 20 looks like the following:

total times=1 totalTime=35.5624
wSimu times=1 totalTime=35.48974
rendr times=1 totalTime=21.06408
hudDr times=1 totalTime=11.28658
wSimA times=1 totalTime=5.84862
wSimSV times=4 totalTime=4.25627
xim times=168 totalTime=3.7769000000000026
ximO times=111 totalTime=3.750509999999998
wSimO times=1 totalTime=3.0589
sound times=1 totalTime=2.77545
simSo times=30 totalTime=2.5965800000000008
wSimM times=30 totalTime=2.5632399999999995
wDraw times=1 totalTime=2.50926
preLd times=1 totalTime=2.50432
wPSnd times=1 totalTime=2.48584
oPrep times=1 totalTime=2.2682
wPrep times=1 totalTime=2.12109
recCB times=15 totalTime=2.07396
preLV times=1 totalTime=1.57556
o1Drw times=5 totalTime=1.5229300000000001

The rest of this summary can be found here: http://pastebin.com/zG11qptS

The total more than doubles,the first trace is ~ 60 fps and by the end of the mission we are at 28 fps. Most elements increase from 1.5x - 2x across the board. Notable call outs however in what changes more dramatically:

wSimA grows enormously

wSimSV grows enormously

xim grows enormously

ximO grows enormously

These leave the top 20 in the second trace

wsSet drops off

pdDrw drops off

sceAC drops out

wSimR drops off

sceACPrx drops off

simSo drops off

And these all grew to take the place of those that dropped out:

wSimO 3.0589

sound 2.77545

wSimM 2.5632399999999995

wDraw 2.50926

wPSnd 2.48584

oPrep 2.2682

preLV 1.57556

What strikes me about this summary view of the best and worst traces is that there are clear big gainers responsible for most of the time but its also across a lot of aspects that performance is dropping, from sound, rendering and the simulation itself. Mostly its about 1.5x but there are some things that grew more than 5x as the mission went on.

I hope this summary picture where I have summed up all the things with the same name and counted the entries by thread helps you understand what is going on, I personally found it more enlightening than the raw output. If you want access to the code just contact me directly (Dwarden knows how to reach me via Skype for example) and I can provide you details.