How to increase dedicated server FPS

[Preacher1974 0]

I have a Core2Quad 2.83GHZ server with 4GB of RAM running Windows 2008 x64 on a 100Mb internet connection. I can't seem to get more than about 25fps on the server for missions (even with only 1 player connected). Looking at the Windows performance monitor, I can see all 4 cores are utilised, but no core seems to go over 50-60%.

So given that there doesn't seem to be a CPU bottleneck, why can't I get the server's maximum framerate of 50fps (48/48fps in reality)? I've tried every arma2.cfg network setting known to man, but it doesn't make a difference.

Here is the screenshot of a mission with one player not doing anything, and getting 23fps.

Edited October 31, 2009 by [ZSU]Preacher

[.kju 3239]

No need to get more FPS.

Only below 10 FPS you get slight more warping etc.

Only below 5 FPS you get real troubles and mission no longer working correctly.

[Rommel 2]

Even on an blank map with no one connected?

[Taxman 10]

Post your arma2.cfg buddy... some of the guys here may be able to help with that.

[Game__On 10]

No need to get more FPS.

Only below 10 FPS you get slight more warping etc.

Only below 5 FPS you get real troubles and mission no longer working correctly.

You need the server to be above 25 fps. Maybe not when all you use is the dmr , but with machineguns your rate of fire drops dramatically, and your groups (when you fire a burst of say 5 or 10 bullets) scatter all over the place, meaning you can't hit sh^t at larger distances (300 to 500 meters)

Edited October 31, 2009 by Game__On

[cross 1]

Many missions have scripts in them running when you start the mission and most of them are buggy/unoptimized so it is common to have 20-25 FPS when you load a mission.

Make a mission yourself with no AI/scripts in it and test with that.

http://cross.kellys-heroes.eu/public/mpsimple.utes.pbo

[Proton 11]

I am also trying to improve the dedicated server's performance with a complex MP mission. Just right now the mission runs with 554 units and 124 vehicles, usually hovering around 10 FPS. A bit higher would be desirable. The question is what is the most important influencing factor.

I can view the CPU usage of threads in ProcessExplorer. According to that, the server runs 12 threads, of which 4 is taking measurable CPU time. One is fully using a core (25% on my quad machine), and 3 others using half of a core together (7%, 4% and 1%). Probably the one consuming 25% is the bottleneck.

Can a developer please enlighten us that what actually the threads do? Especially the 25% one. I would split the server into 4 main threads: one for general game logic, one for scripting, one for network and one for AI pathfinding. Is this guess true?

The reason I am asking this is to decide how to try optimize performance. If the scripts actually take a separate thread and they are not executed in the 25% one, it makes no sense to reduce the amount of running scripts, as the advantage is going to be zero. The mission uses about the same number of scripts at start and has 40 FPS, so probably not the *amount* of scripts which matters.

If the number of AI units is the most important factor, does it help if I do not move them when not necessary (to reduce pathfinding)?

[rexehuk 16]

Post your bandwidth settings here.

[Proton 11]

language="English";

adapter=-1;

3D_Performance=1;

Resolution_W=800;

Resolution_H=600;

Resolution_Bpp=32;

Windowed=0;

MinBandwidth=800000;

MaxBandwidth=25000000;

MaxMsgSend=256;

MaxSizeGuaranteed=512;

MaxSizeNonguaranteed=256;

MinErrorToSend=0.003;

MaxCustomFileSize=100000;

Although I am testing the mission with just a single client at the moment.

[terox 316]

what we really need is a server benchmark mission.

Heavy AI useage with some heavy cycling function calls

Bunch of continously firing mg guns

large groups of ai looping waypoints, continously changing formation

made say for 5 players

with benchmark times every say 3 minutes and mission running for 10

Outputting to the RPT

Mission has no player interference, just to be there so that it creates network traffic and therefore every mission replay isd the same, so possibly no enemy to interact with

A mission which basically stresses all elements of the .cfg settings using a set amount of players that a typical small community can gather for optimisation testing

With this we may at last be able to put together a guide for server settings. To date there hasnt really been such a thing and I have found the BIS WIKI entries to be somewhat misleading and most server's .cfg's seem to be nothing like one n other which just creates more confusion. Ideally this is a task for BIS, but that aint gonna happen anytime soon

---------- Post added at 04:01 PM ---------- Previous post was at 03:56 PM ----------

language="English";

adapter=-1;

3D_Performance=1;

Resolution_W=800;

Resolution_H=600;

Resolution_Bpp=32;

Windowed=0;

MinBandwidth=800000;

MaxBandwidth=25000000;

MaxMsgSend=256;

MaxSizeGuaranteed=512;

MaxSizeNonguaranteed=256;

MinErrorToSend=0.003;

MaxCustomFileSize=100000;

Although I am testing the mission with just a single client at the moment.

Thats bloody unusual, they are exactly the same settings we use and I mean exactly. I dont think its as optimised as it could be though

Edited July 23, 2010 by Terox

[galzohar 31]

The bottleneck (thread that runs 25% aka 1 core) is clearly the AI thread. You can easily see it if you test different amounts of AI. Once the AI thread starts running 100% of the time, the more AI you add you will actually see less CPU usage, as AI thread continues to run 100% of the time but FPS goes down meaning other threads run a lower % of the time. This doesn't just happen on servers, happens on hosted servers and single player as well.

The only real solution is to get a faster CPU (not more cores, but simply a faster one) or wait for the Messiah to arrive and re-program Arma 2's AI so that it uses more than 1 thread (or at least makes it less heavy on the CPU).

If you're getting low FPS in a mission with absolutely no AI then I really don't know what to say.

Scripts, assuming they aren't running through dirty hacks like using eventhandlers and the likes to get extra runtime, are actually not allowed to run long enough to affect FPS. Instead the game engine just makes the scripts run less. This can be easily tested by spawning a lot of scripts that run infinite loops that do nothing and see how your FPS remains the same (though other scripts that are running may become slow/unresponsive).

[Proton 11]

A mission which basically stresses all elements of the .cfg settings

I doubt that the problem is with the .cfg settings as I barely use networking now. What kind of data output you get via #monitor? I have around 1800 Kbps when the mission runs in its full glory.

Thats bloody unusual, they are exactly the same settings we use and I mean exactly. I dont think its as optimised as it could be though

Probably I took them from the same place... :)

---------- Post added at 06:40 PM ---------- Previous post was at 06:22 PM ----------

The bottleneck (thread that runs 25% aka 1 core) is clearly the AI thread. You can easily see it if you test different amounts of AI.

If its true, it does that in a strange way. I just tried respawning one side's entire army, destroying all vehicles and units when the mission was already running for a long time. It means removing hundreds of units and vehicles at once. It had no hit on the server FPS. So what is going on here?

The only real solution is to get a faster CPU (not more cores, but simply a faster one) or wait for the Messiah to arrive and re-program Arma 2's AI so that it uses more than 1 thread (or at least makes it less heavy on the CPU).

This is indeed the best hardware option. The devs said something that the next betas are going to speed up AI pathfinding... fingers crossed.

If you're getting low FPS in a mission with absolutely no AI then I really don't know what to say.

I get a decent server FPS with no AI. My question is more about how to improve FPS with lots of units. For example, if I change group waypoints less frequently, it may have an impact on AI pathfinding. Does it matter? From the experiment above, it does not. So only the number of AI units counts?

Scripts, assuming they aren't running through dirty hacks like using eventhandlers and the likes to get extra runtime, are actually not allowed to run long enough to affect FPS. Instead the game engine just makes the scripts run less. This can be easily tested by spawning a lot of scripts that run infinite loops that do nothing and see how your FPS remains the same (though other scripts that are running may become slow/unresponsive).

Thanks for sharing this insight. I suspected something alike, so until scripts do finish in a reasonable time, I should not worry about them.

[galzohar 31]

The only reason to be worried about scripts is that if a script takes up too much of the "script runtime", other scripts might lag. But FPS and network communication will remain untouched. Of course some scripting commands can cause FPS drops or network traffic, like a script that spams creating vehicles or publicVariable, but "normally", scripts won't do that.

It seems that any kind of moving AI is causing pain, especially in urban areas. I'm not sure but it seems that even static AI in urban areas can cause some pain. It also gets much worse if they detect an enemy inside that urban area, FPS can go down to 1/2 of its original value then (though this might be because UPSMON then sends other groups to attack you, but then again those groups should've already been moving).

Another thing I noticed at one of my tests is that when I was using ungrouped AI, I could place more AI on the map while getting the same server FPS (as in, 75 AI each in his own group seemed to have given more FPS than having 75 AI grouped in various group sizes). This was a very specific test, though, and might not always be true, but is another thing you can try.

All in all I really do hope they improve AI pathfinding. There are a lot of mission ideas I'm not implementing simply because of that limitation.

[nobrainer 0]

This is strange isn't it.

We got a i7 960 with 12GB memory, win7 64 and we run most missions at 50FPS.

All 4 cores active. Turned of HyperThreading in the BIOS.

The network traffic is between 150k and 8500k from the server.

[Hud Dorph 22]

I had the impression that only 2 cores would be used by a server. My server have nerver used more than 2 of the four cores, no matter what. Affinity set to four cores it will use 1 core fully and 1 core aprx 30 to 50%. Setting afinity to 2 cores seems to spread the load more evenly over the 2 cores. Never seen it use all 4 cores.

[nobrainer 0]

used all 4 cores with -exThreads=7

Don't know why tho...

[Proton 11]

Of course some scripting commands can cause FPS drops or network traffic, like a script that spams creating vehicles or publicVariable, but "normally", scripts won't do that.

Even if the FPS is dependent on mostly AI pathfinding, in the end, the scripts *do* influence FPS by making the choices to do a lot of pathfinding. So it is worthy to know what is actually making the server running slow. Your observations about AI in urban areas is important, however I do not understand why a standing unit in a town is consuming more time.

---------- Post added at 02:33 AM ---------- Previous post was at 02:32 AM ----------

used all 4 cores with -exThreads=7

Don't know why tho...

I believe -exThreads=7 is the automatic choice with 4 cores. And the server actually runs 4 active threads, so it is obviously distributed amongst the cores. But do they all run on 100%? I doubt it. I also doubt that you can run Benny's Warfare with 50 FPS, especially with hard resistance/occupation defenses.

Edited July 25, 2010 by Proton

[sbsmac 0]

however I do not understand why a standing unit in a town is consuming more time.

IIRC, the AI code does include some aspect of detecting possible directions of threat and enemy lines of fire. In addition, AI clearly have 'lines of sight'. In an urban area, that's more complex because you need to figure out how all those buildings occlude you.

Whether those things are active during 'safe' mode I don't know - you could try changing the mode of the soldier from safe to danger and back again to test.

[Proton 11]

Here is the result of a small study I made over the weekend.

I use a modded Warfare mission on Chernarus with the latest beta (72197) of OA, where each town immediately spawns all defending teams. The mission was started by a client, who immediately disconnected after start (server is persistent DS). I measured the performance in the first couple of minutes after that.

Considered parameters:

1. Hardware/OS. Two pieces were used, one running Q6600/XP64 Pro, the other one is i7 920/Win7 64-bit.
   
2. Number of groups/units created. Each group consists of 6-12 units. I tried 30 and 120 groups.
   
3. The effect of AI. After creating the groups, one version let the default AI run and created random patrol waypoints for each group which kept them moving. I also set the group behaviour to "COMBAT", but this was almost immediately set back by engine to AWARE for most groups.
   The other, "AI disabled" version immediately sets the group behaviour to SAFE, and issues
   

   { doStop _x;
   _x disableAI "TARGET";
   _x disableAI "AUTOTARGET";
   _x disableAI "MOVE";
   _x disableAI "ANIM";
   } foreach (units _team);

   
   

4. Location. One version used a limited distance from town centres which kept all units close to or within town borders. The other one let the groups randomly spawn over the entire map.
   

This is 2^4=16 parameter combinations, of which 12 were actually tested. The benchmark consists of 4 parameters:

1. Real execution time vs. engine time. This measures whether all the scripts could run in their devoted time. Measures were scheduled to run in each 30 seconds via the 'time' function. If the measures executed at longer intervals in real time, it means that the engine was unable to run all scheduled scripts as requested, and the mission will inevitably slow down to a crawl (until it can reduce the data input to match its processing capability), or crash at some point. I waited for at least three measurements.
   
2. server FPS via diag_fps,
   
3. minimal server FPS via diag_fpsmin,
   
4. the running time of a small benchmark script:
   

   for "_i" from 1 to 1000 do { [_i] call {_this} };

   
   

The results are usually getting worse with later measurements as the player teams (controlled by AI here) start buying units and vehicles for themselves.

Results:

Conclusions:

• It is definitely worthy to own the latest hardware for a DS. The i7/Win7 combo seems to handle scripts much better, even when FPS is low - or it just has enough power to finish in the scripting time slice, unlike the Q6600.
  
• The single most important factor is the number of units in play. As a rule of thumb, 4x more units make the server run 4x slower, even when the AI is not doing anything!
  
• AI seems to have a significant effect, somewhere between 5-10 FPS. However, (at least per-unit) AI calculations seems to come only second to the impact of the actual existence of units.
  
• It is not true that towns slow down AI calculations, at least random starts looks like being harder to crunch.
  

Two questions remain: why in hell does it matter that a non-moving, non-controlled unit is created? It should be treated like a tree or a house. Why does it push the FPS down almost linearly?

The other thing I do not really understand yet that why my server goes down to 10 FPS by time, as it rarely have more than 30 of these groups spawned at the same time. Temporal analysis follows.

Edited July 26, 2010 by Proton

[=wfl= sgt bilko 10]

Purely guessing here, but for the first question:

- Non-moving unit is not same as a placable object like a house or tree. AI's still carries a lot of logic to loop through even if the response to the logic is filtered to do nothing.

Edited July 26, 2010 by =WFL= Sgt Bilko
spell check

[Cato Larsen 10]

You can take a look at these settings. This is what I use on our server. It goes for ArmA 2 and works for ArmA OA aswell.

Play a bit with min_bandwith and and the MSG stuff.

The info on Kellys is very very good, so you should be good to go.

[rexehuk 16]

Hmm, one thing I do notice here is everyone seems to have their bandwidth configs in their PROFILE CFG, this is not right as far as I'm aware?

I have always ran it from the server.cfg, not the actual game profile itself. I'm trialing various bandwidths on our server at the moment

maxPlayers=128;

maxCustomFileSize=102400;

Kickduplicate=1;

verifySignatures=0;

equalModRequired=0;

BattlEye=1;

MinBandwidth = 524288;

MaxBandwidth = 10000000000;

MaxMsgSend = 512;

MaxSizeGuaranteed = 1024;

MaxSizeNonguaranteed = 128;

MinErrorToSend = 0.002;

That is our config for the moment, still messing around with it trying to find the sweet spot. I really do think you guys are better off running it from the server.cfgs, it allows you to trial a variety of different configs over many servers.

Edited July 26, 2010 by rexehuk

[rekrul 7]

Great work Proton!

However I do believe the problem with a static AI is that the game schedules threads/process time slots/whatever for every AI (or at least AI groups) so the AI has guaranteed timeslots with the processor(s) (QoS so to speak).

[galzohar 31]

I don't think small towns make a difference, as I only noticed additional slowdown due to urban areas in 2 cases: 1. The little city I made on the airfield in Chernarus with 150 buildings and 2. Zargabad.

[Taxman 10]

Hmm, one thing I do notice here is everyone seems to have their bandwidth configs in their PROFILE CFG, this is not right as far as I'm aware?.

As far as I know buddy thats the correct way, using the profile.cfg, (didnt think the Server.cfg would even effect it)

That is our config for the moment, still messing around with it trying to find the sweet spot. I really do think you guys are better off running it from the server.cfgs, it allows you to trial a variety of different configs over many servers.

you can run diff configs using different profile cfg's using the -profile=SERVER1 or -profile=server2 etc etc .. parameter (server1 and 2 can be anyhting you wish.