Server Performance with High Player Count (80players)

[VoRtReX-uk 10]

Hello,

Just wanting to query a few things about the arma2oaserver.exe.

The server spec is:

Windows Server 2008

Intel I7 930 @2.8ghz (HT enabled)

8GB Ram

750 GB raid 0 HDD

Using start-up bat file for auto restart on crash:

TITLE XXX

SET port=2302
SET mods=;

SET rank_log=_ranking.log

SET cfg=_performance.cfg
SET config=_server.cfg
SET profiles=_profiles
SET WORKING_DIRECTORY=%cd% 

:Loop
cd "%cd%"
start /wait /HIGH arma2oaserver.exe -malloc=system -cpuCount=4 -exThreads=7 -port=%port% -mod=%mods% -cfg=%cfg% -config=%config% -profiles=%profiles%
GOTO Loop

We have had an issue on our server recently where we had around 80 players, all playing. The issue happened after about 1 hour 20 minutes. The best way to describe it from a player’s point of view is desync:

I was in a jeep as a passenger when we crashed through a fence. Using TS3 to speak to the driver I found that on his screen he was still driving down the road as normal. A few seconds later my screen froze with the engine sounds still running, I was able to look around (my game didn’t crash). I would say half of the people were still playing as normal and DID NOT have any issue.

After talking for a few minutes I disconnected from the game server, at which point I could not see the game server in question in the browser (after many refreshes).

I then logged into the server using RDP, the arma2oaserver.exe was running as normal with around 20% CPU & 1.1GB Ram being used, information was being updated in the game server console window. Yet it took 5 minutes for the server to re-appear in the game server browser, and when it did it took ages for people to connect.

Any idea what could have caused this? It’s the second time out of 9 games something like this has happened with around 80 players. The game is a private “war†so having this issue means we have to re-play the “War†again.

1. The arma2oaserver.exe only appears to use around 25-30% of the CPU capacity of the system (with 80 people on the server). Seems a bit strange that the game server should struggle with low fps in game when there is so much system resources left to use. Is this normal, and if so is this going to be fixed?

2. Once a map has finished and all players have left the game server, the memory usage stays the same. Infact when new players connect and start a new game the server doesn’t appear to release the unused memory. Has anyone else had this issue?

Thanks

Vortex

Edited February 18, 2012 by VoRtReX-uk

[.kju 3240]

a) What was the server fps (#monitor 1 - as admin)

b) What is your server.cfg

http://community.bistudio.com/wiki/server.cfg

c) What is your arma2oaserver.cfg/basic.cfg?

http://community.bistudio.com/wiki/basic.cfg

[VoRtReX-uk 10]

Hello,

None of the information below has been changed over the whole 9 weeks.

Server FPS with around 80 players was between 6-11 FPS.

The server.cfg below was being used:

// GLOBAL SETTINGS 
hostname = "XXX";	// The name of the server that shall be displayed in the public server list
password = "";						// Password for joining, eg connecting to the server
passwordAdmin = "";					// Password to become server admin. When you're in Arma MP and connected to the server, type '#login xyz'
reportingIP = "arma2pc.master.gamespy.com";		// This is the default setting. If you change this, your server might not turn up in the public list. Leave empty for private servers
logFile = "_console.log";			// Tells ArmA-server where the logfile should go and what it should be called



// WELCOME MESSAGE ("message of the day")
// It can be several lines, separated by comma
// Empty messages "" will not be displayed at all but are only for increasing the interval
motd[] = {
"xxx"
};
motdInterval = 7;						// Time interval (in seconds) between each message

// =======================
checkfiles[] = {"dta\bin.pbo"};	
// =======================



maxPlayers = 100;						// Maximum amount of players. Civilians and watchers, beholder, bystanders and so on also count as player.
kickDuplicate = 0;					// Each ArmA version has its own ID. If kickDuplicate is set to 1, a player will be kicked when he joins a server where another player with the same ID is playing.
verifySignatures = 1;					// Verifies the players files by checking them with the .bisign signatures. Works properly from 1.08 on
//equalModRequired = 0;					// Outdated. If set to 1, player has to use exactly the same -mod= startup parameter as the server.



// VOTING
voteMissionPlayers = 1;  					// Tells the server how many people must connect so that it displays the mission selection screen.
voteThreshold = 0.33;					// 33% or more players need to vote for something, for example an admin or a new map, to become effective



// INGAME SETTINGS
disableVoN = 1;						// If set to 1, Voice over Net will not be available
vonCodecQuality = 5;					// Quality from 1 to 10
persistent = 0;						// If 1, missions still run on even after the last player disconnected.



// SCRIPTING ISSUES
onUserConnected = "";					// self-explaining
onUserDisconnected = "";
doubleIdDetected = "";
regularCheck = "{}";



// some ArmA specific stuff - signature verification
onUnsignedData = "kick (_this select 0)"; // unsigned data detected
onHackedData = "kick (_this select 0)"; // tampering of the signature detected
//onDifferentData = ""; // data with a valid signature, but different version than the one present on server detected
BattlEye = 0; //Server to use BattlEye system



// MISSIONS CYCLE (see below)
//class Missions {};

Performance.cfg

language="English";
adapter=-1;
3D_Performance=93750;
Resolution_Bpp=32;
MaxCustomFileSize=0;
MaxMsgSend=256;
MinErrorToSend=0.0020000001;
MaxSizeGuaranteed=1280;
MaxSizeNonguaranteed=128;
MinBandwidth=4194304;
MaxBandwidth=2097152000;
Windowed=0;

Also just found that Intel SpeedStep appears to be enabled on the server. Does anyone else run with this enabled? cpu speed is going up (2.2Ghz) and down(1.6Ghz) with just a few people on the server

Edited February 18, 2012 by VoRtReX-uk

[.kju 3240]

So you are talking OA 1.60 only?

What about beta patches (server or clients)?

Small fixes:

reportingIP = "arma2pc.master.gamespy.com";

=>

reportingIP = "arma2oapc.master.gamespy.com";

checkfiles[] = {"dta\bin.pbo"};

=>

checkfiles[] = {};

verifySignatures = 1;

=>

verifySignatures = 2;

+ check this thread:

http://forums.bistudio.com/showthread.php?129754-Dedicated-server-performance

[VoRtReX-uk 10]

Yes, Server is 1.6 OA only. No beta patches on the server. But the clients....... can you force it so if they are using the beta they cannot join?

I have ammended the issues you pointed out, thanks!

Its funny you put that thread up as we are currently hosting the game for BDR Sibolo (ATC4) as they have had more issues that us, and our servers are supposed to be identical!

Edited February 18, 2012 by VoRtReX-uk

[NeuroFunker 11]

it is even worse in arma 2 vanilla, when around 50 people are playing at same server, it gets quite unplaybe when your driving as a passenger in someones car, or flying chopper. Just after few meters, car runs into a building or fence and stucks there, engine sound is running, nothing happens, but driver who was with me talking via skype, said he was driving, when the car stuck in building for me. Same with passenger in a chopper - it looks like chopper is crashing in to the water or on the ground, but the chopper is still flying (for person who's a pilot in that chopper).

[Dwarden 1124]

1. The arma2oaserver.exe only appears to use around 25-30% of the CPU capacity of the system

do we speak about utilization of single core or multiple cores?

because for A2:OA the primary concern is IPC (instruction per clock) on primary core

while the game is able utilize more cores (so dualcore is must, quadcore is worth it and 6 to 8 cores is nice bonus)

yet any gains beyond 6-8 native cores is negligible due to multicore sync

1.

also my suggestion is : try disable HyperThreading in BIOS

2.

change

MinErrorToSend=0.0020000001;

to

MinErrorToSend=0.001;

this value has new default since 1.60 build read http://community.bistudio.com/wiki/basic.cfg#Performance_Tuning_Options

3.

add

MinErrorToSendNear=0.01;

this is new value since 1.60 (tho if missing then engine should use default anyway)

you may try increase this value (e.g. to double, to lower stress on server)

4.

as seen in other topics You might not need

minbandwidth entry

and You may change MaxMsgSend to lower value

this is up to tweaking tho

Edited February 18, 2012 by Dwarden

[VoRtReX-uk 10]

Hi,

Total CPU utilization was 30% using multiple cores. But each individual core never goes over 35% CPU utilization, AKA it isn’t even maxing 1 core out!

I just did a test using (AAS100) [C5] The Gauntlet, with 800 AI. Total CPU usage is 20% none of the individual cores have gone over 35% usage, and the server is running at 6-7 FPS.

Task Manager pic from the test below:

We are going to disabling HyperThreading. I have also ammended the config settings you suggested. The server has 100mb download and 40mb upload, so I hopfully set the min and max bandwidth accordingly!

Thx

Vortex

Edited February 19, 2012 by VoRtReX-uk

[VoRtReX-uk 10]

Right some info from tonights game.

83 Players, pictures below are after 1hour 20 minutes of play out of a 2 hour game.

Task Manager, Core Temp and Server window.

Ingame Server FPS.

Pretty much 2 fps for the last 40 minutes.

HyperThreading is the issue 100%

Vortex

[naizarak 4]

I can't answer your questions, but I'll just add that I've experienced this "desync" in 1 of my recent games, too. My ping to server was 100ms, average player ping was 120ish, but the game was very glitchy. On my screen, my humvee would get stuck in a tree while my driver said that we were moving the whole time. Outside of vehicles, other players teleport around and don't animate properly. Server only had 10 players, too. Arma 2 free 1.10

[.kju 3240]

2 FPS is too low. The server should remain at least above 5.

What mission/game mode are you playing?

Did you make sure it is as optimized as possible?

[Dwarden 1124]

I can't answer your questions, but I'll just add that I've experienced this "desync" in 1 of my recent games, too. My ping to server was 100ms, average player ping was 120ish, but the game was very glitchy. On my screen, my humvee would get stuck in a tree while my driver said that we were moving the whole time. Outside of vehicles, other players teleport around and don't animate properly. Server only had 10 players, too. Arma 2 free 1.10

arma 2 free 1.10 ? quite obsolete build it is, latest is 1.11

also please realize that ARMA 2: OA (ARMA2: CO) is 1.60 (public betas 1.61) where the netcode is miles ahead of old ARM 2 version 1.1x

[Sibolo 10]

I can recognize the core usage pattern (at least for this mission) involving mainly 3 threads in three different cores. I'm also quite sure that disabling HT is going to give it a boost, but I don't know how much that will be in terms of FPS. Anyway I think the server managed the game quite well even with those 2 FPS, didn't see anything too strange and at the very end both vehicles and infantry units were moving smoothly. Did any of your guys complain about lags?

There was only one point in time that I disconnected and couldn't see the server on the list and someone was asking me if the server was stuck, but it was only for a few seconds. I then reconnected succesfully and didn't have any problems for the rest of the game.

[VoRtReX-uk 10]

What mission/game mode are you playing?

Did you make sure it is as optimized as possible?

I will have to leave Sibolo to answer those questions as he edits the map.

Vortex

[.kju 3240]

The basic.cfg/arma2oa.cfg certainly can have a significant effect on performance overall:

http://community.bistudio.com/wiki/basic.cfg#Performance_Tuning_Options

However before try to tweak it, for a high player situation its not easy to test,

you may want to try other potential areas of improvements first (like mission design/addons).

[enforcer1975 0]

Mission design is the key. You can have a mission that's 10mb large or tweak it to half the size with better performance.

There is a lot of data sent during a mission. If you limit client data to clients and update every x minutes or when there is a change and only send server / gameflow relevant data continuously traffic will be minimized and the mission can run longer without getting out of sync.

[nomad_man 10]

Size != performance.

[VoRtReX-uk 10]

The mission is only around 544kb!

[VoRtReX-uk 10]

Update!

Tonights game was ok till the last 30 minutes.

As usual I was connected via remote desktop so there was no network issue.

CPU usage was between 85-98% on 1 of the 4 cores (HT now turned off).

Server was running at 4fps! 50% improvment!

The issue started when a player lost connection to the server and tried to rejoin (he is a repeat offender and rejoined the game 19times in the 2 hours!:confused:)

It appeared to lock up the server for around 3-4 minutes. When it did start moving again it dumped all of the #monitor messages and side chat messages onto my screen in one go!

Looking at the remote desktop console the CPU usage and network usage stayed the same during the lock up.

To me this looks like a issue with players connecting!

I though BIS sorted this issue?

Thx

Vortex

[Sibolo 10]

...

CPU usage was between 85-98% on 1 of the 4 cores (HT now turned off).

Server was running at 4fps! 50% improvment!

...

Good!!! (Although it is in fact a 100% improvement :cool:)

As far as mission optimization that some of you mentioned, basically I inherited the mission file from the previous campaigns where we didn't have so many problems even with a higher player count (50vs50). I did change many things, but always trying to do my best to keep it optimized. Most of the scripts are running only client side, variables are sent through the network only if really needed and so on.

Mission dimension is not always the most important fact (although big missions are often overbloated) as you can write only one line of code and if it's done wrong it will mess everything up.

But there is always a lot to learn on this topic.

Edited March 1, 2012 by Sibolo

[PreedSwe 18]

Our server also has some issues with alot of players being disconnected.. The server says it's due to high ping, and during this mass-lag before people get disconnected. the affected players ping goes up to and over 2000.. Everyone elses ping is just normal.. CPU is holding around where it usually is so it does not seem to be caused by the CPU getting too much to handle..

We recently upgraded our mission from 52 players to 70 and that is when this started happening.

Our server is running linux on a i7 2600, 16Gb ram and 1gbit uplink.

Not sure what #monitor 1 said at the time of these happenings as I have not been on, but right now(47 players) it says:

FPS: 6

Memory used: 844 MB

out: 9813 Kbps

in: 583 Kbps

CPU load in top is around 145%(not even 1.5 out of the 8 cores..

I will try and grab some data when we have a higher player count, and if Im lucky, even get some data during one of those disconnects..

When the players complain, we just blame it on BattlEye as usual though ;)

I hope this issue gets fixed, because we were hoping to keep increasing the player count on our mission maybe even up to 100.

[.kju 3240]

What mission are you using?

[Sibolo 10]

...

CPU is holding around where it usually is so it does not seem to be caused by the CPU getting too much to handle

...

FPS: 6

Memory used: 844 MB

out: 9813 Kbps

in: 583 Kbps

CPU load in top is around 145%(not even 1.5 out of the 8 cores..

Looking at these I believe that you do have a CPU bottleneck (although it might not be the only problem).

Basically my understanding is this:

Your overall CPU is divided in 8 cores (4 real and 4 logical due to Hyperthreading). To use your entire CPU power the programs you are running should be able to spread the workload on all cores generating and using enough threads. This does not appear to be the case of the Arma server. From what I have seen on our server and on our mission, only 3 threads (out of about 15 created by the server) where actually using CPU power. One was maxing out it's assigned core, the other two where only using a fraction (around 10-20%) of their assigned cores. This basically sums up to what you point out to be 145% CPU load (on a 800% maximum). But I believe that the mistake in it's interpretation is that it's not using more CPU power, not because it doesn't need it, but because it is not capable of doing it. So you end up using only a small amount of your overall CPU but experience low FPS.

Infact, in general, one thread can only be executed on one core, so if an application would generate only one thread it would never be able to use the remaining 7 cores, even if it was struggling for CPU cycles.

My advice is to disable HT as this has shown to give better performance (almost a 100% increment in FPS) by not splitting the cores. The main thread will run on a complete core and so will have more CPU power assigned to it raising your FPS.

But in my experience this has not been enough, we are still having quite low FPS on large player count, and have disabled BE because when the server occasionally lags (freezes) many players are kicked out by BE and have a hard time rejoining. Also having 20-30 people disconnecting at once and trying to reconnect is very hard on the server and makes things even worse.

Good luck.

[Party3AH 0]

Vortex, we have PvP-only (no AI at all) 120-150 person on server, hardware is like your (differs only CPU - we have Q9550 @2,8 MHz)

I may give You our configs, if you want.

Also, please note: some mapmakers may add enough syclic checks (triggers etc) - and this may also cause such problems.

[nomad_man 10]

Party3AH - i wouldn't mind the config.