High spec dedicated server configuration

[teacup 2]

Hi all,

Thanks to the generosity of one of our members, we (Hungarian ArmA Community) have the good fortune of running a dedicated server on a beefy machine. Here comes the bragging part:

------------------------------------------------------------------

VMWare 3.5 ESXEnterprise Virtual Machine

IBM HS21XM Blade

2 socket Intel Xeon 5450 Quad Core 3Ghz processors

16Gbyte RAM

FC storage (currently 50Gbyte allocated to the server)

The blade system has a direct dual 10Gbit connection to the net, the current server gets a 1Gbit connection from it. The effective maximum is about 700Mbit/blade server.

Software: MS Windows 2003 Standard, 32bit

------------------------------------------------------------------

Plenty of brute power to throw at the server both CPU/RAM and bandwidth, but still we're having performance issues (no surprise there). Our ambition right now is to run as smooth as possible with 60+ players, and maximize server performance. The maps we're playing are a flavor of C&H, optimized for minimal CPU/Network load. No mods, no AI.

So far, the biggest problem is clearly JIP, and the resulting desync. We have experience running our own servers (for fewer players), we played in IC-ArmA, AToW, played on some of the more populous servers - we know "JIP lag" is endemic to ArmA. Still, if you're experienced with running games this large, we would really appreciate some good advice. Your settings, fixes, workarounds or just related experiences and best practices.

The main questions i suppose are:

1. What performance settings do you use/would you recommend? Why, and what are the valid ranges? Ours settings at the moment:

------------------------------------------------------------------

MinBandwidth=15000000;

MaxBandwidth=500000000;

MaxMsgSend=1024;

MaxSizeGuaranteed=128;

MaxSizeNonguaranteed=64;

MinErrorToSend=0.001;

MaxCustomFileSize=1;

------------------------------------------------------------------

2. How do you check clients, if at all? Do you have verifySignatures on? If so, how does that impact performance? What about BattleEye?

We could ignore all this and elect to just trust all clients, since it's not a public server. Our players are friends, guests, mostly people past the age of cheating. But they can be careless, so some way of running even basic checks could be nice.

We're testing and trying to figure things out, but if you guys could help us, you'd be saving us (admins & players) some frustrations and headaches.

Thanks in advance.

[cross 1]

You can increase the max size guaranteed.

You'll need bigger packets as the server FPS gets lower so that you carry more info with low frequency.

Try;

MaxSizeGuaranteed=1024;

I'd suggest make disallow any customfile size by as it can cause some JIP lag.

MaxCustomFileSize=0;

Rest looks fine.

Try these first and if you are satisfied, you can try increasing minbandwidth.

Try;

MinBandwidth=50000000;

You can also try increasing maxmsgsend but 1024 should handle it fine.

MaxMsgSend=2048;

Mission and scripting in it, is important in JIP issues. Maybe more important than server side settings. A well (or less :) ) scripted mission mission will give you more server FPS and will improve your JIP desync issues more than tweaking these settings.

Also make sure your there is enough CPU allocated for the Arma2 server..ie it can utilize 2 CPUs at least.

Edited December 17, 2009 by Cross

[teacup 2]

Thanks for the tips Cross. Tried them last night, 44 people, 2 hours. It was smooth gaming all the way with people joining during the fight too. Server FPS started out at 19 and bottomed out at 9 during the last half hour. Sounds kinda' bad, but the game was running without a hitch. We had one core maxed out, one at 80%, a third core at around 10-20%, and the fourth unused. Viewdistance was 2.5 Km, and i don't think the mission we played was optimized for many players.

Mission and scripting in it, is important in JIP issues. Maybe more important than server side settings. A well (or less :) ) scripted mission mission will give you more server FPS and will improve your JIP desync issues more than tweaking these settings.

Is there anything that stands out, that impacts server FPS or JIP disproportionately? Too many triggers, busy init scripts, public variable broadcasts or endless loops? Evil commands?

We were again unable to use verifySignatures, it really bogged down the server.

[brit~XR 0]

I dont get it. You have 16gb ram but use only 32bit os :confused: 12gb wasted. Not that you need 16gb on a quad for a server anyway. 4 Gb is more then enough.

As for the performance settings i fill the max bandwidth settings are way to high. Heres what we use for 32 player warfare

MaxCustomFileSize=0;

MaxMsgSend=512;

MinErrorToSend=0.008;

MaxSizeGuaranteed=1024;

MaxSizeNonguaranteed=512;

MinBandwidth=18000000;

MaxBandwidth=19000000;

Them settings handle 32 player warfare fine and also tested them on 50 player berzerk and server fps was at almost 50fps still This was on arma and arma 2 berzerk on a q9550 @ 3.4ghz 8gb ram

19fps with 44 players and 9fps 30min later just seems very bad Either the mission is badly scripted which i dout sence most c&h maps run pretty smooth due to no ai or its your proformance settings. And my guess is its the high proformance settings killing your server fps. We get more fps then that on warfare with 32 players with tons of ai and vehicles on map. around 15fps hour or 2 into game. If 4 to 5 hours into game then we hit around 8fps due to so many player controled ai on the map and many islands buildings being destroyed

Just because you got a fast connection doesnt mean you should config a server for that. We have 100mbit but task manger shows around 4mbit usage per server for 32players. Configing server to use more bandwidth then it really needs will just make problems and have a inpact on the server fps and proformance

Thats just my views from testing i done in the past sence the ofp server days

Edited December 18, 2009 by brit~XR

[RN Escobar 0]

19 FPS at mission start, wow, that horrible.

CPU affinity set? Using anything like FireDaemon to lock the process to selected cores or letting the OS move it around with free abandon inducing micro-stutters as the the cache is moved from CPU to RAM and back again?

What other servers are on this shared box? and if the box isn't shared why the hell are you running the server on a Virtual machine?

What are the resources actually allocated to your VM area?

its great to bragg about the box, but i get the impression that it isn't solely yours so depending on what other servers are also running from your shared box you may never get the results your looking for. A big Dbase will hammer the storage system, as an example.

as it stands there are just too many variables to point the finger of blame at the server settings as the cause of the problem.

Esco

Edited December 18, 2009 by RN Escobar

[brit~XR 0]

VMWare 3.5 ESXEnterprise Virtual Machine

Didnt notice that when i made my post. So the actual vmware is Windows 2003 32bit And the real server os is 64bit i take it. Either way its not a high spec dedicated server as you claim. Dedicated means you own the system box and have all its resources to use.

[teacup 2]

Actually, prior to installing FireDaemon and locking process to cores, and with our old settings, we had one core maxed out, one not even half way, the rest unused. And we were getting between 45 and 50 FPS with 40 people. But the game wasn't as smooth as this time around, crappy server FPS and all. We made several changes, so now it's hard to isolate and figure out what affected what. It's hard to get 40 people, just to do boring testing.

The "box" is more like a "chicken coop" really. It's a blade system, and virtualization is required to span across all the blades and other devices. Server performance depends on the quotas allocated to your server OS, and this one has more than the ArmA server could ever max out (they're listed above). Our server is also located on a backup/spare, with very little else competing for resources.

I doubt storage would be the thing slowing it down, since it was perfectly capable of running near 50 FPS before with 40 players, and the server is nowhere near the disc hog the client is. But we'll see..

Edited December 19, 2009 by teaCup

[cross 1]

Thanks for the tips Cross. Tried them last night, 44 people, 2 hours. It was smooth gaming all the way with people joining during the fight too. Server FPS started out at 19 and bottomed out at 9 during the last half hour. Sounds kinda' bad, but the game was running without a hitch. We had one core maxed out, one at 80%, a third core at around 10-20%, and the fourth unused. Viewdistance was 2.5 Km, and i don't think the mission we played was optimized for many players.

Is there anything that stands out, that impacts server FPS or JIP disproportionately? Too many triggers, busy init scripts, public variable broadcasts or endless loops? Evil commands?

We were again unable to use verifySignatures, it really bogged down the server.

Great you were able to run it without a hitch. Yes not the best perf but I think but with 44 people playing 2 hours without any problems and finishing @ 9FPS is not bad at all. Actually 9 FPS is OK when you play. Its just that AI may become a bit less responsive and you may get some desync when people try to connect at that point.

All the factors you listed about scripting are valid. Lots of global commands and so many different ways of doing it so it is impossible to list them down I think. But generally these were the factors I was talking about.

Now we now know that server perf is a combination of mission_and_scripts+addons_used+arma.cfg+hardware, and in that order.

Personally I came to a conclusion that there is no single config that gives you max performance on every mission. For example there is a nice misison called Airbase-ACE. On KH server, with 30 people it starts @ 5 FPS, with 3 players it starts @ 17 FPS, and it crashes everytime. AirCav starts @ 30 FPS with 10*15 players if I am not mistaken. I can't say Airbase-ACE is bad mission, its a good mission and I'm sure there are many servers running it without any issues, but it just doesn't like my combination.

As XR suggested, yes..you can lower you max bandwidth to a more reasonable level. Check in game IN & OUT peak values and you can adjust depending on number of players. I know there are servers having 30-40 Mbit traffic with many players. At KH what I see is max 14-15Mbit (usually less) with 30-35 players.

Personally, I've never seen ill effects of max BW but if you keep the min BW @ too optimistic/high values, it does have some reverse effects on performance.

One note on his comparison with warfare though. Warfare may have dead body removal eventhough you have tons around and the other mission may not. So it really really impossible to compare performances across servers unless you have the same mission with same numbe of people and with same number of people connecting/disconnecting creating extra traffic.

Now that said, I am neither an IT professional (besides having enough knowledge on IT) nor I've come to these values by doing deep network traffic / processing analysis. These are the values with which I don't have too much warping/teleporting of AI and have playable FPS in most missions.

For further tweaking, I suggest you create a test server, keep it clean of 3rd party addons, have 2-3 selective missions like 1 heavily scripted, 1 lean, 1 warfare etc.. and test out FPSs and performance by making changes to arma.cfg one at a time. Especially, change minBW & Max BW and see the results yourself.

Just a small note so that you can understand how important non-arma-config stuff is. Initial day of ACE release; KH server runs great with 30-35 people on for hours, even 50 people on at least for 2 hrs last friday and not a single crash. Some betaserver & ACE & mission updates and our server started crashing at least 5-6 times a day during mon-thu period. Got a bit better after we turned signature checking on and thank god its doing fine so far with todays update. And same cfg file of course.

Phew long post :P

PS..if you are running betaserver, make sure you add BattlEye=0 to server.cfg to disable battleye check which come on as default .

[teacup 2]

Huh! Thanks for the effort Cross. We were hoping for a quick copy-paste of some proven config files, but after what you're describing - i guess we gotta do the leg work and set up test cases, isolate all the factors, yadda-yadda.. and adapt to the peculiarities of our server and missions. Man i hate actual work! :icon_razz:

[Szepy 10]

Hi guys!

Personally it's me who's responsible for the HW/Virtual machine of this server.

Regarding notes about "it's not dedicated" and stuff it's a blade server farm, running VMWare virtualization OS. It's true, that on this very blade server there are also other VMs - just to see the truth the others are alltogether not using 15 percent of this servers CPU and memory and they're locked down to this rates. Furthermore I plan to move this VM to a really "empty" blade server as soon as the new shipment arrives.

The Virtual machine currently running 32bit WIN 2003 Server OS because I was not sure how it would bear the 64bit version. At least I did not even found clues about it running on W2003 server, only on XP/W7, etc.

So the truth is that we have the performance of the above mentioned server's 85% (at least), only the memory that is maxed out at 4Gbyte because of the OS version.

If you say, that this application would use more, I'll also change it to 64bit version, but never seen more than 600Mbyte of memory usage, so I kept this change useless to our performance problems.

I also monitored the CPU usage, even if the server is running at 10 FPS it would not use more than 30% of the whole server's CPU power. The VM is set to use the whole CPU and RAM resource on this server that's avaliable, so it's not about the VM's boundaries. I've seen impressive desktop configs at your server's stats can you make it sure, that the whole CPU and memory resource is used by your Arma2 server?

I can change it to simulate 2 CPU cores, de difference beetween 2 and 4 cores, that 2 cores are used evenly, 4 cores the way teaCup described it - 1 on full, another 2 partly, 4th is almost at 0 percent. No change on warfare server FPS beetween 2 CPU configs.

I'll try changing the network numbers - as I've seen it 100mbit on our config would be enough for even 256 players. :) I'll max it in 100mbit and set the minimum to 75mbit.

Cross: thanks for your help regarding your experiences with different mission types. We've thoughts about it but wasn't sure whether it really makes difference or not.

Edited December 19, 2009 by Szepy

[SWAT_BigBear 0]

MaxCustomFileSize=0;

MaxMsgSend=512;

MinErrorToSend=0.008;

MaxSizeGuaranteed=1024;

MaxSizeNonguaranteed=512;

MinBandwidth=18000000;

MaxBandwidth=19000000;

I've been using this set for a couple of months and they seem to be on spot for our dedi.

We play cooperative with usually tons of AI.

The box is a Q9400 2.66Ghz, 8Gb ram on Server 2003 64bit, 100Mbit port, Peer1 (25-27Mbit upload).

It's just for games, nothing shared, no web/ftp/SQL ect.

It also host 3 RvS and 1 CoD4 servers, (I know, quit laughing).

The ArmA server is still stock 1.04 atm now 1.05, pinned to 2 cores w/FireDaemon Trinity (others use the remaining cores).

We do get the red chain upon players joining, though game play doesn't seem to be affected.

Thanks again brit~XR, for the numbers.

Edited December 24, 2009 by SWAT_BigBear

[Game__On 10]

I don't get all these '9 fps is good enough' .

It shows bigtime on the players side, especially if they have full auto weapons.

Rate of fire is easily cut in half or worse .

9 fps server side may not be bad for those with their dmr sniping , but if you need to fire bursts you need higher fps on the server's side, or you simply won't hit sh^t as your bullets scatter like a shotgun .

I beg all of you who are running a server, keep the server side FPS as high as possible ! I think it needs at least 20 fps to be completely smooth on the players side.

[Szepy 10]

Thanks for additional datas BigBear! Maybe we should try to increase the MaxSizeNonguaranteed and the MinErrorToSend values. I suppose MaxMsgSend has nothing to do with that problem.

One more question, I'm seeing MaxCustomFileSize=0 value in your configs. For us it seemed that with this setting everyone could use any custom files, but with setting it to 1 they were unable to connect with them. That's why we use 1 intead of 0 at this setting.

Szepy

[SWAT_BigBear 0]

Good point.

brit~XR would have to answer that for us.

[cross 1]

//MaxSizeNonguaranteed = 64;

//*Maximum size of non-guaranteed packet in bytes (without headers). Non-guaranteed messages are used for repetitive updates like soldier or vehicle position. Increasing this value may improve bandwidth requirement, but it may increase lag. Default: 256

//MinErrorToSend = 0.005;

//*Minimal error to send updates across network. Using a smaller value can make units observed by binoculars or sniper rifle to move smoother. Default: 0.01