Server Performance

[CouchMonkey 0]

I was going to post this in the dedicated server thread but that thing is going all over the place.

I wanted to briefly talk about some server experiences I have. For background here are our server specs:

Dual AMD Opteron 265 (Dual Core) Processors(4 CPUs)

2GB PC3200 Registered ECC Memory

160 GB Hard Drive

100Mbit dedicated port

We own it, we don't share it.

Now, as I understand it, ArmA does not currently support multiple cores. So if the processing is dispersed, it will slow things down considerably. The problem as I see it is that unless you have some insane single core machine, there is no way fully experience what ArmA has to offer. With our server we can have multiple smaller servers running which can sustain about 20 or so comfortably but when we try to fit all of our members on on server (about 60) we get server side FPSs of about 10-20. I know it all depends on the mission itself and such.

For the mission we are running about 30 objectives with associated triggers. We also have slotted for about 30 players on each side. There is no AI. I am well aware that this in and of itself is taxing but should be well within the scope of advertised 100+ player warfare you all advertise. In addition we have top minds working on the mission file to increase performance so let’s keep the discussion to the hardware or game please.

My question is this. Are there any steps being taken to either:

A. Reduce the load out required by the server (reduce the system requirements of the server)

B. Creation of a dedicated server which can fully utilize multi-core technology.

I think you will find that more and more server providers are exclusively offering multi-core server because most servers out there can run perfectly fine on them. ArmA is a different beast all-together.

If you would like more information about our mission or our server please let me know.

Thanks for your time.

-CM

[CouchMonkey 0]

Update:

We had approximately 45 member on the server tonight. We made them log on in shifts.

1-10: 45 Server FPS

11-20: 33 Server FPS

21-30: 15 Server FPS

31-40: 7 Server FPS

41+: about 4 Server FPS

As you can see the more people who join the more lag there is.

Is BIS working on a better server file or working on ways to improve the netcode? I need something to tell my guys.

Thanks,

CM

Proof: http://www.armainteractive.com/images/other/Beta2.gif

[SWAT_BigBear 0]

100mbit port doesn't mean much, unless it's "un-metered", which could cost well over a grand a month depending on your host company.

The prob you just seen was the "metered" 100mbit port slows everyone down as more joined.

[CouchMonkey 0]

That's a good point, but I am getting about 1300 KBS down and 300KBS up according to the server monitor. Obviously that grows as more people join.

[SWAT_BigBear 0]

I was disappointed after purchasing the 100mbit port(just $25), just to find out they meter them to only keep a connection alive when ALOT of players join. Even though your server box is elite, it doesn't help. My provider, Server Beach, want's $1300 per month to unmeter the port, which would run any game, any amount of players! Our backbone is dual oc48 direct to Peer1. I've ran many test on a few rental boxes for performance/speeds, here is an example:

I ran this test (all game servers closed) to a local test site (blocks away) which gave over 9800kbps up, but the real test was from Virgina to Texas which gives a better "actual" speed-Results

When I have the 5 different games running (RvS & OFP) and say, 40 players in RvS, and 10 or so in OFP, those test #'s will be several thousand less! If I were to purchase the unmetered port...the test would show the higher kbps, as if the games were not running, and no one would ever say "lag" from the servers when they are all full.

[CouchMonkey 0]

Thanks for the input. I honestly don't know if we are running unmetered. I will post the question to my providor. If we are running metered, I will see if we can run unmetered for our next battleday and see what happens. I will post the result here when I get it.

Thanks

[CouchMonkey 0]

OK, so I figured out part of my problem. I had the firewall on

Let's see what the server does this weekend. I will keep you all updated.

[-ASW-AllShotUp 0]

I was disappointed after purchasing the 100mbit port(just $25), just to find out they meter them to only keep a connection alive when ALOT of players join. Even though your server box is elite, it doesn't help. My provider, Server Beach, want's $1300 per month to unmeter the port, which would run any game, any amount of players! Our backbone is dual oc48 direct to Peer1. I've ran many test on a few rental boxes for performance/speeds, here is an example:

I ran this test (all game servers closed) to a local test site (blocks away) which gave over 9800kbps up, but the real test was from Virgina to Texas which gives a better "actual" speed-Results

When I have the 5 different games running (RvS & OFP) and say, 40 players in RvS, and 10 or so in OFP, those test #'s will be several thousand less! If I were to purchase the unmetered port...the test would show the higher kbps, as if the games were not running, and no one would ever say "lag" from the servers when they are all full.

We run a 100mb metered port since which you are correct in saying, Unmetered is better but expensive. I am unsure how many game sites run like that though.

However I have no problem running 2 BF2142 servers and MS FLt SIM /PAC Fighters off of a 100mbps line.

We used Server Beach a couple years ago, great people but we did experiance lag even with a 100mbps port out just running BF42/DC however the best location we found with them was to have our server in VA, since it helped the people here in the US and Europe the best.

I would not go back to a 10mbps line - it would not be worth the hassle.

s/ ASU

[-ASW-AllShotUp 0]

OK, so I figured out part of my problem.  I had the firewall on  

Let's see what the server does this weekend.  I will keep you all updated.

Are you running a seperate firewall on your box then what your provider has? Best is to open the ports needed for each game and leave it at that.

If you have a dual core and a windows machine...you might be able to just use one core for your ArmA by changing the AFFINITY.

Use your Task Manager - from RUN - type in taskmgr

Then when Task Manager opens - go to processes, and highlight ARMA - then right click on that and you should see a dropdown block that says SET AFFINITY - there you should be able to select core #1 or core #2.

Then you can use your TASK MANAGER to see what is your performance before and after besides hopping in the game and checking it there also.

BTW - that was the "fix" that was out for BF2 when they came out without dual core support.

[CouchMonkey 0]

Thanks.

I talked to my provider and I am running unmetered. I think the problem eas that all the traffic was being funneled through the firewall, since it does checks, it was ultimately slowing it down.

As for setting the affinity to a single core, that really isn’t an option. When you have over 40-50 people connecting, you will use more than one processor can handle. We have a quad processor machine. If you set the affinity to one core it run slower than crap for obvious reasons.

We made some other tweaks. I will find out Sunday (our next batttleday) and let you guys know what happens.

Thanks again for the suggestions.

-CM

[raedor 8]

Please stop double-posting, CouchMonkey

[CouchMonkey 0]

I double posted? I am sorry, where is the other post?

If you are referring to the server files post, that is a completely separate post. Which I am still waiting for an answer on

[-ASW-AllShotUp 0]

Thanks.  

I talked to my provider and I am running unmetered.  I think the problem eas that all the traffic was being funneled through the firewall, since it does checks, it was ultimately slowing it down.

As for setting the affinity to a single core, that really isn’t an option.  When you have over 40-50 people connecting, you will use more than one processor can handle.  We have a quad processor machine.  If you set the affinity to one core it run slower than crap  for obvious reasons.  

We made some other tweaks.  I will find out Sunday (our next batttleday) and let you guys know what happens.

Thanks again for the suggestions.

-CM

Just so we are on the same page and in case terminology is different.

"Unmetered" usually means in amount of bandwidth used not speed. Hence an 10mbps "unmetered" conn. is still 10mbps - it is just you are not charged for an overage's since you do not have a limit.

We run a 100mbps conn, which means we are going out 10 X as fast as a 10mbps machine...You do not need to ask your server provider this unless you do not have "remote desktop" access.

Like any windows machine, you go to SYSTEM - read your CPU type and Memory, go to Device Mgr - look up all your onboard equipment, then go to network connections and read your bandwidth connection. It will say either 10 or 100mbps.

We do not even come close to using 2000gb per month and that is with running a BF2142 Tournament Server, MS Flt Sim 2004 / X server and a PAC Fighters server off the same box. Even our BF2142 and BF2 ranked servers never pulled that much with a good load. Since 2003 when we started running BF42 we never went over 2000gb...

So see what you are running and if you do not need it - save your money. BTW - 10mbps "unmetered" is usually around $275.00 US extra versus a 100mbps link out for $20.00

As far as "Quad Cores', I thought you had "dual core", my bad for not reading correctly. However the procedure I gave you was an actual fix from EA"s Tech Dept. last year for server and client machines when we had a problem with BF2 and "dual core" before they released a patch.

[CouchMonkey 0]

Well, I talked to my provider and they do not meter it in any way. I don’t know what to say. Also when I take a look at the task manager it says 100.

In any event, I cannot set the server affinity to one core because that core gets maxed out at the level I am running. If I have 40 people on I run at around 30% processor when the affinity is not set and I max out the processor when it is set. Believe it or not, the server run a lot slower than if I didn’t set the affinity.

I have run a BF2 server before on a multi core system. It didn’t take nearly the amount of processing power ArmA does. I had a 64 slot server and was able to set the affinity no problem. But I can’t get 40 people on in ArmA without maxing out one processor.

Long and short of it is, setting the affinity to one core is not an option.

[SWAT_BigBear 0]

AllShotUp @ Jan. 09 2007,01:15)]We used Server Beach a couple years ago, great people but we did experiance lag even with a 100mbps port out just running BF42/DC however the best location we found with them was to have our server in VA, since it helped the people here in the US and Europe the best.

Since that time, they built DataCenter2 which qualified for direct to Peer1 and skipping the Cisco networks. That alone dropped 4-6 hops to every member of our group.

They have several DataCenters, but most arer for web hosting..i,e L.A./Dallas/ Virginia #1 datacenters.

You would be suprised at the larger companies that rent from SB. They do not support gaming, just the boxes on their fat pipe. At 1 point, I was going for a 2nd box in L.A. for my guys on the west coast, but I was refered back to Datacenter2 for gaming.

CouchMonkey....

Since you have 4 cores, have you tried to pin ArmA to 2 of them and pin other apps to the others?

[-ASW-AllShotUp 0]

AllShotUp @ Jan. 09 2007,01:15)]We used Server Beach a couple years ago, great people but we did experiance lag even with a 100mbps port out just running BF42/DC however the best location we found with them was to have our server in VA, since it helped the people here in the US and Europe the best.

Since that time, they built DataCenter2 which qualified for direct to Peer1 and skipping the Cisco networks. That alone dropped 4-6 hops to every member of our group.

They have several DataCenters, but most arer for web hosting..i,e L.A./Dallas/ Virginia #1 datacenters.

You would be suprised at the larger companies that rent from SB. They do not support gaming, just the boxes on their fat pipe. At 1 point, I was going for a 2nd box in L.A. for my guys on the west coast, but I was refered back to Datacenter2 for gaming.

CouchMonkey....

Since you have 4 cores, have you tried to pin ArmA to 2 of them and pin other apps to the others?

Good Point Big_Bear, Server Beach got bought out a year or two ago by Peer1, which I believe is a Canadian company. Good service etc even before they got bought out. Alot of the lag problems was traced out to crossing over the different lines that were owned by Cogent and other companies.

We have no problem using them since they gave fine service, but at the time we were going through an upgrade of our servers, we went to a different provider with better specs at a lower cost. Most of our traffic comes out of central and eastern North America (Can / US ) and Western Europe.

In fact we are getting ready to go through another upgrade again. So we are exploring all our options now for what is most cost effective.

s/ ASU

[CouchMonkey 0]

CouchMonkey....

Since you have 4 cores, have you tried to pin ArmA to 2 of them and pin other apps to the others?

Great question Bear. Yes we have. In reality it ran about as fast as no affinity, yet we were still getting about 7 or so server FPS.

So far running on all 4 cores is still faster. But not very.

[sickboy 13]

Great question Bear.  Yes we have.  In reality it ran about as fast as no affinity, yet we were still getting about 7 or so server FPS.  

So far running on all 4 cores is still faster.  But not very.

What happens if you let those 41 players join in a Player versus Player map (no ai, no scripts, just plain, default player versus player)?

Is there any difference, and how big is the difference?

Then you could also make a coop map with let's say:

45 player slots

then different versions with added:

25 ai

50 ai

100 ai

200 ai

(just placed in the editor doing nothing)

and check what your server fps is then.... by doing this you can figure out what the strain is of the amount of players, what the strain is of the amount of ai with this amount of players and if it got something to do with a ratio between player and ai numbers, or possibly that the engine is not yet optimized proper enough or that it's simply not possible to play with 41 players in the configuration of the maps

I think our server, same cpu configuration, got 7 server fps with ~500 fighting units on the screen with 2 civilian players overwatching it... but I dont remember the exact results

BTW, quad server or not, ArmA only supports using 1 core at this time, which basicly means that it doesn't matter if you put the process on 1 cpu or 4 cpu's, it will always only use 1 of them, and if you select more cpu's... it's possible that it starts switching the process between the cpu's which could result in slower operation in some cases..

[CouchMonkey 0]

yes, I know that ArmA only supports one processor/core. What I am saying is that with 40+ real people and not AI, it more than maxes out the processor. Distributing the load actually increases performance but as you know, it is not ideal.

As fro the AIs, I will definitely check that out. Great suggestion.

[CouchMonkey 0]

Alright. So here is what I did.

1. I took our custom mission and enabled the AI. We have about 40 slots per side.

2. Started up the mission as a civilian (we have three sides, Russian, US, and Civvies).

3. The server hung tough at around 25% processor (~1 full core), 300 MB memory used, and 47 server FPS.

4. I have the AI move around a bit (they aren’t scripted because they are just placeholder slots). The server bumped up some, but hung at around 40 FPS.

So, from what I can tell it has to do with the number of connections and not the mission file.

As I said above, I had the firewall turned on so it was funneling the traffic. I will see what it does this weekend after turning it off and report back here.

Note: Most servers that you rent slots from (i.e. 64 slot BF2 servers) do not have the firewalls on at all.

[sickboy 13]

Quote[/b] ]Distributing the load actually increases performance but as you know, it is not ideal.

It's simply not possible to 'balance' the load, as it will simply only use 1 of the 4cpu's... it can't use more than '100%' of '1 cpu', so in any situation it should be evenly fast...

The only reason why it can be faster when you select all cpu's compared to selecting 1 cpu, is when other programs start to use that specific cpu to do their tasks...

what I would do is keep 1 cpu free for all other tasks, and host servers on the other 3 cpu's with priority 'above-normal' this will rule out any other process trying to make use of those cpu's, and they will usually be scheduled on the free cpu then You can then aswell leave some game processes running on the free cpu, but not in higher priority or your whole system starts lagging under load

Anyway, as said, it can easily be a problem about the AI/Player ratio... so per example: 1 player 500 ai is fine,  2 players 250 ai is fine, 3 players 200 ai is fine, 10 players 100 ai is fine, 20 players 50 ai is fine, etc. etc. etc.

Besides, as said, make a BARE new mission and test in there, DO NOT test in your current framework as there are way too many variables to rely on; when ur testing you want to rule out as many variables as possible if you wish to find the cause soon

So simply build a test mission with 45 player slots in 4 versions:

- No AI

- 50 AI

- 100 AI

- 200 AI

or something similair, then run every mission once and see what the server fps does...

If you still have a good server fps with 100 and 200 ai, with ~40 players, ... then it doesnt seem to have to do with amount of players/ai or the ratio between them, but with the scripting/setup of your mission

It is important that there are ~ the same amount of players on the servers while testing, so I hope you can lock ur m8s in for about 30 minutes 'for the better'

When you have these test results, you can base ur next steps on that and draw conclussions

BTW, I think it is important that no player has desync created by his connection (I say 'by his connection' because the server is generating desync itself when the server fps goes lower than 15!... If there are players with connection problems (not generally 'higher' pings, but really desync), then this will lag the server aswell afaik...

[-ASW-AllShotUp 0]

Hey CouchMonkey -

What kinda load are those other CPU's pulling when the 1 core is at 25%? I am curious for future reference on an upgraded box we are going to be switching into soon.

s/ ASU

[CouchMonkey 0]

Sickboy,

What I am saying is that during the battledays we have no AI, therefore no lag couple be caused by them.

However I see you point about the testing. I can say that we had about 20 people on the other night testing out a particular part of our mission. It dealt with chase cams for the admins to monitor the members to watch out for cheaters and to watch the general course of battle.

The mission contained 20 slots (all human), minor camera scripts, no weapons, no add-ons, and the stock Sarhani map.

The server was cruising at about 22% and about 30-35 Server FPS. It is only slightly better than if I had our full custom mission with 20 players on.

Oh, just a heads up, the mission listed above is one piece we merge together (thank you BIS for allowing the merging of missions).

Anywho I will try to follow your suggestions sickboy and return my findings. Again, I believe it has more to do with humans connecting that the ancillary factors such as the mission or AI.

Also, your comments about not being able to spread the load between servers is false. I constantly see spikes where the load shifts from server to server. The spikes are not uniform so it is not just transferring 100% to the various processors. However it is not smooth either so it isn’t delegating the load very well. I know it sounds weird, but that is what I see ( look at that damn task manager all day long ).

Finally When I did set the affinity to one processor I moved all of the other applications to the other cores. It made no difference, the baseline for my server is a constant 4% without my ArmA servers running so divide that by 4 and it isn’t using that much of one core anyway.

Shotup: It depends on what you mean:

1. If you mean when I am running army with the affinity set to 1 core, it bumps my other cores to 2-4%.

2. If you mean when I do not have the affinity set, it fluxuates quite a lot. Usually there is one core that takes the brunt of the processing power and the other ones fluxuate quite heavily.

3. I you mean outside of battledays when we have our three servers running (main pub server, and 2 army training servers), then it varies to how many people are connected but the baseline is 4% processor and about a gig of ram being used idling.

I would definately reccomend a multicore server if you plan on running several smaller servers. However the purpose of this thread is to figure out if multicore servers are best for large scaled combat simulations.

[aus_twisted 0]

Then you could also make a coop map with let's say:

45 player slots

then different versions with added:

25 ai

50 ai

100 ai

200 ai

(just placed in the editor doing nothing)

Having AI doing nothing wont cause much FPS loss on the server, it's when you give them commands to move somewhere then the AI group leaders give orders to there solders to do this and that which causes problems.

Having 100's of dead units all over the island like BIS's CTI makes things far worse aswell, I cant believe BIS have not added a delete dead unit script after 5 - 10 mins on that CTI mission. It's pretty much useless at the moment.

[deady 0]

It just goes to show how difficult efficient scripting is in OFP/Arma.

If BIS struggle, feel for the community modders