Dedicated server performance

[maddogx 13]

Okay, I thought it would be best to PM Suma about this, and he has just confirmed that a "simulation cycle" as mentioned on the Biki is, in fact, a "frame" on the server.

Am I correct in interpreting "simulation cycle" as "frame"? Meaning that if MaxMsgSend=128 and the server FPS is at 20, the server will send out a maximum 20*128=2560 messages per second?

Exactly. The above is correct.

In other words, the server framerate must be taken into account when calculating maximum bandwidth:

"maximum bandwidth usage in bytes per second" = MaxMsgSend * Framerate * MsgSize

If we plug in 1024 for MaxMsgSend, 40 for the framerate (fairly conservative), and 1300 for both MsgSize values:

1024 * 40 * 1300 = 53248000 bytes = ~50.78 MB per second.

Too much for a 100MBit server, as 100Mbits = 12.5 MB/sec. (Though in practise it's more like 11 MB/sec.)

All things considered, it seems like the default MaxMsgSend of 128 isn't so bad after all.

[.kju 3244]

Well done MDX. Can you guys please merge the info into the BIKI page:

http://community.bistudio.com/wiki?title=basic.cfg

[friznit2 350]

Interesting read and will definitely have a go tweaking some of these settings.

VCB and 16AA just attempted a joint op on our server with 65 people. Needless to say it wasn't so much unplayable, as completely unplayed - we never got beyond the load screen. We were running several popular scripts (ACE, ACRE, UPSMON) and we had TS running on the same dedicated box.

We eventually gave up and restarted with just 25 of us. After a several shaky minutes at the start it calmed down and despite a low server framerate (4) it was very playable.

Given that PvP clans regularly play with those numbers and other coop groups have reportedly played with 70+, I'd be very interested to hear any other tips on how to make this work.

I don't suppose there's anyway to test multiple clients without imposing on 70 people to spend 2 hours watching a server pile in?

Edited January 30, 2012 by friznit2

[maddogx 13]

Well done MDX. Can you guys please merge the info into the BIKI page:

http://community.bistudio.com/wiki?title=basic.cfg

Okay, I've added some info on bandwidth optimization: here.

I've tried to keep it simple with some examples, but it can definitely use some polishing.

[.kju 3244]

Don't forget that AI means additional traffic AND CPU use - BOTH for the server AND the clients.

[Sibolo 10]

How does the number of people connected to the server come into this calculations about bandwidth?

The server generates packets and sends them to people connected. Everyone receives different packets regarding his situation update from the server.

We have said that if I put MaxMsgSend to 100 then the server can generate up to 100 packets for each "simulation cycle".

If there's one person on the server than he will be receiving up to 100x50=5000 packets per second at 50 server FPS.

Two people will receive 50x50=2500 packets per second at 50 server FPS.

...

50 people will receive 2x50=100 packets per second at 50 server FPS.

100 people will receive 1x50=50 packets per second at 50 server FPS.

Is this right?

[maddogx 13]

I still have some questions regarding the sending of packets from the server. Hopefully a dev will stumble across this thread and be able to answer some of these:

1. Assuming a server has MaxMsgSend set to 64, what would happen if there were 128 clients connected? Obviously the server would only be able to send a packet to 64 clients on every simulation cycle, but what would the actual consequenses be? A 64 player limit, with the rest dropping because of no data? Or will every player recieve a packet, but only on every second sim cycle?

2. Is it sensible to set MaxBandwidth to the exact theoretical bandwidth available (e.g. for 100MBit, MaxBandwidth = 100000000 104857600)? The example config on the Biki shows a much higher value (10Gbit), but this is not explained.

3. What is the approximate average ratio of guaranteed to non-guaranteed packets during normal server usage?

4. Since the default MaxSizeNonguaranteed value as seen on the Biki is lower than the MaxSizeGuaranteed value (256 and 512, respectively), does it make sense to keep the size of non-guaranteed packets lower? If so, why?

:)

Edited January 30, 2012 by MadDogX

[jedra 11]

I still have some questions regarding the sending of packets from the server. Hopefully a dev will stumble across this thread and be able to answer some of these:

1. Assuming a server has MaxMsgSend set to 64, what would happen if there were 128 clients connected? Obviously the server would only be able to send a packet to 64 clients on every simulation cycle, but what would the actual consequenses be? A 64 player limit, with the rest dropping because of no data? Or will every player recieve a packet, but only on every second sim cycle?

2. Is it sensible to set MaxBandwidth to the exact theoretical bandwidth available (e.g. for 100MBit, MaxBandwidth = 100000000)? The example config on the Biki shows a much higher value (10Gbit), but this is not explained.

3. What is the approximate average ratio of guaranteed to non-guaranteed packets during normal server usage?

4. Since the default MaxSizeNonguaranteed value as seen on the Biki is lower than the MaxSizeGuaranteed value (256 and 512, respectively), does it make sense to keep the size of non-guaranteed packets lower? If so, why?

:)

Good questions. It would be good to see a few example configs based on various bandwidth setups. Say 20Mb, 50Mb and 100Mb. At the moment everything is explained in the Biki, but it is somewhat out of context. Like many, I remain confused!

[maddogx 13]

Good questions. It would be good to see a few example configs based on various bandwidth setups. Say 20Mb, 50Mb and 100Mb. At the moment everything is explained in the Biki, but it is somewhat out of context. Like many, I remain confused!

Same here. I'm currently trying to optimize a private server I've been running for online battles, and found that the material on this subject is rather lacking. There is also a lot of contradictory information floating around on the forums, as evidenced by the confusion regarding MaxMsgSend. At least that much seems to be cleared up now. :)

At this point, my 100Mbit server config would look like this:

MaxMsgSend = 256;

MaxSizeGuaranteed = 1300;

MaxSizeNonguaranteed = 1300;

MaxBandwidth = 104857600;

MinBandwidth = 26214400; // (Next question: what is an ideal factor for MinBandwidth? :D)

MinErrorToSend = 0.001;

MinErrorToSendNear = 0.01;

This is assuming an average server FPS of 30-35 under load on a heavy mission.

Of course it would take mass testing to nail the best values, but as friznit2 says, getting 70+ people to test such things with you is fairly difficult. :D

Fortunately I have far fewer people to worry about.

Edited January 30, 2012 by MadDogX

[suma 8]

1. Assuming a server has MaxMsgSend set to 64, what would happen if there were 128 clients connected? Obviously the server would only be able to send a packet to 64 clients on every simulation cycle, but what would the actual consequenses be? A 64 player limit, with the rest dropping because of no data? Or will every player recieve a packet, but only on every second sim cycle?

The value is per client.

2. Is it sensible to set MaxBandwidth to the exact theoretical bandwidth available (e.g. for 100MBit, MaxBandwidth = 100000000 104857600)? The example config on the Biki shows a much higher value (10Gbit), but this is not explained.

Yes, it is sensible. That said, you will not see any effect in most circumstances, as the bandwidth estimation is very unlikely to estimated the bandwidth higher than the real available bandwidth.

3. What is the approximate average ratio of guaranteed to non-guaranteed packets during normal server usage?

This heavily depends on a mission type and on mission activity. All position / object state updates are nonguaranteed. Guaranteed are publicVariable, boarding vehicle, killing the unit and similar. When there is no combat and no scripting, it is common that there are almost no guaranteed messages send.

4. Since the default MaxSizeNonguaranteed value as seen on the Biki is lower than the MaxSizeGuaranteed value (256 and 512, respectively), does it make sense to keep the size of non-guaranteed packets lower? If so, why?

The purpose of the max message size is to reduce the latency. The minimum possible latency of the message is size/bandwidth (the time needed to send the message). A 1300 B message sent over 256 kbps connection has a minimal latency of 1300/(256000/8) sec ~= 40 ms. The latency is more important for non-guaranteed messages (position/state updates) in general.

Next question: what is an ideal factor for MinBandwidth?

The ideal factor is: leave it at the default unless you really see something goes wrong. The built-in bandwidth estimation is built on a packet-pair probing QoS, which has shown to be very reliable on the internet as we know it today.

Edited January 30, 2012 by Suma
ideal factor for MinBandwidth

[sickboy 13]

Thanks for asking MDX, and thanks for sharing Suma.

Is anyone planning to update the BIKI accordingly?

[maddogx 13]

The value is per client.

Hold on, what? That's news to me.

In other words, if the MaxMsgSend value is set to 128 and there are 10 clients connected, then the server could still be sending out 1280 packets per cycle?

Is anyone planning to update the BIKI accordingly?

I intend to, yes, and have already started. :)

[friznit2 350]

It would be really helpful if we could get some real world example configs - perhaps from groups who successfully run ops with large turn outs (I'm looking at the likes of Shacktac and United Ops). For example, take an average Rented Dedicated Windows server on 100Mb b/w, running A.Coop missions with 70 clients and lots of AI; B.PvP with 70 people (No AI); C.Coop with <30 clients etc etc.

The Kelly's Heroes guide is the only place I recall seeing any real examples of a config and I'd love to hear from these people who run big ops regularly to see what we can learn from them (because simply put, I've tried with 60 clients and our server hates me even though it's a fairly new one).

[suma 8]

In other words, if the MaxMsgSend value is set to 128 and there are 10 clients connected, then the server could still be sending out 1280 packets per cycle?

Yes, subject to bandwidth limitations - understandably, you will never send more messages than the connection can transfer.

[gossamersolid 155]

Wow, have we all been setting values wrongly then?

I don't know if anybody has known that it was per client...

[visceralsyn 10]

For years, the dev's never answered any questions regarding the configuring the server network settings.

Information is power.

MadDog, still not sure how I came with that number either, since i was using the calculator too. But it doesn't matter now. And, if you assume anyone can type condescending, and you don't like it, Tuff-titty, I don't care what you assume.

...Syn...

[Dwarden 1125]

so

128 players with 128 maxMSgSend and 1300 bytes = 21 299 200 bytes / s = 20MB/s = 162.5 mbit /s

256 maxMsgSend = 40.625 MB/s = 325 mbit/s

ofcourse this should be throttled by values set in maxBandwith if lower

this is per frame (fps) on server and gets throttled by bandwith optimizations

Edited April 27, 2012 by Dwarden

[maddogx 13]

so

128 players with 128 maxMSgSend and 1300 bytes = 21 299 200 bytes / s = 20MB/s = 162.5 mbit /s

256 maxMsgSend = 40.625 MB/s = 325 mbit/s

ofcourse this should be throttled by values set in maxBandwith if lower

i'm i right or missed something else?

You missed the server fps. :)

It seems that MaxMsgSend is "packets per player per simulation cycle", so your examples above would be 20/40 MB per frame. Multiply with the server fps for MB per second. Quite high numbers there, which will never be reached due to bandwidth optimization. :o

So it seems that the MaxMsgSend value is mostly irrelevant in the grand scheme of things, unless you set it incredibly low for some odd reason. This is at least my current understanding.

[Dwarden 1125]

ou yes, forgot per frame ... silly me ... {where is emotion for dig into sand}

[On_Sabbatical 11]

Don't forget that AI means additional traffic AND CPU use - BOTH for the server AND the clients.

Please put this in a new locked topic,in capital letters and make it sticky !!!

Some server admins have problems understanding it.

[visceralsyn 10]

So it seems that the MaxMsgSend value is mostly irrelevant in the grand scheme of things, unless you set it incredibly low for some odd reason. This is at least my current understanding.

Actually, old titles like Descent used fairly low packets-per-second/client, like values as low as 6 to 14.

So I ran a test, dropped maxmsgsend to 64, loaded mountain warfare, dropped in 2 clients, left the AI in. Server fps dropped to mid to upper 30's. CPU usage was fairly high, 70% to 90%, but not as high as before.

Now running maxmsgsend at 32, server fps is now mid to upper 40's. CPU usage is stayin in the 60% to 80% range now.

...Syn...

[gossamersolid 155]

Please put this in a new locked topic,in capital letters and make it sticky !!!

Some server admins have problems understanding it.

Yep, people often forget that... it's really annoying.

Also when people say "Oh the unit is local to this client, means it only affects them"... WRONG!

[maddogx 13]

Actually, old titles like Descent used fairly low packets-per-second/client, like values as low as 6 to 14.

So I ran a test, dropped maxmsgsend to 64, loaded mountain warfare, dropped in 2 clients, left the AI in. Server fps dropped to mid to upper 30's. CPU usage was fairly high, 70% to 90%, but not as high as before.

Now running maxmsgsend at 32, server fps is now mid to upper 40's. CPU usage is stayin in the 60% to 80% range now.

...Syn...

I'm testing MaxMsgSend=8 at the moment. Looking good so far. :D

Even "1" worked in small missions. :eek:

[visceralsyn 10]

I've been running MaxMsgSend=32; for a little bit into the mission. I am seeing a little desync, like 1 to 8, randomly, once i spotted 200 something. but has gone away just as quickly as i saw it.

Are you seeing any desync, other than in the player-tab?

what I really like, is the CPU usage isn't pegged at 100% anymore, and the server FPS staying above 30, is just plain nice...

...Syn...

Edited January 30, 2012 by VisceralSyn
typos and other grammatical errors, as usual...

[Dwarden 1125]

tweak more to find the perfect balance (it again may shift depending on the mission script load, AI load, player load, connection quality, hardware quality and so on)