NetServer: trying to send too large msg

[VictorFarbau 0]

BETA SERVER 1.7.0.5157 (updated BETA 1.07)

I am regularly checking my "arma_server.RPT" file to scan for coding and generic errors.

I managed to get rid of all current script errors but my Mega-Coop-Dynamic War mission suffers from Async issues; meaning that after playing for about 20-30min it can happen to any player that he gets killed by invisble enemies / he doesn't see any enemies anymore / he cannot kill any enemies.

So I assigned markers to enemy units to trace them. After a while I basically saw a marker walking by right in front of me but I couldn't see any man or vehicle there. Next thing was that I got shot (just fell, didn't see or hear anything).

Since I am testing these things on a dedicated server in my local LAN first I would exclude bandwidth issues (100mbit local).

I have studied the included "DS-Admin.rtf" which talks about optimizing performance. There it says to tune paramters in the "Flashpoint.cfg" file. Well, since we rather talk about the Arma.cfg file I added the following lines to the Arma.cfg as well as to the server specific "sample.cfg" file:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

MaxMsgSend = 256;

MinBandwidth = 512000;

MaxBandwidth = 768000;

MaxSizeNonguaranteed = 4096;

And still my arma_server.RPT records hundreds of lines with the identical text:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

NetServer: trying to send too large non-guaranteed message (3291 bytes long)

Message not sent - error 0, message ID = ffffffff, to 1884877695 (Victor)

The "non-guaranteed message" size varies between 3000 - 3800 bytes. Can anybody recommend a better setting? Seems like setting these variables didn't make any difference at all - same errors before (using default) and afterwards.

This causes Async issues all over the place. Positions not being updated, people not getting in or out of vehicles, the invisible death, player's head getting stuck in free-look mode etcpp.

So I just did a "monitor" on the server and it shows "Outgoing: 2422kbps, Incoming 181kbps" on average. Isn't that a bit much for a simple mission with a few trucks, 2 - 3 tanks, 4 Helos and about 120 soldiers? I remember we played this mission often w/o any problems lately. This started very recently all of a sudden (maybe after updating to 1.07b-b?).

Puzzled,

Victor

[whisper 0]

Ethernet MTU 1500, with 20 IP headers, I would say keep your messages under 1480 bytes

Now, looking to the description of the parameter :

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">MaxSizeGuaranteed=<limit>;

Maximum size of guaranteed packet in bytes (without headers). Small messages

are packed to larger frames. Guaranteed messages are used for non-repetitive

events like shooting.

Default: 512

MaxSizeNonguaranteed=<limit>;

Maximum size of non-guaranteed packet in bytes (without headers).

Non-guaranteed messages are used for repetitive updates like soldier or

vehicle position. Increasing this value may improve bandwidth requirement,

but it may increase lag.

Default: 256

I think it is used by the engine to pack information together in a single packet to lower bandwidth requirements, but it makes latency higher (information waits more for packet to be filled before sent), so my guess is that putting it high is perhaps not helping if you have bandwidth enough.

Moreover in your case your over network MTU, that means fragmentation, never really good.

[shinRaiden 0]

The issue isn't with the server tuning, the issue is with your scripting. You're trying to slam too much info around, which is exceeding the buffer sizes. Look through your code to see how you can streamline it.

[VictorFarbau 0]

@Whisper: you're relating this to the Ethernet MTU? This is a quite courageous assumption I guess. Your assumption about latency would in fact imply that the server itself would create the packets to be sent. But I highly doubt that; unless the server uses a proprietary protocol it should be up to OSI layer 4 (Transport) to manage that.

Nonetheless I'll try this of course with an MsgSize of 1400

@shinRaiden: what can I do with a statement such as "the issue is with your scripting"? I cannot actively influence the message size or amount of network traffic unless I limit everything down to 10 units. As said before; it did really work before (up to 1.05) and my mission never changed. It is also said that ArmA has been particularly optimized to allow "massive" MP environments. And up until recently this really seemed to be true for me.

Ah well, I will keep trying.

Victor

[shinRaiden 0]

If you're trying to send an update that contains info on all the content in a session, of course there's going to be problems as you've indicated. You're going to need to look at re-coding your message handling system to send more smaller messages.

[whisper 0]

Moreover, if you reread the description : You want to make this parameter high if you have bandwidth issues, knowing that it will increase lag.

So why do you try to force it as high as possible?

[suma 8]

Ethernet MTU 1500, with 20 IP headers, I would say keep your messages under 1480 bytes

I think it is used by the engine to pack information together in a single packet to lower bandwidth requirements, but it makes latency higher (information waits more for packet to be filled before sent), so my guess is that putting it high is perhaps not helping if you have bandwidth enough.

Moreover in your case your over network MTU, that means fragmentation, never really good.

Your explanation is absolutely correct. One should not set this value that high that a resulting packing would be larger than a PPoE MTU (1490). From MTU you need to subtract IP + UDP header (28B) + our internal messaging overhead (24B), and you get 1438 as a maximum allowed message data size.

I see no reason to set any of those two values to anything larger than 1024 - the lag introduced is probably much worse than the bandwidth saved.

[sickboy 13]

I see no reason to set any of those two values to anything larger than 1024 - the lag introduced is probably much worse than the bandwidth saved.

Heya Suma,

Could we get these settings more properly explained inside the BIKI item for dedicated servers? Maybe with some thoughts, some theories, some more examples etc? That might help ppl in the future understanding more easily and quickly without having to "Try" and "Figure out", especially in case of server settings it's not always easy to do tests to figure it all out

[Rambo-16AAB 0]

Could use some nice brainy person to wright a little calculator script so you can enter the Upload/download speed of your server and it would give a recommended setting for you. it would make figuring stuff like this out a lot easier.

[VictorFarbau 0]

Thanks for your thoughts guys. There could be one misunderstanding. When I talk about "messages" to be sent then I don't mean any custom made messages. It is the engine-controlled server-client messaging that has problems. So in that sense I really have little influence on size or shape of these. They just transport game information.

One way would be to just decrease the amount of data being shoved around by limiting the mission. A pity but maybe the only solution long term.

@whisper: "high if you have bandwidth issues, knowing that it will increase lag."

Hm, right, that makes sense. I don't have bandwidth issues so I should decrease msg size. So I did to 1400 and voila, much better already

A lot of msg size errors still in the log (less than before, though) but a lot less Async issues in the game.

In general I will look at methods to reduce traffic and keep the msg size down.

Cheers mates,

Victor

[VictorFarbau 0]

@Suma:

Thanks for confirming the MTU influence, I would have never crossed my mind. But you know the code...

What I don't understand is:

I have set the parameter

"MaxSizeNonguaranteed=1400"

now and still the server logs errors such as

"NetServer: trying to send too large non-guaranteed message (2861 bytes long)"

So why does the server not just limit the message size as specifed and send 2 messages instead? I just fail to see where this is my fault if you know what I mean

If everything gets slow then ok, I'll look at my errors. But don't complain to me that the engine tries to send messages with invalid sizes.

Not bitching here; but I'd just like to be able to work on my mistakes. Looks like I can't really complete fix this one all by myself.

Regards,

Victor

[sickboy 13]

Victor, you made the changes inside the ArmA.cfg yes?

Can you check if it's the ArmA.cfg that the server really is using by trying to move the file away or save changed content when it's running?

Still wouldn't explain it though as the standard setting is 512, while your message is still about a size way above that

[VictorFarbau 0]

Howdy Sickboy.

True, I added these lines to the Arma.cfg file. And they did make a noticeable difference in the game so I assume the game uses that file. Will verify tonight.

>Still wouldn't explain it though as the standard setting is 512,

Yes, exactly. I only added these lines after I had problems, not before. Seems like the default 512 is too low; my test with 4096 makes matters even worse and 1400 seams to be reasonbly ok.

To me the question remains as to why the netcode engine doesn't chop data messages into pieces with a maximum size when there is such a parameter in the first place.

Since there's no effective control or feedback to the user (e.g. message on screen saying "Bandwidth overload") this will mostly go unnoticed. The folks I play with have certainly never opened their Arma.RPT files ever. And why would they?

Risk is that people will just complain how badly ArmA performs yadda yadda. Another area of improvement?

Regards,

Victor

[whisper 0]

I think otherwise : if you know you won't have bandwidth issues (like a 100Mb/s access to internet), then put it as low as possible, with a minimum equal to the maximum single message size sent by the game (there we need Suma's input).

Try 256, but in fact we really need to know first what is the maximum size a single guaranteed and non-guaranteed packet can take.

EDIT : after thought, if set too low, is it possible that server can overload a client downstream? Assuming ADSL 128k is todays minimum standard

[VictorFarbau 0]

@Whisper:

I am using a separate computer to host my dedicated server and test and play missions. When my buddies connect they do so through my DSL line so it's only me with a 100Mbit connection. So I guess it would be best to play with a couple of performance parameters to get the best results for all parties.

But what's the point of limiting it down to 128 byte e.g. when the netcode can't work with these parameters and complains about failed messages due to huge size anyway?

So I am not sure how to systematically resolve this if I can't trust the netcode in that sense.

Regards,

Victor

[VictorFarbau 0]

I tried these parameters now:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">MaxMsgSend = 256;

MaxSizeNonguaranteed = 256;

MinBandwidth = 512000;

MaxBandwidth = 768000;

Result: NO MORE error entries in the server log but terrible sync issues again, team members not accepting commands anymore, running on the same spot for minutes and every single respawn ends in the irreversible "head freelook" mode. Very frustrating. I'll try to reduce the MaxMsgSend now again and if that doesn't work I'll revert to

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">MaxMsgSend = 128;

MaxSizeNonguaranteed = 1024;

MinBandwidth = 512000;

MaxBandwidth = 768000;

Sigh - and that on a local server Maybe I should take out the Bandwidth paramters - but those I left in to simulate the limitations of my DSL line. Only I would expect no lag at all with these when the server ping is almost 0.

Victor

[whisper 0]

@Whisper:

But what's the point of limiting it down to 128 byte e.g. when the netcode can't work with these parameters and complains about failed messages due to huge size anyway?

Regards,

Victor

The net engine complains you try to send 3000+ bytes packets, that is all.

See Suma's answer, you're problem is not the amount of information you try to send (or if it is, changing the parameter will not solve anything and you will have to follow shinRaiden advice, because this means some of your script is constantly overloading the net engine), but by the size of the packet you try to send.

It should work along these lines :

The engine tries to send information messages. These messages are piece of information that should take somewhere between a few bytes and ... well, let's say 128 bytes.

These messages are then sent to the net engine, which takes several messages and pack them together into a single packet up to the point where the packet reaches the maxSize(Non)Guaranteed parameter. The net engine then handles the sending of the packet.

Why packing information messages together? To gain bandwidth, because each packet information has a "ArmA header" (24 bytes) added to it before being sent. If you send 1 packet per 128 bytes information message, you will in fact send 128 + 24 = 152 bytes, losing 19% of bandwidth for usefull information (the header is not informational). If you have a maxSize of 1024, you can pack 8 128 bytes information packets inside, with only 1 24 bytes header, ie only losing 2.3% of bandwidth for usefull information. That's quite an improvement.

OTOH, the engine has to wait for the 1024 bytes packet to be filled before sending it, meaning the 1st information message inside this packet will wait longer before being sent. Information will arrive to client in burst and the oldest information in the burst will be quite old, meaning desync and lag appearing.

So it's a matter of finding the correct fine tuning between low size and high size to meet your bandwidth requirement and accetable lag.

But your original issue was even worse : after the ArmA packet was formed, it was too big for your "physical" network. Like Suma stated, typical ADSL accepts 1462 bytes maximum per single UDP packet. If you try to send packets above this size, the net engine will, on top of the work described above, need to fragment the packet into smaller packets fitting inside the 1462 bytes limit, provided the engine can do fragmentation, which is not even sure. In case it cannot, you will have error messages (so maybe the issue originally was that), and information will wait even more for resending over the network.

[shinRaiden 0]

I'm surprised that none of the CoC guys have chimed in here, they're the ones with the most historical baggage in the circus of poking at the message sizes.

What I strongly suspect, based on your prior posts about this being a "Mega-Coop-Dynamic War mission", is that you're passing some data somewhere that's resulting in a relatively large amount of data being transfered. Whether it's an overload of your data, or if ArmA data is being tacked onto it I don't know and I'm too busy to do a code audit anyway.

What you need to do is look at the sizes of dataset's you're working with. Mother-of-all Arrays slammed around can cause you problems. Or you can get size creep in plenty of other places. Look through your code and see where your largest chunks of logic are, and see how they can be cleaned up, whether you change to incremental element updates, working locally with full arrays while syncing with indexes, that kind of stuff.

[VictorFarbau 0]

@Whisper: thanks for your extensive reply. I got it now. Funny thing is that normally I am at proficient level in networking myself somehow but up until now I failed to see the close relation between protocol and the ArmA netcode. I guess I am too much into Application design recently. I was blind, now I can see

@ShinRaiden: thanks for the tips, I shall do that.

Quote[/b] ]working locally with full arrays while syncing with indexes

I use that for example for my MP Intercom system, just tokens and indexes being sent around to exchange messages. But the traffic caused by massive presence of moving enemies at times is beyond my influence. Here I have to trust Arma's netcode to just get it done.

As said, network lag is ok, it would be obvious when the amount of objects gets out of hand and it would get better again when less objects are present. But in my case it broke the complete mission. So dropping network messages completely because they are "too big" is something the ArmA netcode programmers must resolve, not my fault. Imagine TCP or UDP would throw these errors all the time because some stuff you try to download would be too big or something. A bit absurd.

Worst case I have to artificially limit the amount of objects in my mission. Pity but ok.

Regards,

Victor