JIP Lag

[deady 0]

So, every server out there suffers JIP lag at some point or other.  I'm talking about when a player connects, everyone gets a red chain and crazy desync.  This is mostly seen in complex missions such as Evo, and usually gets worse the longer the session has been playing.

The issue doesn't appear to be with the server's hardware, since they are not maxing out their CPU use or available bandwidth - it appears to be some kind of internal congestion in the dedicated server coding, or possibly congestion caused by laggy clients.

I imagine the JIP lag is caused by the sheer number of updates needed from the server - every felled tree, signpost, soldier position, vehicle position etc. But why does that cause desync for others? Theres plenty of bandwidth available, and spare CPU power - but it doesn't seem to use whats available.

I'd love to see BIS tackle this problem - perhaps setup their own server, stick evo on and run some kind of debug arma server so they can collect data on what exactly is causing desync and why.

[The_Captain 0]

As far as I know, "desync" is caused by a backlog of packets sent to the clients: the clients can't receive the current updates because they're too busy trying to receive/acknowledge the old ones.

off the top of my head, I can think of two causes for the JIP desync:

1) A bunch of publicvariables that are sent by mission scripts when a player joins. Publicvariables are sent to everyone, so it's possible everyone would get desync, especially in a complex mission. (They can't keep up with processing all the PV's *and* the midgame events). I think this occurs when the client has already joined the mission (after receiving data, when init.sqf or whatever is run, or perhaps when the server executes the onplayerconnected code.)

2) The client that's connecting trying to find out the position of all objects, dead soldiers, felled trees, etc etc etc that have happened since mission start. This I think occurs during "recieving data".

I'm wondering whether this JIP lag happens when the client is "connecting", or when they have already joined and trigger the mssion to send out the info that needs to be synchronized: setvehicleinits, publicvariables, etc.

Something to test...

[RN Escobar 0]

from my experience the server app will run with as much resources as are available, especially on the bigger more complicated maps.

was the same in OFP, ram usage it very dependant on server mods running and the type of map/number of units etc etc

when u say u have spare CPU power available, how much are u talking about, dual/quad core CPU?, process locked to one core or left to float in OS threading hell?

the various servers we run pretty much max out whatever core they are assigned to, especially with the EVO and other big COOP maps.

Esco

[sickboy 13]

Few things that can reduce this lag:

[*] Do not execute publicVariable when players join. Because you will be resending the same data over the network which is unnececairy because arma syncs the publicvariables to jip players on connect anyway

[*] Use a low amount of publicVariables, if possible try to reuse the same variables whenever possible

[*] If you are sending code over in a string, or any big publicvars, then set it to "" after a while so there is no useless data sent to jip players when connecting: some_pv=""; publicVariable "some_pv";  I asked Suma for making it possible to really Nil the variables so they aren't sent at all, not even the empty strings. But for now, better send empty strings than useless long vars to jippers.

[*] Do not use setVehicleInit for executing things on other computers, unless its totally nececairy that such code is ran when players JIP. I was using setVehicleInit for my network services, but vehicleinits aren't overwritten, they just add on top of eachother, and even vehicleinits of dead units/vehicles aswell as deleted units/vehicles are still sent to players on JIP. The publicvariables you can set back to empty strings, but you can not manipulate the vehicleInits after you executed them, so my advice is to stay away from vehicleinits unless you really really have to.

[deady 0]

Is a big PublicVar necessarily worse than a small publicVar? I was under the impression it was the overheads and handshaking involved with a PV that caused the congestion rather than its actual contents. But I have very little knowledge on how they actually work.

Also it's interesting that you say PVs are automatically sent to new connectors. You mean once something like publicvariable "blah" is done, and then 10 minutes later someone connects, without any new commands the new player gets "blah"? Who sends it? The server?

[zyklone 1]

Also it's interesting that you say PVs are automatically sent to new connectors. You mean once something like publicvariable "blah" is done, and then 10 minutes later someone connects, without any new commands the new player gets "blah"? Who sends it? The server?

Yeah, that's a very interesting statement.

It contradicts everything I saw in the early arma versions where publicVariables where not synchronized in any way upon JIP.

[Dwarden 1125]

add to sickboys list

"increase server setting for bw per player"

"get very! fast CPU/ram (3GHz C2D+)"

[whisper 0]

Also it's interesting that you say PVs are automatically sent to new connectors. You mean once something like publicvariable "blah" is done, and then 10 minutes later someone connects, without any new commands the new player gets "blah"? Who sends it? The server?

Yeah, that's a very interesting statement.

It contradicts everything I saw in the early arma versions where publicVariables where not synchronized in any way upon JIP.

Same, I'd really like some clarifications there, Sickboy

[The_Captain 0]

Quote[/b] ][*] arma syncs the publicvariables to jip players on connect anyway

This is interesting. I didn't know this. However, I'm packing all my publicvariables into a single string which is sent in short intervals, which should reduce the amount of handshaking. Is this solution preferable to having lots of publics, even though the string is large?

Quote[/b] ][*] I was using setVehicleInit for my network services, but vehicleinits aren't overwritten, they just add on top of eachother, and even vehicleinits of dead units/vehicles aswell as deleted units/vehicles are still sent to players on JIP.

What about clearVehicleInit? It seems that this might clear an init, but I haven't tested it myself. It was added in 1.08, and looks like it solves the problem you outlined.

Edit: oops, looks like clearVehicleInit only removes pending commands, not executed commands...

[rundll.exe 12]

the example suggests it otherwise tho.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">tank1 setVehicleInit "hint 'hi all'"; processInitCommands; clearVehicleInit tank1

[ryankaplan 1]

biki bug report: not many people seemed interested tho.

Bug Report: JIP Lags

get voting

[Doolittle 0]

That's a big statement to make, sickboy! But stuff like that is good to know! The other day I noticed param1 and param2 aren't synced with JIP people! So I publicVariable them when people connect.

Like The_Captain said, I thought clearVehicleInit removed any inits set for anyone joining in the future.

I've noticed deleteVehicle doesn't seem to remove an object from the game.........! Meaning it will be sent to JIP when they join!

Maybe we should be using these commands?? isMarkedForCollection, deleteCollection, and addVehicle (move vehicle to client's locality?)

Like... deleteCollection after we deleteVehicle?

[The_Captain 0]

I haven't looked at deletevehicle, but I have used deletecollection for my infantry-obliteration script. When people get a high velocity when damaged (ie flying through air), I deletecollection them to remove them from play and show a blood effect, but don't deletevehicle them in case they're a player (and need to respawn before I can do that, since deletevehicling a player crashes arma). It seems to work properly over MP, removing a unit from everyone's visibility... well... then again I execute it locally...

I'm rambling but I should test this one out.

Doolittle, what happens when you deletevehicle, in terms of connecting clients? Do they still see an object even though the server deleted it? Or are they sent position/location information though the object is removed? Time to test...

I wonder if a vehicle which has been deletecollectioned takes up less bandwidth when it is sent over the server...

[sickboy 13]

You got to be kidding me! I was fully unaware of the clearVehicleInits command! You guys are my heroes Thanks for pointing to it

PublicVars are indeed synced at JIP.

I myself switched from vehicleinit to publicvar because I was able to clear the vars and rePv them so there wouldnt be 100s/1000s of lines of code sent to JIPpers (as every single vehicleInit is synced to JIPpers). I am unsure myself as to what's worse, many lines of codes or the handshakes per PV, I havent tested it.

[Doolittle 0]

The_Captain, I was creating an invisible house, damaging, then deleting it to get a global explosion effect for everyone. I noticed when I did a JIP to the mission you would hear BANG BANG BANG 30 or so times... however many that object had been created and deleted. This showed me even though an object has been sent to deleteVehicle, it is still sent to JIP people!!! They just turn it's state into deleted once they connect.

I solved that issue by holding onto my invisible house and just damaging it, healing it, damaging, healing... so that I could make my explosions but just do it with one object. This was my simple solution/hack to avoid having a lot of junk transmitted to JIP people. But maybe if you deleteCollection an object as well as deleteVehicleing it... then it isn't sent to JIP?? If that is true then that is a very good thing to know!!!

I can understand why you would still want a vehicle around even though it was deleted... what BIS might have been thinking. Say you have a car named "car" in the mission on map start, and it has an init of "foobar = true".... then in init.sqf you deleteVehicle car. You would still want every client to get the foobar = true command. ....so if JIP people still do get that command... then I guess the object wasn't deleted? Or maybe objects on map start vs. objects created during game play are treated differently?

I wish Suma would talk to us more like he used to a few years back.

[Doolittle 0]

Are you sure setVehicleInit stacks the inits?!?!

I have this in my onPlayerConn.sqf:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">timelogic setVehicleInit format ["setDate %1", date];

processInitCommands;..so that whenever someone connects, their time is synced with the server time. But this would mean... assuming 50 people logged on before you.. that it would need to transmit 50 init lines? So I should always always clearVehicleInit before reassigning a setVehicleInit to an object??

Doolittle

[.kju 3242]

What about trying to get info by BI here? (PM to Suma)

[sickboy 13]

Are you sure setVehicleInit stacks the inits?!?!

I have this in my onPlayerConn.sqf:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">timelogic setVehicleInit format ["setDate %1", date];

processInitCommands;

..so that whenever someone connects, their time is synced with the server time. But this would mean... assuming 50 people logged on before you.. that it would need to transmit 50 init lines? So I should always always clearVehicleInit before reassigning a setVehicleInit to an object??

Doolittle

100% correct.

Try it:

test="LOGIC" createVehicle [0,0,0];

test setVehicleInit "test2=0";

processInitCommands;

sleep 5;

test setVehicleInit "test3=0";

processInitCommands;

sleep 5;

test setVehicleInit "test4=0";

processInitCommands;

You will find that test2, test3 and test4 all are 0, on every client, also JIP.

It was my main reason to switch to publicVariable to create a Network Service alike CoC's. Now that I found about the clearVehicleInits im going to switch back to my earlier system (But revised) for sure

[cross 1]

Edit: oops, looks like clearVehicleInit only removes pending commands, not executed commands...

@Sickboy...captain edited his post. FYI.

[sickboy 13]

Edit: oops, looks like clearVehicleInit only removes pending commands, not executed commands...

@Sickboy...captain edited his post. FYI.

Thanks a bunch! Well that totally sucks....

Going to contact Suma about this.

[Doolittle 0]

Yeah but how do you define "pending"?? For anyone who's going to join the server in the future it's a pending command waiting for them to init when they connect.

That is how I interpreted pending.

EDIT: sickboy plz post if he responds.. thanks!

[UNN 0]

Quote[/b] ]Yeah but how do you define "pending"?? For anyone who's going to join the server in the future it's a pending command waiting for them to init when they connect.

That is how I interpreted pending.

As per my post on the wiki;

clearVehicleInit, clears the pending init commands, not those already processed. Once the init commands are processed, they can not be removed. Even if the vehicle is killed or deleted.

In this case, pending means waiting for a call by ProcessInitCommands.

Quote[/b] ]You will find that test2, test3 and test4 all are 0, on every client, also JIP.

It was my main reason to switch to publicVariable to create a Network Service alike CoC's.

There is nothing wrong with the way this works in Arma. ProcessInitCommands and JIP on there own, do not cause desynch in MP. The scripts called during JIP can cause lag, so you have to be careful. I processed 30,000 simple JIP's commands as a test, the only noticeable difference was a delay of 3 to 4 seconds, before the mission started.

But stacked init commands are something you have to take into account, it's inevitable during a mission that some of the init commands will become redundant. But you can avoid that with a few checks at the start of your JIP script.

If you want to avoid calling scripts during JIP on units that have been deleted add this at the very start of your script:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">If (IsNull (_This Select 0)) ExitWith {}

If you want to avoid calling scripts during JIP on dead units as well:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">If (IsNull (_This Select 0)) ExitWith {}

If !(Alive (_This Select 0)) ExitWith {}

If you start calling scripts with ProcessInitCommands that aren't linked to objects, then things get a little more complicated. But there are plenty of ways of handling that.

[Doolittle 0]

Okay. I'm still shocked init commands are run on objects that have been sent to deleteVehicle!

From Suma:

Quote[/b] ]DeleteCollection is related to "collections", which was work in progress for buildings for Game 2, and I doubt the function does anything really useful, as it is unfinished and untested.

[Doolittle 0]

So.... how does the game know which variables are publicVariables and should be sent to everyone when they join?

Do you have to publicVariable something once for it to get on the "list" of stuff to be sent to client?

[UNN 0]

Quote[/b] ]Okay. I'm still shocked init commands are run on objects that have been sent to deleteVehicle!

I was to at first, but after using the command for a while, it turns out to be really handy.

Imagine you have a truck that drives around deploying, say machine gun positions, using SetVehicleInit. Each of these machine gun position runs a script on all the clients once it's deployed. If that truck gets destroyed and eventually deleted as part of some cleanup script. You would still want each of the machine gun positions to be deployed and their scripts run, for anyone who JIPs.

But it does complicate things. The opposite example would be a truck that drives around deploying a single command position, at various locations during a mission. If anyone JIPs, you would only want to deploy the command position at it's last location, ignoring any of the previous locations. In that case you could use the Time command:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">_StrInit=Format ["[This,%1] ExecVM ""DeployCom.sqs""",Time];

SetVehicleInit _StrInit;

MYTRUCK ProcessInitCommands;

In DeployCom.sqs you would have this check:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">_StartTime=_This Select 1;

If (IsNil "UNLOADTIME") Then

       {

       UNLOADTIME=_StartTime;

       }

       Else

       {

       If (_StartTime>UNLOADTIME) Then

               {

               UNLOADTIME=_StartTime;

               };

       };

Sleep 0.001;

If (_StartTime<UNLOADTIME) ExitWith {};

UNLOADTIME=Nil;

Quote[/b] ]Do you have to publicVariable something once for it to get on the "list" of stuff to be sent to client?

Looks that way, also if you then set the variable to nil, then publicvariable it again. It will still be broadcast to the JIP with it's original value.

So far I've managed to almost avoid using PublicVariable. I had to use it once to coordinate the same script across client and server. Other than that, ProcessInitCommands has covered everything else.