Serious Bug: JIP-LAGs (+ possible easy-fix idea)

[mr.g-c 6]

I don't know why this isn't fixed since the first patches, but somehow i really feel that the vital part of the game (the MP part) has suffered way to long under this serious Bug, which could be fixed probably very very easy.

Problem Description with a few examples:

Situation A:

Person A, Person B, Person C, Person D & Person E are playing a MP Game on a dedicated server. Persons A & B having both around 2-10 custom sounds and custom faces loaded.

The Server-Limit for custom sounds and faces is set at 50kb. There is no signature checking, nor is BE enabled. The game ran already for 30 minutes and some cars were abandoned as well as trees cut and stuff.

Person X connects during this running game (JIP - Join in Progress) with some custom files and a custom face, he also don't have the actual played mission on his PC.

Now what happens is this:

Person X connects and needs to up/download the all the custom sounds and faces from himself and the other connected players. This leads that the server seems to stop sending actual data between the already playing players and seems to concentrating completely on the new JIP-Player. I have monitored this during play with various tools on both the server and me (the client) and really i received no data from the server anymore, IE it dropped from 5kb/s to 150bytes/sec.

Persons A - E gets huge Desync (the more custom files the longer), and a yellow or red chain appears. This is already bad enough as it can lead to a fcked-up game with kills and such but it gets even way worse.

Now Person X has finally reached the "Side-Selection Screen" and after he chosen his preferred side & position, he clicks "ready" which lead to a download of the missing mission - and guess what? Yes it produces again a terrible LAG (Yellow -> Red chain, huge Desync, people seems to standing at position with walking animations and such strange things) for Persons A - E, for the amount of time needed to download the mission for Person X.

Now Person X finally downloaded the mission and Arma starts to loading the needed Addons/Island and stuff and while it does this (Player X sees the "Loading Screen") it starts to download the actual Game Progress in the background (some time you see it as "Receiving Data" when it takes longer than the actual Mission/Island load) and Guess what again? RIGHT it produced the same LAG again for Persons A - E again. Its actually that the LAG can be like Minutes Long, leading from just Desync around 100- 3000 (yellow chain) or even much worse (red chain).

This was the basic JIP-Bug and its caused because the dedicated Server is bad programmed and can only do one thing at once as it seems.

Now it Gets more interesting and even way worse for the players.....

Situation B:

EXACT Same conditions as in Situation A, Except that the mission itself is like 2.5mb large in size and that Signature-checking is ENABLED.

So what happens differently to Situation A when Person X connects is, that some of Persons A - E can get kicked because of "Signature Checking Timeout" and that is really really bad. Again the signature-checking timeout happens because the Dedicated Server seems to stop transferring messages between the already playing players (in this case Persons A - E) and therefor logically, the signature checking gets timed out.

I have monitored this now for over a month in many many tests (we have always above 20 players on our server in prime-times) on our own server, as well as a client on any other dedicated-server. And people get really often kicked because of these LAGs which are leading to signature checking timeouts.

Situation C:

Exact the same conditions as in Situation A & B, except that now BattlEye is enabled.

What happens differently to Situations A & B is, when Person X connects and downloads either "customfiles", "mission-file" or "gameprogress" and therefor creating LAG & Desync for Persons A- E, is that Persons A - E will get most likely kicked from the Server because of "Signature Checking Timeout" AND "BattlEye Client not Responding" timeouts!

Its basically the same reason as for Situation A & B - the server stop to transferring data between the already connected clients and as i said i monitorred that whenever some JIP joins, my PC reaching data drops tremendously , from 2kb/s - 20kb/s down to ~120bytes/s.

So it is a TERRIBLE Bug, every People in MP knows it and it happens in its full hardness WHEN EVER people joining during a running Game (JIP).

EDIT: I forgot to tell you about some Super-GAU in JIP as a worst case scenario:

I noticed this 2-3 times when i was playing on a server with a brand new mission and 5 people nearly instantly connected to the server, all probably had to download all this data + the Mission Files, and there was a terrible 2-3 minutes long LAG , and after the LAG was gone 15 Players were kicked because of guess what? Yes, because of "Signature checking timeout".

<span style='color:red'>The most easy fix could be this:</span>

You don't need to reprogram the whole Dedicated Server, you just could make it so, that whenever someone joins during a running game (JIP), the server spawns some sort of a SECOND small child-process on the physical machine, which is only responsible/existent for sending "customfiles", "mission-files" and "gameprogress-data" to the JIP-Clients.

So the MAIN-Server-Process gets not Interrupted on JIP anymore and thus all of the already playing Players gets no LAG & Desync anymore.

Thats all....Sounds very easy, doesn't it?

I'm not a Software Programmer ( i only programm PHP, (x)HTML and CSS), but i think this could be done very quick...

That Fix and you can wipe out the most recent reported BattlEye issues aswell as the other mentiond issues with this JIP-Bug.

Other than that you could temporary raise the time until a "signature-checking timeout" or a "BattlEye client not responding" Kick appears.

Please BIS fix that for the next Version - And I'll kiss your Feet for that.

Best Regards, Christian

[sparks50 0]

Supported 100%. Proper usage of several cores wouldn't be bad either.

[dm 9]

...serious Bug, which could be fixed probably very very easy.

Thats all....Sounds very easy, doesn't it?

I'm not a Software Programmer ( i only programm PHP, (x)HTML and CSS), but i think this could be done very quick...

If it was that easy, dont you think they would have done it by now?

[sickboy 13]

Mr.G-C's "Teaching BI" Episodes are back

Haven't read the whole story, but im fairly certain they are aware of it. I don't think it's as easy as you sound it to be.

As ArmA2 seems to concentrate on Multi-Core, who knows if we see such improvements ported to ArmA1. In any case, ArmA2 seems to be going to handle this better.

[whisper 0]

Haven't read the whole story, but im fairly certain they are aware of it. I don't think it's as easy as you sound it to be.

I'd love to hear BI's input on this, though (not that they owe us any explanation, ofc )

If anything, maybe adding a limit to bandwidth allocated per client, leaving ressources for other clients.

AFAIK there is no tuning parameter that affect single clients, they are all global (global minimum bandwidth available, global max bandwidth, etc...) apart from customFileSize.

Though ofc, that may touch to the core of the dedicated simulation cycle, meaning it would require much reprogrammation.

[Striker90 0]

Yeh i also support that. It would be sweet if BIS would implent that in the next patch, that bug is very annoying.

[HeinBloed 0]

Here are some extended experiences from our server.

ArmA Windows dedicated server.

Core2Duo E6600, 2GB, 100Mbit

custom sounds and faces disabled

Signature-checking disabled.

BattlEye disabled

We are running ArmA and Team Fortress 2 on this server. It does not make any different whether both games running on both cores, or each game on its own core.

As long as Arma does not produce the JIP Lag everythin is fine. The ArmA cpu load for one core can reach 100%. It does not matter.

If ArmA is running with many players and a JIP mission which produces the JIP Lag, this Lag does not only exist in Arma. This Lag affects Team Fortress 2 in the same way.

If we are running a mission like Warefare, Domination or Evolution and players are connecting and disconnecting, it will be impossible to play Team Fortress 2 because of that Lag.

Its not the servers Bandwidth. A Player with a 1Mbit connection is producing the JIP Lag. One or more fullspeed downloads with 16Mbit connections does not have any effekt on any game. Team Fortress 2 is also transfering files (up to 40MB) to clients without any effekt on any game, also not if 10 players are connecting on the same time.

It looks like Arma is taking controll over some part of the operating system or the hardware.

[Anunnaki 0]

Here are some extended experiences from our server.

ArmA Windows dedicated server.

Core2Duo E6600, 2GB, 100Mbit

custom sounds and faces disabled

Signature-checking disabled.

BattlEye disabled

We are running ArmA and Team Fortress 2 on this server. It does not make any different whether both games running on both cores, or each game on its own core.

As long as Arma does not produce the JIP Lag everythin is fine. The ArmA cpu load for one core can reach 100%. It does not matter.

If ArmA is running with many players and a JIP mission which produces the JIP Lag, this Lag does not only exist in Arma. This Lag affects Team Fortress 2 in the same way.

If we are running a mission like Warefare, Domination or Evolution and players are connecting and disconnecting, it will be impossible to play Team Fortress 2 because of that Lag.

Its not the servers Bandwidth. A Player with a 1Mbit connection is producing the JIP Lag. One or more fullspeed downloads with 16Mbit connections does not have any effekt on any game. Team Fortress 2 is also transfering files (up to 40MB) to clients without any effekt on any game, also not if 10 players are connecting on the same time.

It looks like Arma is taking controll over some part of the operating system or the hardware.

Right...that looks like arma server is very bad wrotted. As is mentioned in first post here. When is some player connecting, arma server start some subprocedure to do send files to the clients (even with very low bandwith) and almoust HANG whole main application (dedicated server) and almoust whole one CPU core !

I can't understand why they dont do that thru another separate thread for JIPing players with low CPU priority, so game can continue withouth problems or lags.

And that was tested on dual CPU and 100Mbit FullDuplex Ethernet LAN, betwen 10 playeres on that network.

Please, BIS, there is nothing so important than fixing this school bug, which is not fixed longer than one year now from releasing your/our ARMA game ! It is shame !

[mr.g-c 6]

Mr.G-C's "Teaching BI" Episodes are back

Ha ha Sickboy, very funny.... is this your english humor?

Quote[/b] ]Haven't read the whole story, but im fairly certain they are aware of it. I don't think it's as easy as you sound it to be.

As ArmA2 seems to concentrate on Multi-Core, who knows if we see such improvements ported to ArmA1. In any case, ArmA2 seems to be going to handle this better.

Well if you havent the whole story then why do you answer here? Can you explain that?

And to wait for Arma2 isn't the solution. Its the same as if you would have a seriuos manufacturer-caused "bug" at your car, and a well-so-smart co-worker says: Well wait for our next modell of this car and it will be fixed....

Regards, Christian

[sickboy 13]

And to wait for Arma2 isn't the solution. Its the same as if you would have a seriuos manufacturer-caused "bug" at your car,  and a well-so-smart co-worker says: Well wait for our next modell of this car and it will be fixed....

Regards, Christian

No it is not the same, and the day that you will see that too, is the day that we walk hand in hand

Have you tried tests with no mods whatsoever, and missions without any scripts that contain vehicleInit's, publicVariables etc?

To my experience, the lag and problems caused during JIP, are caused by improperly written scripts with a thousand vehicleInits, pv's etc. etc.

The fact that this can lag the rest of the server, is surely reason to say the Server / Engine has bugs / problems in this area. However they can be workedAround by smart programming.

About the Signature timeouts; seems also somewhat bugged, but simply workaroundable by rejoining.

Quote[/b] ]Ha ha Sickboy, very funny.... is this your english humor?

"English Humor"? Anyway, I find it indeed hilarious, how you come up with ideas how BI could / should do it. That's all I tried to express with the "Joke".

AFAIK (Im not a programmer either) Multi-Threading doesnt automaticly mean that an application is multi-core compatible (or has benefit by it), simply because of the synchronization of the data between the threads and cores.

So hence I expect that there surely are things to optimize / fix for BI, but the real multi-core optimizations etc, should imo not be expected before ArmA2 (Simply because the program strategy is different from a single-thread/core design). (Unless they port the changes to ArmA1).

[mr.g-c 6]

Sickboy in case all of that answer was directed to me, i still think you didn't have read it entirely.

I never talked/request about any Multi-core optimizations and to ask for that in Arma1 is indeed hilarious.

But, the JIP bug is well known in ALL missions, no matter how the Server configs are and so on.

Yes i have to confirm that it is as worst (with longest duration) in script-heavy and big-mission-file size missions like Evolution, Sahrani-Life, Domination whatsoever but it also happened to me in other smaller missions.

So its definitely a bug, no need to defend here anything or to say it is caused by wrong programmed missions - really Sickboy.

Even the Owner of Squadserver posted in a MP Topic about that issue and he has already done the same test and he also said that he think the server "stops" or "halts" for all current player at JIP.

The golden rule to replicate this is: The longer the Game Process, The more Customfiles people are using and the bigger the mission-file to download for JIPs are - the longer will be the LAG and this can mean that signature checking is timed out aswell as in case of BattlEye, that the already playing ones get kicked for "BattlEye not responding".

And when you read it entirely then you must have found out that i don't teach BIS anything, nor do i insulted them or whatsoever.

You know me Sickboy, if i wanted to do this i already had - instead i took like 1 hr to write this report in a hop for a fix - thats all. And the idea for a fix is just a IDEA or a SUGGESTION, nothing they have to.

Best Regards, Christian

[sickboy 13]

I guess it's tough to overcome the past sometimes, and im pretty sure that language barriers are at work aswell

I'll leave you guys to it, and sry for bothering

[brit~XR 0]

Even when you disable custom files and squad xml pics uploading to server iv still seen hevay jip lag on the warfare server.

And if you do #monitor 1 and wait till someone connects you will see the monitor status completely pause for 5secs from a player connecting to his connected. Dout they can fix this easily

Needs to be a away that players can join server and download map without it afecting any of the players on server like other games. But yeah jip sucks so bad and even makes good servers look laggy. Not suprised alot of clans gave up on arma and gone to other games like cod4 or what ever

[mr.g-c 6]

Quote[/b] ]But yeah jip sucks so bad and even makes good servers look laggy. Not suprised alot of clans gave up on arma and gone to other games like cod4 or what ever

Exactly thats the point.... the Arma MP Playerbase has shrink so tremendously because of all these bugs.

JIP-Bug can really destroy someone game and i often disconnected because of this (and i'm sure other do also)

[.kju 3245]

JIP lag bug has different reasons or params AFAI cann tell.

Custom files, map download, too much GV, too much data needs

to be transfered, player amount.

There are definitely problems with JIP.

As the debug options are nearly none existent, its hard to tell

the actual source(s).

[Yoma 0]

Actually i think this may be very very hard to fix.

Imagine something like you have a train of data that has to be processed to get you in sync with the other players.

Now while you try to get onboard, the train moves on. While the train moves it generates more data to sync.

This means the server will have to give you data at a certain speed or else you'll simply never catch up and stay on "receiving" forever...

Maybe they actually press the brake on the whole train, so a connecting player can catch up.

I know this sounds a bit rediculous and is most likely far from how it really works but it may be something along the lines.

If this is the case "multithreading" might not be the simple answer to solve the problem. As no matter how many threads you have, somehow you'll have to get synched with some kind of main server thread.

One wonders how much data (how long/big the train is) is involved.

There may be some stuff in there that has room for optimisation but it's up to BIS to find that out (if it's possible i'm pretty sure they'll come up with something in the end)

[suma 8]

Thank you for detailed repro steps - this most important part of any bug report, and this is much more important than things like if the bug report is submitted via BTS or a forum.

Thanks to your repro steps we were able to identify the problem and I hope to have some solution available soon. A little bit more information follows:

The "bandwidth balancing" is already implemented and it should work in such a way than one player is never getting extremely disproportional traffic. The problem we have found is in a different area, and surprisingly simple:

For bandwidth control the game needs to know approximate frame duration, so that it knows how much information it should try to send in one frame. What we do is we ask rendering subsystem to tell us average fps for last 16 frames, as this is something rendering already knows. Now the problem is on the dedicated server there is no rendering done at all, and the fps returned by the rendering subsystem is wrong - often numbers like 5 seconds are returned when in fact server is running at 50 fps. We will fix this by not relying on rendering when quering fps for this purpose. We are convinced this fix should not only greatly reduce the JIP caused lag and desync, but hopefully improved bandwidth control and balancing overall.

If someone wants to test meanwhile how balancing code works without this problem, you can try the repro steps above with a normal non-dedicated host. One important gotcha: when testing, make sure the host really is rendering, e.g. by letting someone to really play at it, as once the host loses focus (by alt-tabing or in a similar fashion), it would most likely show the same buggy behaviour again.

[mr.g-c 6]

Ohh great Suma.... im on my way to minnesant pod brody to kiss your feet!

Unbelievable, first you fix the Bike Bug, then the JIP Bug.... whats next?

This is really great!

Thanks alot.

Best Regards, Christian

[suma 8]

Please, do not rejoice too much, until there is some verification the fix really fixes what you (and me) hope to fix it. Our internal fix testing is still in progress, and there is still no independent verification. Luckily, the host testing could give you or anyone else to preview the fix potential even before we release it. Who will be the first one to report results here?

[mr.g-c 6]

If you can do that:

Quote[/b] ]We are convinced this fix should not only greatly reduce the JIP caused lag and desync, but hopefully improved bandwidth control and balancing overall.

with it, then i think it's what i'm hoping for.

And by the way, does that also mean that the Arma server will not need so extremely much ressources anymore? I mean it's the most likely biggest proccesor-time eater on this Planet....

Ok so to test, we just need to open a Multilayer-game where the Host actually is not a dedicated server and really has rendered game-engine on his screen?

If so, that should be doable because one Member of our forums has a 512kbyte/sec upstream connection.... I'll ask him and we do some JIP-Lag/Desync tests hopefully today and we use some script-heavy mission for that (Evolution or such).

Best Regards, Christian

[suma 8]

And by the way, does that also mean that the Arma server will not need so extremely much ressources anymore? I mean it's the most likely biggest proccesor-time eater on this Planet....

I do not expect any change in this.

Quote[/b] ]Ok so to test, we just need to open a Multiplayer-game where the Host actually is not a dedicated server and really has rendered game-engine on his screen?

Exactly.

[vipermaul 246]

Hmm...

We should be able to do this over at http://www.theatre-of-war.com

But perhaps I can get together with Mojo over at http://www.ic-arma.com and combine our tests.

Because our two tournaments are big time effect by this as we average 60+ and 100+ player games each weekend respectively.

I wish we had a community BTS back online somewhere.

Thanks Suma. I'll contact Mojo and see if we can do a big test.

[sickboy 13]

That's really awesome news. So no need for child processes and threads / multi-core solutions

Hooray for Suma

And thanks to Mr.g-c for his time and effort (Sent yah a PM), sorry to've bored the thread with my crap.

[Actium 0]

A way of improving both server performance and map download speed, would be a configurable option to forward map downloads to an external httpd or ftpd (I'm certain, for example, lighttpd scales much better concerning both load and throughput (due to its non blocking io, use of kernel functions, etc.) in comparison with the arma dedicated server (since that is a httpd/ftpd's purpose)).

Another idea for a server-side setting would be a possibility to switch off the deprecated "spawn vehicle (even the one's which have been destroyed for hours) -> exec its init command -> despawn it"-feature.

[whisper 0]

Two words :

Awesomeness incarnated

But let's wait for the tests and fix results