How to improve MP warping

[daikan 1]

Guessing from the more or less steady number of complaints and discussions that are somehow related to the issue of multiplayer "warping" it's not very surprising that a lot of people seem to hope/wish for a better (different?) handling of network latency related problems when it comes to the synchronization of object's position and movement in ArmA3 (just do a search for "warping" in this forum)...

Regardless of still not knowing whether BIS will be revisiting this topic for ArmA3 I thought that maybe this is the right time and place to have a proper discussion going...

1. Do we all agree on a what "warping" actually is and why it occurs in the first place?
   
2. What would be your ideas (regardless of your in-depth knowledge of the matter) on how to deal with the dilemma of network latency vs. immersion and coherent gameplay so that it would improve the status quo?
   
3. More specifically: Would you consider achieving significant movement "smoothness" at the cost of potentially more frequent causality issues due to greater positional error (like getting hit even after reaching cover) an improvement?
   

My personal opinions:

1. "Warping" occurs when the error between an object's current position in the client's game world and an incoming update exceeds a certain threshold (due to lag etc.) and the engine decides to instantly reposition it based on that received update in order to sync to the server.
   
2. If possible the mechanism of resolving positional errors should always act within the given limits of game physics and the vehicle's parameters (steering/acceleration capabilities, inertia...) in each client's game world. E.g. instead of instant warping the rendering engine should try to override the vehicle's "steering" inputs in order to correct positional errors and "catch-up" - up to certain limits of course (e.g. if the error becomes too big -> fall back to warping). I don't know if something similar is already implemented and just needs more tweaking but at least for me it looks like currently warping will never go away regardless how much you tweak server parameters like minErrorToSend etc.
   
3. Personally, I would definitely be willing to live with a certain causality trade-off in exchange for smoother object movements. However, it would be nice to have a choice as server admin to select between different methods of positional sync, each one with it's own set of advantages and disadvantages...
   

Discuss! :)

[ProfTournesol 956]

Have you read this ?

[daikan 1]

Nope... can't keep up with everything nowadays :)

Thanks for the link, however! That's a very interesting blog indeed! And funny I was apparently thinking along the same lines as BIS - to some extent at least..

In any case the topic is still up for discussion I guess ;)

[dupa 10]

As far as I know, there could be 2 reasons for warping:

1. Bad Internet connection on server and/or client

2. Overloaded server

These problems can be avoided if the client connects to servers that are geographically close to him (i.e. don't play on Australian servers if you live in Norway) and the server admin does not overload his server (typical home server will stutter badly with 30 players).

I have encountered warping mainly because of bad ISP. It was so bad, that I've even been accused of cheating. The moment i switched to a better ISP, the gameplay became as smooth as a baby's butt.

The bottom line is that you can solve it yourself without any help from BIS.

p.s.

If possible the mechanism of resolving positional errors should always act within the given limits of game physics and the vehicle's parameters (steering/acceleration capabilities, inertia...) in each client's game world. E.g. instead of instant warping the rendering engine should try to override the vehicle's "steering" inputs in order to correct positional errors and "catch-up" - up to certain limits of course (e.g. if the error becomes too big -> fall back to warping). I don't know if something similar is already implemented and just needs more tweaking but at least for me it looks like currently warping will never go away regardless how much you tweak server parameters like minErrorToSend etc.

From what I've observed in ArmA 1, which I play a lot, this is pretty much the way it works now - the vehicle maintains its last received velocity/direction and if no updates are received for a while then it freezes. After it unfreezes again, you get the red chain.

[tacticalnuggets 24]

As far as I know, there could be 2 reasons for warping:

1. Bad Internet connection on server and/or client

2. Overloaded server

These problems can be avoided if the client connects to servers that are geographically close to him (i.e. don't play on Australian servers if you live in Norway) and the server admin does not overload his server (typical home server will stutter badly with 30 players).

I have encountered warping mainly because of bad ISP. It was so bad, that I've even been accused of cheating. The moment i switched to a better ISP, the gameplay became as smooth as a baby's butt.

The bottom line is that you can solve it yourself without any help from BIS.

p.s.

From what I've observed in ArmA 1, which I play a lot, this is pretty much the way it works now - the vehicle maintains its last received velocity/direction and if no updates are received for a while then it freezes. After it unfreezes again, you get the red chain.

From what I can understand they are working on a dynamic interpolation system that even uses some form of abstract differentiation between past and future states.

[cyop 10]

Of course, all of this is fine for discussion. :) (Really.) However, regarding Arma 3, I think the discussion is somewhat unnecessary. With the experimental betas addressing things like (AI) warping, and the release date a year away, I, personally, will wait a lot longer to try and seriously discuss this. (I need to do more research on the betas, anyway.) And, with other discussions about warping, and BIS already knowing what people's complaints are and what they may need to improve, the 'discussion' would be more like... rehashing. But, I guess rehashing is OK. I do see your reason for this thread, though.

Anyone have a good link about any work BIS may doing with the net-code?

As for Arma 2, and current game-play, yes, people need to try and find servers to play on that suit their geographical location. The less hops, the better. But, probably, everyone knows this already. And, of course, it is better to play on a (good) dedicated server than a hosted one, and, probably, everyone knows this, also. Everyone (probably) already knows, too, if your ISP sucks, you're screwed, no matter what.

ASIDE: I have not messed around with a dedicated server and COOP with any Arma sim. But, for those that like to build COOP missions for themselves and friends (or for squad practice, etc.), if you do not have a dedicated server, it should not take much money to build a sufficient server to host the game. Hey, all the friends can pitch in. - It could even be located with the person that has the best Internet connection.

1. (I am sure 'we' do not, or some people would have not been called a cheater, from time to time.) To simplify, I just usually call it, 'losing' packets.
   
2. The simplest thing would be forced dynamic netspeed for each client. I do not know if any developer has ever implemented that 'properly'.
   
3. Maybe you have come across different people than I, but I have not known anyone that likes to die after reaching cover. My personal answer is, no. However, your personal opinion is good enough to make me want to say, yes.
   

1. -
   
2. If possible the mechanism of resolving positional errors should always act within the given limits of game physics and the vehicle's parameters (steering/acceleration capabilities, inertia...) in each client's game world. E.g. instead of instant warping the rendering engine should try to override the vehicle's "steering" inputs in order to correct positional errors and "catch-up" - up to certain limits of course (e.g. if the error becomes too big -> fall back to warping). I don't know if something similar is already implemented and just needs more tweaking but at least for me it looks like currently warping will never go away regardless how much you tweak server parameters like minErrorToSend etc.
   
3. -
   

I do not like that. I will try to elaborate on that further at a later time.

[daikan 1]

Of course, all of this is fine for discussion. :) (Really.) However, regarding Arma 3, I think the discussion is somewhat unnecessary. With the experimental betas addressing things like (AI) warping, and the release date a year away, I, personally, will wait a lot longer to try and seriously discuss this. (I need to do more research on the betas, anyway.) And, with other discussions about warping, and BIS already knowing what people's complaints are and what they may need to improve, the 'discussion' would be more like... rehashing.

Actually another reason for starting this thread was also that I find it a very fascinating problem to ponder about regardless of what will happen with ArmA3...

After all, the problems and paradoxes arising from the loss of an absolute frame of reference can be exquisitely intriguing and counter-intuitive (= very annoying in MP games) by themselves. See special & general relativity theory in physics for a real-world example :) And multiplayer games just happen to be a nice hands-on example where you can actually get a feel of what it means to live in a (sort of) quirky relativistic universe with no true synchronicity...

Speaking of physics:

I'm really curious how BIS will model hardware accelerated physics over a network link so that everybody gets to see approximately the same outcome of those nifty calculations (remember chaos theory? :))... Unless the path chosen will be the trivial one, meaning that those physics effects will be tucked away safely in the eye-candy domain (e.g. client-side only without any coherent multiplayer experience).

Sorry, I am just rambling...

Edited July 7, 2011 by daikan

[cyop 10]

I'm a science fan! :D ...... (daikan did not write that, CyOp did.)

I just call it, 'losing' packets. :p

I am not going to elaborate on your #2 opinion, after all. I will say, if someone could code that in a way as to not make people mad, that would be fantastic.

Some of the whole warping problem comes down to client common sense, too. (See also, geographical location/host/dedicated/ISP.^ And, I am not going to be as scientific as you, daikan. :)) One thing we have not mentioned is client personal settings. For instance, if a persons computer is already 'choking' in SP, he will be worse off in MP. - When I would play a game a great deal in both SP and MP, I would make sure my (graphics) settings were a lot less demanding for MP. If possible, I would also tweak the game Internet connection settings (in a game file).

I cannot guess how many people hinder themselves, now a days, but a long time ago, it was pretty common for people to have their graphics settings way too high for MP, which would 'hurt' their MP experience. I am assuming this would apply to Arma 2 (3?) regarding warping? EDIT: Or, 'stuttering', anyway.

Edited July 8, 2011 by CyOp

[NordKindchen 12]

Just wanted to show with this post that i too am concernced about this problem.

Pls BI look for a solution.

Best Greetings

[Baff1 0]

One key to reducing and indeed eliminating warping is unfortunately map design.

To design maps that block the views of other players. So that not to many of them can see or hear too many of their peers at any one time.

The rise in network traffic, as a new person enters the Sphere of Influence of other players is exponential in nature.

So the bandwidth used when 5 players are looking at each other, is 1/6 of the bandwidth used when 6 people are. And the bandwidth used when 60 players are looking at each other is 1/61st of the bandwidth used when 61 players are looking at eachother.

(Quite aside from the CPU load).

Unfortunately, people who play ArmA typically want to see massive battles and have view distances of 8 km from their helicopter gunships alowing them to see all the other players all at the same time from their eagle eyed vantage point of death.

This game does not lend itself well to netcoding.

It could do, but the compromise you would end up with would reduce your game maps to something better approximating Battlefield's scale and arena styled layouts. (Where hills and mountain ranges, buildings and fogs.. cunningly mask off choke points and control points throughout the game so that at no point are you ever expecting to see every single player on the server simultaneously).

Actually, I think this would be no bad thing to include some dedicated multiplayer arenas' in ArmA 3.

Show people what the engine can do, open up the game to new players for whom the lag/warping is too off putting.

[daikan 1]

So the bandwidth used when 5 players are looking at each other, is 1/6 of the bandwidth used when 6 people are. And the bandwidth used when 60 players are looking at each other is 1/61st of the bandwidth used when 61 players are looking at eachother.

I dont' think that's true. Care to elaborate?

Here's my math:

Assume x being the number of players and 1 being the bandwith cost of transmitting a single player update from/to the server. Then the cost of each player sending his/her update to the server (= server inbound) is x. And the cost for each player receiving the updates from other players (= server outbound) is x(x-1).

This yields a total of x^2 updates. So in the case of 5 vs 6 it would be 25 vs 36 updates which is far from being a 1/6 ratio...

Also, strictly speaking it's not exponential growth but rather polynomial.

Edited July 15, 2011 by daikan

[Baff1 0]

With 6 players...

I send the location of Player 1, 6 times. Once to each player.

I also send the location of player 2, 6 times. Once to each player.

I also send the location of player 3, 6 times, once to each player.

I also send the location of Player 4, 6 times, once to each player.

I also send the location of Player 5, 6 times, once to each player.

I also send the loaction of Player 6, 6 times, once to each player.

I if add a 7th,

I send the location of Player 1, 7 times, once to each player.

I send the location of Player 2, 7 times ,once to each player.

I send the location of Player 3, 7 times, once to each player.

I send the loctiaon of Player 4, 7 times, once to each player.

I send the location of Player 5, 7 times, once to each player.

I send the location of Player 6, 7 times, once to each player.

I send the location of Player 7, 7 times, once to each player.

So when I add a player, the total size of each gamestate I send is increased by factor of one, and the number of times I have to send it is increasd by a factor of one.

Lets say our gamestate = 2bytes in size per player

Each tick....

For 6 people, I transmit, 2bytes x 6, six times.

For 7 people I transmit, 2bytes x 7, seven times.

So the increase in bandwitdh being used is exponential in nature. From five players to six players, the bandwidth is not 6 times greater, it is 6x6 greater.

jump to 7 players and it is not 7 times greater, it is 7x7 greater.

The map optimisations come into play when because some of those players can't see each other, I don't have to send their gamestates to the other players.

Edited July 15, 2011 by Baff1

[daikan 1]

With the above method you are counting the transmission of your own update to yourself, which IMHO is not necessary.

[Baff1 0]

It's necessary for a dedicated server.

Edited July 15, 2011 by Baff1

[daikan 1]

I don't think so. Why would the server have to send you information you already have (e.g. your location)?

[Baff1 0]

Because, typically, the server controls the game.

Not the client.

All gamstate calculations are serverside.

So your location is not where you tell the server it is, it is where the server tells you it is.

When you move, you send a request to the server to move, it calculates that move and then replies with an update of the gamestate.

The client is a dumb terminal. All the processing is done at the server. All the client does is transmit, receive and display.

The server typically refreshes the gamestate to all clients many times per second. On a game such as World of Warcraft you will see that your FPS are capped to this amount of times. Other games such as Unreal use prediction so that they their FPS is not capped by their tickrate. The client guesses the answer that the server will respond with and draws it on the screen.

When it gets it wrong, you get "rubberbanding" and the player see's himself returning to where the server says he is, not where his client predicted he would be.

For most games the server controls the gamestate for 2 reasons, one, synchronicity and two, anti cheat.

It is easy to hack your own machine, it is much harder to hack anyone elses. Likewise it is easy for 32 machines to be told what to do and when to do it, it is difficult for 32 machines to all do the same thing at the same time independantly. (particularly when you add internet errors and lag into the equation).

Some types of game it should be noted typically do not use this master-slave approach to networking.

RTS would be a good example of this.

Standardly in an RTS all clients calculate everything themselves all the time. If the host "server" disconnects, the game does not go down while in a standard shooter title, if the server goes down, the game ends for everybody connected to it.

Edited July 16, 2011 by Baff1

[sickboy 13]

I am not too sure it is related to Line of Sight. What evidence supports this?

Rather only the distance. While at larger distance, the frequency of object updates is reduced (and therefore Interpolation plays a big role here).

iirc the warping occurs even with low amount of players / units (but does increase with the amount), and rather is a result of the implementation method (bug/design) of interpolation and reduced simulation at distance.

Also afaik the warping occurs less with player units than with (server-based?) AI units.

Related; http://www.bistudio.com/index.php/english/company/developers-blog/230-experimental-betas-interpolating-the-future

http://dev-heaven.net/issues/1915

Edited July 16, 2011 by Sickboy

[Baff1 0]

In order to save on bandwidth a server usually only sends the player the information he "needs to know".

So instead of sending each client all the other players locations, asimuth, gear loadouts etc, 30 times per second, it instead only sends each client that which they need to know.

That being all the actors he can see and hear.

When you put an aeroplane in the sky, that everyone can see.... bandwidth use spikes enormously; because so many people can see and hear it, they all have to be kept updated of all the changes to the gamestate made by it.

Experimentation with Unreal taught me that some maps only worked well in single player. Those that were too open... lagged uncontrollably. That a serious chunk of Unreals netcode optimisation was in it's map design.

The article you have linked describes the relationship between server gamestate updates and clientside prediction. This is not a method of reducing bandwidth usuage, but rather is an optical illusion of sorts to trick the player by animating over any lags. The lags are still going to be there... but you won't notice so many of them.

Edited July 16, 2011 by Baff1

[daikan 1]

So your location is not where you tell the server it is, it is where the server tells you it is.

When you move, you send a request to the server to move, it calculates that move and then replies with an update of the gamestate.

I don't think that's how it is currently implemented in ArmA2 for two main reasons:

1. Your own character can always move (and smoothly, too), even if the server is lagging or the connection drops for a short moment.
   
2. You never experience warping of your own character, only other players and AI.
   

Of course it makes sense that the server is the gamestate master and sending updates accordingly, but only if there is something your client isn't already aware of in accordance with the server's gamestate.

As all your character's actions and movements originate on your client (and there's no need for the server to override them unless there would be a collision etc.) it wouldn't make any sense to send this information back and forth, because it would be the same most of the time.

Plus I'm sure there is a distinction for guaranteed and non-guaranteed gamestate updates. Guaranteed would be gamestate updates like kills, collisions, object states etc. while non-guaranteed would be typically events related to movement. This distinction would help keep bandwidth usage within reasonable limits as movement updates are by far the most frequent but also less vulnerable to packet loss precisely because they're frequent and continuous in nature (e.g. they get updated by the simulation anyway) and therefore don't have to be guaranteed (= re-transmitted in case of packet loss).

---------- Post added at 12:06 AM ---------- Previous post was Yesterday at 11:42 PM ----------

I am not too sure it is related to Line of Sight. What evidence supports this?

Rather only the distance. While at larger distance, the frequency of object updates is reduced (and therefore Interpolation plays a big role here).

I have to agree with Sickboy in that the current evidence is not quite in favor of the proposed "line-of-sight player state masking" theory.

For example: In Arma2 it is still possible to hear all the sounds originated by a vehicle (engine, movement, weapons) even if it's hidden behind a small hill or over a mountain crest and therefore visually masked by the map design. You can easily test this in any multiplayer mission.

Edited July 16, 2011 by daikan

[Baff1 0]

I don't think that's how it is currently implemented in ArmA2 for two main reasons:

[*]Your own character can always move (and smoothly, too), even if the server is lagging or the connection drops for a short moment.AI.

a) This is prediction.

Your client animates what it predicts the gamestate will be. This diminshes the amount of warping that you will notice. This doesn't diminish lag in anyway, only your perception of it.

For example, if you are walking forward, your client doesn't wait for the server to reply telling it to display that you have walked forward, it animates it anyway. Then if needs be, if the prediction your client made was inaccurate, it updates your position.

b) I assume you have had moments were you lagged and at the end of the lag you found yourself to have been shot? Lag death.

You need to update your movement to the gamestate, otherwise you cannot accurately calculate collision. If the server does not know where you are, if cannot tell if you are where a bullet is.

It cannot tell the other players, if they can see or hear you if it does not know itself.

The more often the server refreshes the gamestate, the more accurate the collision detection.

When you play on MMO sized servers you will get far sloppier hit detection, Planetside is a good example. They will also use less hitboxes. A game like WoW won't even calculate collision or hitboxes at all. Lags in these games can easily be upto a second or more long without a player noticing anything.

A simple way to test wether or not BIS uses standard netcode technics is to run a dedicated server and measure it's CPU usuage. Connect with a client and run that's CPU usage too. I would expect the client to be using comparatively little CPU on a lockstep FPS and the same as the server on an RTS.

As I understand it, the gamestate signals the server sends are not a continous stream, but rather a series of very short bursts. Perhaps 25-40 of them a second depending on the server settings and the game.

If you lose a packet, the next one will be along with all the very latest important data soon after.

Packetloss, I assume, is corrected by the next gamestate tick.

It must take longer to check and confirm a packet than it does just to send the next one.

If your character is not there for .03 of a second once in a blue moon, the chances of this being the critical moment a bullet hits you are very slim.

I have no idea if BIS netcode resends any data to confirm it got through O.K.. I haven't heard of this being done before, although I have experienced bugs in Ravenshield, which this practise, in my opinion, would have fixed.

I also very much doubt if this game calculates whether or not two players can hear eachother based on Line of Sight. Sphere of Influence typically includes line of sight, but is by no means restricted to it.

If all players were in hearing range of a BMP then, even if they could not see it, the server bandwidth use would still go up exponentially.

Edited July 17, 2011 by Baff1

[Mr. Charles 22]

A player is always local to his machine, same with AI under his command and vehicles, driven by the player or his AI.

[Baff1 0]

Wow.

No wonder it lags then. Mystery solved.

Thanks to all for taking the time to explain it to me.

Edited July 17, 2011 by Baff1

[dmarkwick 261]

Because, typically, the server controls the game.

Not the client.

All gamstate calculations are serverside.

So your location is not where you tell the server it is, it is where the server tells you it is.

When you move, you send a request to the server to move, it calculates that move and then replies with an update of the gamestate.

The client is a dumb terminal. All the processing is done at the server. All the client does is transmit, receive and display.

This is incorrect. The player, the player's vehicle (if he's in one) and the player's AI are all local. The server does not control them the client does, and the client updates the server. The server then farms out the positions to the other clients.

This means that from the player's perspective there is no lag when flying or driving, as the vehicle is local. Any other player who is a passenger on your vehicle however, might experience quite severe lag, as there is essentially 2 tiers of update to account for.

I think your general bandwidth math is wrong, or at least very innaccurate :) I'm pretty certain that adding each new clinent does not increase the bandwidth needed by orders of magnitude.

---------- Post added at 01:32 PM ---------- Previous post was at 01:31 PM ----------

Wow.

No wonder it lags then. Mystery solved.

Thanks to all for taking the time to explain it to me.

Oops I guess I missed some posts :D sorry for the unnecessary explanation :)

Edited July 17, 2011 by DMarkwick

[xeno 230]

Wow.

No wonder it lags then. Mystery solved.

Yep. Specially for a smooth PvP experience it is better if the server is completely in control of client updates.

But as the game also allows x AI units under player control I guess the other approach was chosen

(plus you have a real body in ArmA compared to other games where you just have a weapon attached to the screen).

Would be cool if the engine could support both options. But I guess that won't happen.

Xeno

Edited July 17, 2011 by Xeno

[Baff1 0]

I concur, it does seem to be a compromise for the sake of large AI armies.

Like you, I would gladly settle for smaller numbers of AI and smoother netplay.

It would take a lot of the fun out of mission editing however. Setting up those big living worlds...

Having to use spawn triggers instead of legions of pre-spawned actors with all their patrol routes etc.

I still used to cram about 80 AI's into my Ravenshield server though. That would still leave me with a lot of potential to make intresting missions.

For a lot of my players, the lag in ArmA is an unforgiveable sin.

I think your general bandwidth math is wrong, or at least very innaccurate :) I'm pretty certain that adding each new clinent does not increase the bandwidth needed by orders of magnitude.

Thanks for the explanation, my general bandwidth math is correct however. The progression is exponential in nature. At least this is the way Epic teaches it.

There are ways to mitigate this however. For example intelligent map design.

3! + 4! < 7!

So if we design maps in which we don't expect all the players to be in LOS of eachother at the same time, but instead all fighting in different areas, compartmentalised..while the maximum possible bandwidth use remains the same... in practise this rarely occours.

Edited July 17, 2011 by Baff1