Multiplayer not in playable state

[Lokyi 10]

Dorph;2555714']Pls remember when testing this - the most important issue will be having 20+ clients on and have the session running for some hours.

Although starting a mission fresh with over 30 clients waiting in the lobby is just as fun ;)

[Greenfist 1863]

I'm willing to sign up and be a test subject on these test servers.

I will promise to be the last guy in after the fps has been below 2 for the last few hours while devs debug the server.;)

[lappihuan 178]

Well it seems the main issue in MP is the high bandwith usage...

Some weeks ago i tried to engage this with a script that handles simulation on every object on the map.

sadly there are two main issues:

1. Collecting Objects on alits requires a huge effort and space in ram, but with the "optimisations" of the map there were classnames deletet from serval objects (results in non collectable objects :rolleyes:)

2. the command enableSimulation has no impact on the map objects (only on editor placed ones :rolleyes:) but then i saw a module that should handle this...

sadly it does only affect editor placed objects too..

So my suggestion to this topic would be:

Let handle the engine every object ingame as simulation disabled.

Or split simulation and syncronisation to standalone functions and disable the syncronisation as default for each object.

Let EH's open a short window to syncronise the changes on objects trough the network.

This would lower the bandwith usage drastically.

E:

to make this even more clear:

Lets say, there are 20 Clients connected to a server.

The mission runs on Altis.

With the current system, all objects on altis get syncronised constantly over the network. But with 20 clients it all objects gets syncronisated 20^19 times over the server. This is ha huge load on data that gets synced.

Whit my system, there would only changes (like destruction etc.) on objects be syncet to other clients.

This would be less effort for the server with 20 clients connected while destroying a whole city. than the 20 clients with the old system only sitting in some bushes and constantly syncing their whole map through the network...

To be clear, this is everything just speculation on how the system works right now and there may comes a dev around and hit me in the face. But from what i've seen this is how it works right now.

I hope a dev will respond to that :)

Edited November 12, 2013 by Lappihuan

[tarabas666 10]

MP is very good playabler atm. On Altis EUTW TeeTimes Warfare with 60 Players i get 40-60 FPS in almost every Situation. ViewDistance is 1300. Mostly the rest ist high-very high (FSAA2x).

Problem is when playing in big Citys and some houses get destroyd the FPS drops down to 30 some times which is unplayable for me.

This is cause the VR Engine syncs so much , almost all stuff ongoin on the map. DRH solves this in dayzStanaldone by doing his NEtwork Bubble Stuff.

Thats only my guess, i am no Programmer and i dont work for BIS, so who really knows whats the cause...

[Hud Dorph 22]

MP is very good playabler atm. On Altis EUTW TeeTimes Warfare with 60 Players i get 40-60 FPS in almost every Situation. ViewDistance is 1300. Mostly the rest ist high-very high (FSAA2x).

Problem is when playing in big Citys and some houses get destroyd the FPS drops down to 30 some times which is unplayable for me.

This is cause the VR Engine syncs so much , almost all stuff ongoin on the map. DRH solves this in dayzStanaldone by doing his NEtwork Bubble Stuff.

Thats only my guess, i am no Programmer and i dont work for BIS, so who really knows whats the cause...

Hmm you could be on to something there, i have noted that with many players things runs well at first, but when first town taken and target switches to next town, things (server-fps) start to go down.

[xeno 230]

Dorph;2555862']Hmm you could be on to something there' date=' i have noted that with many players things runs well at first, but when first town taken and target switches to next town, things (server-fps) start to go down.[/quote']

Might be caused by destroyed buildings (just guessing here but from a test some weeks ago I know that destroyed buildings cause FPS loss).

Any destroyed building means at least two objects. The non damaged, original building/object gets moved 25 meters under the ground and a new destroyed building/object gets spawned.

There are even models like the tent hangar which have three objects once destroyed. Non damaged object, intermediate half damaged object (both moved 25 meters under the ground) and a destroyed object.

You can execute a setdamage 0 on a destroyed building (needs a reference to the non damaged building object) which will bring back the non damaged building (moves it to the surface again) and remove the destroyed object.

Xeno

[terox 316]

isnt the issue high player numbers and high ai numbers at the start of a mission (Not dynamically created) ?

for those of you who have glanced over this thread. we are NOT discussing client FPS

This is all about serverside simulation cycle rate (Server FPS if you like) in a COOP (Not PvP) environment.

So anything to do with graphics or clientside issues has probably no bearing here

Lets discuss standardisation for the testing

1) We need a benchmark mission that we can all use (W.I.P) Hopefully the one I am creating will initiate the problem

2) We would need lets say 3 dedi servers running similar specced hardware, similar timezone than can cope with high player numbers, somewhere between 40 and 50 players i presume

..... Zeus will host if required

who else ?

3) Should each of the test servers run exactly the same bandwidth config values? probably yes

4) Base vanilla only, no addons etc to cloud the results

5) I would propose that all 3 servers run a scheduled test one after the other, using the same client list (or as close to that as is possible)

.... basically we run a 15 minute test on each server, one after the other

When this is all done we collect all the test data, zip it into 1 file and then raise a ticket attaching the test mission and logging data to that ticket

I am thinking of how we define the number of ai we use for the test. possibly find a middle ground between the 3 servers where, when only 1 player is connecting, the server frame rate is XYZ (Minor dertail)

or we could just run 3 tests per server, one with 300 ai, then 400, then 500 etc

What data should we collect

What should we use

Wireshark logs, rpt logs

what else would BI find useful. Some input here from BI would be good

[dr. hladik 231]

>> Missions like Domi tend to munch a lot of resources. Using something like that for repro is crazy because of all stuff running. It would be nearly impossible to nail down the issue

No, I don't expect simple repro like place 5 tanks, 3 dogs and 1 cat,

but at least precise info about mision name, version, server name and min. number of players.

FPS is low, is not really helpfull.

>> Regarding scripting, most of the "bad scripting" you can do should not affect performance

Not true. Time for scripts is limited, but if you create tons of events, or don't delete destroyed vehicles, you will kill FPS.

edit:

Also, does it affect only server or client as well?

edit2:

>> I can say that each player-client connected = -2 server fps. So 20 players = 50-(2*20)=10 server fps.

So what is bigger FPS eater? Number of AI, or number of players?

edit3:

any server logs?

edit4:

Both Altis and Stratis have this problem?

Edited November 12, 2013 by Dr. Hladik

[terox 316]

edit:

Also, does it affect only server or client as well?

edit2:

>> I can say that each player-client connected = -2 server fps. So 20 players = 50-(2*20)=10 server fps.

So what is bigger FPS eater? Number of AI, or number of players?

edit3:

any server logs?

edit4:

Both Altis and Stratis have this problem?

Only Altis

Only COOP

Ai v Player numbers?. It seems more like a combination of both. When a certain player threshold is achieved (For us it is in the region of 30) with moderate ai numbers, (50 to 100) it kills the server fps

It also seems to depend on the location on Altis, some areas are much better than others.

This has nothing to do with AI engaging with weapons, as the fps is badly effected at the start of the mission, when no contact has been established.

based on A2 experience, we should be able to run a 40 player coop with say 400 AI placed on the map at the start of the mission and get decent serverside FPS. this is not the case for Altis.

Wait until we can get a benchmark mission played across a few high end servers and then we can get you more feedback, logs etc

I believe it is bandwidth related and will most likely not be seen if tested in a LAN environment which is why it is most likely not seen when you carry out your internal tests

Edited November 12, 2013 by Terox

[galzohar 31]

for those of you who have glanced over this thread. we are NOT discussing client FPS

This is all about serverside simulation cycle rate (Server FPS if you like) in a COOP (Not PvP) environment.

So anything to do with graphics or clientside issues has probably no bearing here

Actually, it seems like the same issues are affecting clients as well (though it may be different issues). When there are lots of players and units client FPS goes down even with minimum graphic settings and/or looking at the sky, so it's not graphics that are slowing them down - It's purely CPU (which was backed up again by the situation becoming less troublesome after I overclocked by CPU by 30%). It's just that if your client's CPU is weaker than your server's, then the clients will see poor FPS before the server FPS becomes low, and vice-versa. *Something* is eating up the CPU way more than it seems like it should be, and this affects both clients and servers.

The issue is not coop-only either. It can happen on large enough PvP missions as well, though it seems to be a lot more apparent on coop missions, maybe because they simply have more units/vehicles active and/or more buildings destroyed.

And yes, it does feel like destroyed buildings are a part of what loads up the CPU, but this is hard to measure/quantify and just destroying an entire city in single player didn't seem to reproduce the issue.

Having a significant amount of AI vehicles active in single player (more than would normally be active in a similar situation in a PvP mission with only player-controlled vehicles) didn't seem to cause nearly as big of a performance hit as was observed while playing the PvP mission with 20+ players, though the impact on performance was still noticeable.

>> Regarding scripting, most of the "bad scripting" you can do should not affect performance

Not true. Time for scripts is limited, but if you create tons of events, or don't delete destroyed vehicles, you will kill FPS.

I thought that's pretty much what I said. The scripts on their own don't eat up your CPU due to the limitation, but they could create a lot of things for the engine to handle like creating lots of vehicles and units. It'll be the vehicles and units that slow down the server then, and not the script - Terminating the script in that situation will not help performance, while deleting the vehicles will help. To me that's not really a "script performance issue" as the script itself isn't eating up the CPU power but rather it's the simulation that eats it up.

If you write a script that takes 1000x times to produce the same results ("result" being events external to the script such as vehicles/units being created or deleted) you will not notice any measurable difference in FPS.

Bottom line is the scripting commands on their own can't hurt really hurt your FPS, only their side effects can, but those side effects would still exist even if the script is terminated and thus you can't really account the FPS loss to the script itself but rather the overall mission design (or the engine's lack of ability to deal with the simulating the side effects of the scripts in case they are reasonable, and in any case it's not the engine's lack of ability to run scripts fast enough as that cannot affect FPS - Instead the engine will slow down the scripts and keep FPS almost intact).

[harakka 10]

Chiming in with our community's experience. We have a Xeon E3 @ 3.4GHz on 100mbit connection. We play primarily coop missions taking from 30 to 90 minutes, no Domi or any other long running public server mission style stuff. All missions are custom made (using the F2/F3 framework) and are usually fairly simple, with minimal scripting, and based on monitoring with ASM there's no runaway object generation or anything similar going on. We do not run any mods aside from minor clientside ones such as STHud.

As background, with Arma 2 we've been able to play 80 player coops with ~300 AI with SFPS tanking to 5-10SFPS. Usually we have playercounts of around 30-50 players with 3-4 times the AI, and A2 can handle these numbers with 20-35 SFPS. We could accept losing some performance but being able to sustain playercounts up to 60 is almost a necessity for us, and we've had no problems with that on A2.

With Arma 3 we run into trouble with coop missions as playercount exceeds 30 on Altis, we've not experimented much on Stratis. With 20-30 players we see 40-25 SFPS. With 30-40 players and 70-150 editor placed AI, SFPS is around drops to around 10-5. With 40 players we go below 5 into the realm of completely unplayable. No amount of tweaking server settings, such as MaxMsgSend and min and max bandwidth, will alleviate the issues, or in fact have much discernible impact at all. Pure TvT missions run much better, 35 players still can get 50SFPS on a good day, and we had one instance of 50 players and 5SFPS.

Unfortunately not being able to sustain anywhere near our A2 playercounts in A3 is starting to make A3 seem like not an option. For us the issue isn't with missions or scripting but with a combination of 30+ playercounts and seemingly any 50+ number of AI.

As a strange sidenote, it seems dead players in spectator mode have much, much smaller impact on server performance than live ones.

Will happily post example missions if so desired, but as far as we can tell it really isn't up to specific missions, but in general coops with decent playercounts.

[Tankbuster 1733]

And yes, it does feel like destroyed buildings are a part of what loads up the CPU, but this is hard to measure/quantify and just destroying an entire city in single player didn't seem to reproduce the issue.

http://feedback.arma3.com/view.php?id=7136

[Gudsawn 93]

I run a public server that averages around 18 players. Not sure how useful data from our box would be, but I'd be very happy to contribute none-the-less. We're very much in the same boat that it's hard to convert some of our prior A2 members to join A3 when they get such bad performance in MP (due to poor server frame-rate).

HARDWARE

i7-4770

32GB DDR3 RAM

200Mbit/s line

Eagerly awaiting your test mission Terox.

Edited November 13, 2013 by GDSN

[.kju 3239]

Only Altis

Only COOP

This is not true.

PvP only 30vs30 on Stratis can get the server to very low FPS too.

Also very low FPS for clients in PvP only (Wasteland, AAS, etc) is a big issue.

In addition there are several reports of way too high traffic sent by the server.

It is just probably different issues.

Main issues:

1) In A2/OA the same was fine. A3 is unplayable.

2) No tools to allow the community to narrow it down in a reasonable way.

Thus: Give modders the tools to identify and avoid/solve performance bottlenecks

[nicolasroger 11]

I remember that in A2 there was a lot of soldier warping issues. Specialy when you looked at then from a far distance. I didn't notice this at all in A3 (wich is great!)

My guess is that now every client is forced to be sinc to the server (or wait for it for a particular task inside it's rendering cycle), so that when the server FPS goes that, the client FPS goes that too (with a CPU that stay almost idle because it is in fact waiting for the server)

Please correct me if I am wrong, but really I never noticed any warping issue in A3.

[.kju 3239]

This change was introduced in 1.6x in OA. No relation to A3 issues.

[nicolasroger 11]

Alright good news then.

I hope they find a solution for the multiplayer problems.

[nik21 1]

Actually, it seems like the same issues are affecting clients as well (though it may be different issues). When there are lots of players and units client FPS goes down even with minimum graphic settings and/or looking at the sky, so it's not graphics that are slowing them down - It's purely CPU (which was backed up again by the situation becoming less troublesome after I overclocked by CPU by 30%). It's just that if your client's CPU is weaker than your server's, then the clients will see poor FPS before the server FPS becomes low, and vice-versa. *Something* is eating up the CPU way more than it seems like it should be, and this affects both clients and servers.

The issue is not coop-only either. It can happen on large enough PvP missions as well, though it seems to be a lot more apparent on coop missions, maybe because they simply have more units/vehicles active and/or more buildings destroyed.

And yes, it does feel like destroyed buildings are a part of what loads up the CPU, but this is hard to measure/quantify and just destroying an entire city in single player didn't seem to reproduce the issue.

Having a significant amount of AI vehicles active in single player (more than would normally be active in a similar situation in a PvP mission with only player-controlled vehicles) didn't seem to cause nearly as big of a performance hit as was observed while playing the PvP mission with 20+ players, though the impact on performance was still noticeable.

Pretty much this...

Client fps with 30 or more players often go below 30 or even 20 which is very close to being unplayable, and it's not affected by the rendering at all, you can turn your video settings all the way down and look in the sky, your fps will still stay around 20 (that's at least what players expierience on most of the servers), even though all cores of the CPU are below 50% load. That was one of the biggest issues Arma 2 had and is IMO the #1 issue Arma 3 has. The Multiplayer performance is just a complete game killer. And playing with less than 20 players on a 280 km² map is not that fun either.

[windies 11]

I was on a server with one other person and we were running around on Stratis just talking in TS while waiting for other people to hopefully come on. After 30 min to an hour the performance went from about 25-28 fps down to 10-12 fps as a client and he also said he experienced the same thing. This was Stratis, not Altis and we didn't move into any trigger area's as we pretty much stayed in the base since there was no one else on, ironically we just sat and chatted about the poor MP performance while waiting for people.

It seems like there is some issue happening after a period of time of a mission running that is causing problems. It's exacerbated by player counts and AI counts obviously as I'm sure it's also exacerbated by scripts and such too. The only specific thing is that it seems that you have to have some entities present, like AI or editor placed objects, an empty map devoid of placed entities doesn't seem to cause the performance issue's as much or at all. It seems to be more of an issue with multiplayer and either the netcode, simulation of entities across the network or the syncing between client/server.

You can also see it happen in single player although to a much lesser degree because SP missions are generally somewhat of a fixed length and less dynamic than multiplayer scenarios. The longer the mission runs the more performance degrades, which makes me think it's something to do with simulation of entities.

[bvrettski 10]

I can only play with good framrate on servers with a maximum of 30 players on. So it's funny when BIS says this on their official website:

"ARMA 3 WILL PLAY HOST ONCE MORE TO THE ARMA SERIES' MASSIVE MULTIPLAYER PLAYERBASE WHO, WHEN COMBINED WITH THE ABILITY TO CREATE CUSTOM MISSIONS AND CONTENT, ARE PLAYING A VAST ARRAY OF OFFICIAL AND USER-GENERATED MISSIONS AND GAMETYPES DAILY. FROM 2-PLAYER CO-OP MISSIONS TO 60 VS 60 COMPETITIVE WARS, ARMA 3 FEATURES SOMETHING FOR EVERYONE."

I would love to see if it's even possible to play 120 players on one server.

THIS^^^^

We could end this discussion and start blaming mission designers / coders if BIS could show us the single mission that was "playable" on a remotely hosted server on Altis, for more than 1 hour, with 100+ players. I say it can't be done and I'm pretty damn sure they aren't willing to risk the embarasement of trying it much less claiming its doable with the game in its current form.

Lets face it..the game is about 50% done and still closer to a beta than a final product.

Now who else, besides me, is excited about campaign episode #2..or #1 for that matter?:)

Edited November 15, 2013 by Bvrettski

[pils 49]

Tzzzz man there are some really impolite posts on this Thread. Be glad the devs care about this thread that much.

Main problem seems to be those coop missions with lots of AI. Don't know why they're so popular, because when I want to play against dumbass-AI, I can play SP.

Well anyway, there are some really good CTI, TDM and SC server around (those without AI) where I can play for hours with 50 players having decent 30-40 fps.

But there is some truth. MP needs some improvements. Haven't seen a server with more than 60 players that runs smooth. Usually I stop playing if the fps drop below 20. And I really would like to play with more than just 40-50 players at the same time. And yes, fps seem to drop over time. On some of the TDM servers are planned restarts because of that. But as long as the mission takes less than 3-4h it's been fine.

Edited November 16, 2013 by pils

[galzohar 31]

Well, if you want to test if the game can even handle 100 players with *some* mission, all you have to do is make a mission that automatically deletes all dead bodies and let them go at it in a 50 vs 50 TDM with no extra objects/vehicles/ai. If that doesn't work - Nothing will.

[fred41 42]

Well, if you want to test if the game can even handle 100 players with *some* mission, all you have to do is make a mission that automatically deletes all dead bodies and let them go at it in a 50 vs 50 TDM with no extra objects/vehicles/ai. If that doesn't work - Nothing will.

... yes, this is a good aproach.

Just start test(s) without extra stuff like, a lot of mission objects, scripting (except object cleaning), but with many players (> 60) and observe the performance over time.

If this test show constant performance over time, the next step could be adding AI, while lowering the player number and observe performance over time again.

It should be possible to isolate the source of performance loss this way.

An important point, often overlooked in this discussion, especially when comparing A2 and A3 performance is, that the AI in A3 is alot more "authentic" and powerfull now.

I think such progress would not be possible without increased resource consumption.

Maybe it could be a solution, to provide 2 different, switchable code sets for AI calculation, one simple set (A2 level) and a second more complex set (A3 level).

Mission designers, then have the choice, ether use a lower number more authentic AI, or a higher number but simpler AI.

... just an idea ....

[TSAndrey 1]

Well, if you want to test if the game can even handle 100 players with *some* mission, all you have to do is make a mission that automatically deletes all dead bodies and let them go at it in a 50 vs 50 TDM with no extra objects/vehicles/ai. If that doesn't work - Nothing will.

Some coder already tried 120 players on his CTI version, I was there when they were testing that. FPS was relatively fine, but the server started lagging and players warped around...

[dr. hladik 231]

Will try to provide some profiler for dev. version of DS.