Multiplayer not in playable state

[pawelkpl 29]

;2555147']Wrong thread for SP vs MP. BI designed A3 with SP focus in mind and so far post alpha+beta+release they still focus on SP.

No one serious is saying they should drop SP' date=' yet MP needs dire attention. That's all there is to it.[/quote']

I agree, If they want to make nice campaign or SP missions it is ok but I do have wise question for BI, which mission / mod boosted Arma2 sales years after release ? Was is any SP campaign , mission, etc. or was it dayz MP mod ? Can we play campaign with friends / other players ? MP is left for user made missions with tonnes of game bugs and problems , It makes me sad, very sad... What I'm trying to say is I would like to see any word from BI about MP focus , do they see problem , what they want to do, how we can help, etc.

Edited November 11, 2013 by PawelKPL

[dr. hladik 231]

Agree with Terox

We are running 2 Servers with a heavy modified (but not really optimised) version of Teetimes Warfare with 50 slots and around 40-50 AIs. It runs good (~20 FPS) but if we are adding a bit more AIs the Framerate drops really fast

Hi,

I'm really sorry, but this does not really help us to identify the problem.

We really need something more specific. Some steps how to reproduce it.

[Hud Dorph 22]

Hi,

I'm really sorry, but this does not really help us to identify the problem.

We really need something more specific. Some steps how to reproduce it.

Reproduction steps

1. Set up a dedicated server

2. Put on a popular coop mission (like domi/invade & annex/warfare/etc on altis)

3. Turn on server fps monitor

4.Wait till it reaches 20players and above and watch the server fps

Its really as simple as that.

Edited November 11, 2013 by [HUD]Dorph

[Dwarden 1125]

Dorph;2555180']Reproduction steps

1. Set up a dedicated server

2. Put on a popular coop mission (like domi/invade & annex/warfare/etc)

3. Turn on server fps monitor

4.Wait till it reaches 20players and above and watch the server fps

Its really as simple as that.

totally useless info (in terms of reproduction) ... you just mention 3 MP missions with gazzilion builds each ...

1 mission ' date=' no cripleware scripts (aka reasonable well MP optimized), no addons

create FT ticket

+ details about your server hardware and software and bandwidth

+ upload the mission to that ticket

+ add graph from Arma Server Monitor [url']http://forums.bistudio.com/showthread.php?155881-Arma-Server-Monitor-(very-small-but-useful)[/url]

•simulation cycles per second average (FPS)

•simulation cycles per second minimal (FPSmin)

•condition evaluations per second (CPS)

•number of alive players (PL#)

•number of alive server local AI units (AIL)

•number of alive server remote AI units (AIR)

•number of missionObjects (OBJ)

•amount of allocated physical memory (MEM)

•name of running instance

•name of running mission

post URL to the ticket

---------- Post added at 14:38 ---------- Previous post was at 14:33 ----------

Ours is the same, Plays great for days as long as you dont go above 10 players and whilst your input Dwarden is appreciated, saying its WIP is a little to late when the clan members we are losing week on week because of this are going either backwards to A2 or leaving us for better multiplayer games. Whilst we expect to have some drifting within our Multi Gaming clan, how are we to be expected to keep players when the MP side of Arma 3 is broke.

As Terox mentioned, we too are on stable and tbh the rpt spam doesnt help and before you say use No Logs, how does that help us play test our mission? Then add in the fact the Rpt spam is BIS related errors still present in a released Version of the game? As far as im concerned this game hasnt as yet reached a Beta stage nor reached anywhere near what we were lead to believe the game should be.

Rather than wasting time looking at how shadows form from the camping chairs or rays of sun light... please make the game more playable or lose what people you have left

it's way harder to debug and track MP issues than SP / low level engine ... that's why i said WIP

also MP improvements should be considered ongoing / all-time (before, when and after release of game) work (aka whole active support cycle)

also please stop comparing shadows and other rendering issues to MP, done by different programmers

(don't expect all those who work on rendering to work on MP or AI too :) )

---------- Post added at 14:50 ---------- Previous post was at 14:38 ----------

What I'm trying to say is I would like to see any word from BI about MP focus , do they see problem , what they want to do, how we can help, etc.

everyone who is with us for some years knows i'm trying to focus on MP

e.g.

https://dev-heaven.net/projects/cis/wiki/Dwarden%27s_Top_100_MP_issues_list

http://community.bistudio.com/wiki/Dwarden's_Sandbox

and much more in 'internal services you can't see' ...

but realize that in past the 'human resources and finite time was bigger obstacle (hence why isn't everything already fixed or added to game :) )

expect us to focus on MP improvements but i will try avoid to promise miracles (tho i would do all i can for that to happen)

[xeno 234]

totally useless info (in terms of reproduction) ... you just mention 3 MP missions with gazzilion builds each ...

1 mission , no cripleware scripts (aka reasonable well MP optimized), no addons

create FT ticket

+ details about your server hardware and software and bandwidth

+ upload the mission to that ticket

+ add graph from Arma Server Monitor http://forums.bistudio.com/showthread.php?155881-Arma-Server-Monitor-(very-small-but-useful)

How about you BI guys set up a public server yourself with missions where you think they are well MP optimized?

(Though I would not limit it to "well optimized" MP missions but also others so probably it would also result in some tools for content makers to find out why a mission runs bad).

By setting up a server yourself you could add whatever debug/log output is needed, test changes, etc, etc. It would be much more effective than telling server owners to deliver some data which might not be usefull at all because they are quite limited in what they can do (just think about RPT files filled with engine (error) messages in MP...). And honestly, it's not up to server owners or content makers to find out what's going wrong inside the engine (how will you find out anyway without proper tools). Otherwise it'll take much too long as MP currently is quite a mess (

).

So bascially like you do for dayz.

Xeno

[.kju 3245]

Let me repeat also in here..

Give modders the tools to identify and avoid/solve performance bottlenecks

http://feedback.arma3.com/view.php?id=12658

everyone who is with us for some years knows i'm trying to focus on MP

But who else inside the company? :(

Edited November 11, 2013 by .kju [PvPscene]

[Dwarden 1125]

How about you BI guys set up a public server yourself with missions where you think they are well MP optimized?

(Though I would not limit it to "well optimized" MP missions but also others so probably it would also result in some tools for content makers to find out why a mission runs bad).

By setting up a server yourself you could add whatever debug/log output is needed, test changes, etc, etc. It would be much more effective than telling server owners to deliver some data which might not be usefull at all because they are quite limited in what they can do (just think about RPT files filled with engine (error) messages in MP...). And honestly, it's not up to server owners or content makers to find out what's going wrong inside the engine (how will you find out anyway without proper tools). Otherwise it'll take much too long as MP currently is quite a mess (

).

So bascially like you do for dayz.

Xeno

this is planned too (after the dedicated data server package)

[GottyPlays 10]

Hi,

I'm really sorry, but this does not really help us to identify the problem.

We really need something more specific. Some steps how to reproduce it.

here are some steps: power up your pc, open steam, download arma 3, install it, open up multiplayer tab, play

[das attorney 858]

How about you BI guys set up a public server yourself with missions where you think they are well MP optimized?

(Though I would not limit it to "well optimized" MP missions but also others so probably it would also result in some tools for content makers to find out why a mission runs bad).

By setting up a server yourself you could add whatever debug/log output is needed, test changes, etc, etc. It would be much more effective than telling server owners to deliver some data which might not be usefull at all because they are quite limited in what they can do (just think about RPT files filled with engine (error) messages in MP...). And honestly, it's not up to server owners or content makers to find out what's going wrong inside the engine (how will you find out anyway without proper tools). Otherwise it'll take much too long as MP currently is quite a mess (

).

So bascially like you do for dayz.

Xeno

↑↑↑This

this is planned too (after the dedicated data server package)

Good to hear you are planning to do this, but to be honest Dwarden I'm suprised this was not the first thing that was done when Alpha was released in March; so you could get some reliable data on how the netcode and engine hold up in MP.

I reported some Alpha crashes when playing I&A (and sent rpt plus logs). The guy managing the ticket asked me to see if I get any crashes when playing "Escape from Stratis" or one of the other BIS missions.

How can I get that data when no-one plays those missions? Shall I host official missions myself and wait for people to join? (would be waiting a long time seeing as no-one wants to play them).

You guys need to be doing this yourself imo - the sooner, the better.

[windies 11]

I don't get how we are supposed to give you repro steps when you basically say you won't accept pretty much any community made MP missions as proper repro because of scripts and such when that is the whole "advantage" of the RV Engine and ArmA in general. Sounds like you're trying to cherry pick feedback. Especially given the lack of any official content that's larger scale or meant for serious coop sessions.

Also agree that this is stuff that should have been done back in Alpha/Beta stages, and not 3-6 months after release or who knows how long really before it's actually addressed with more than "wip" and "we'll get to it" responses.

[SavageCDN 231]

How about you BI guys set up a public server yourself with missions where you think they are well MP optimized?

(Though I would not limit it to "well optimized" MP missions but also others so probably it would also result in some tools for content makers to find out why a mission runs bad).

By setting up a server yourself you could add whatever debug/log output is needed, test changes, etc, etc. It would be much more effective than telling server owners to deliver some data which might not be usefull at all because they are quite limited in what they can do (just think about RPT files filled with engine (error) messages in MP...). And honestly, it's not up to server owners or content makers to find out what's going wrong inside the engine (how will you find out anyway without proper tools). Otherwise it'll take much too long as MP currently is quite a mess (

).

So bascially like you do for dayz.

Xeno

+100000 really this should have been done in the Alpha or at least Beta stages... too difficult to rely on server operators to provide all necessary information

[terox 316]

Getting this back on topic

Thanks for getting involved with this thread Dr. Hladik

The issue is with high numbers of AI on Altis and high player numbers.

I run a server with the following specs

Dual Xeon X5570, 16 GB ram

Colocated at a data centre with 100Mb connection

Although, unfortunately its been many weeks since we had to deal with the issue (Numbers of players have dropped significantly making it impossible for us now to run with more than 20 players)

When i did get a simulation cycle rate of 0 FPS, the cpu was only running at 12%

However the bandwidth was through the roof. (I cant remember the actual values, or whether it was data in our out, but it was very high)

So it would appear the actual issue is with data transfer and this is what was grinding the server to a halt.

This was not just with the public domain missions, but also with custom made missions by members of our community with very little scripting

So this isn't a mission issue, although ofcourse they must have some bearing on it.

To reproduce this i would just create a very simple linear mission with (for testing purposes) a set of params that allows you to remove numbers of AI prior to mission start

For example

Number of AI present

a) 100

b) 200

c) 300

d) 400

e) 500

f) 600

That way you can find the limit threshold

Create a very simple linear mission, eg walk down a road clearing ai in your path

have at least 40, better 50 players

No triggers, scripts, modules etc

Make sure its done in a real MP scenario, not via a LAN, so that the data has to be sent and received from a remote location that isnt on your local network

(If i could get that may players together I would do it for you)

and you should see a massive degradation in performance the more AI and players you run with.

For most server admins that I know, they state numbers between 30 and 50 players where the effect starts to be seen

At this stage of the game we should at least have the same performance we had in A2, sadly that is not the case

Edited November 11, 2013 by Terox

[Dwarden 1125]

I don't get how we are supposed to give you repro steps when you basically say you won't accept pretty much any community made MP missions as proper repro because of scripts and such when that is the whole "advantage" of the RV Engine and ArmA in general. Sounds like you're trying to cherry pick feedback. Especially given the lack of any official content that's larger scale or meant for serious coop sessions.

Also agree that this is stuff that should have been done back in Alpha/Beta stages, and not 3-6 months after release or who knows how long really before it's actually addressed with more than "wip" and "we'll get to it" responses.

no, i clealy stated he should pick one where it's happening and upload it to ticket ...

you can't expect us to try randomly pick some mission and HOPE it will replicate ...

[Champy_UK 1]

no, i clealy stated he should pick one where it's happening and upload it to ticket ...

you can't expect us to try randomly pick some mission and HOPE it will replicate ...

I think the general view from this thread seems to be missed, although its hard to replicate the exact reasons, its coming across as a more generalization of Multiplayer missions are struggling with any large number of players on Altis, and some possibly still on Stratis. As some of the posts say... pick any Domination.... All Dominations are the same core mission as im sure you could see for yourselves, minor tweaks with VAS or different Flag poles or signage at the base are the only changes to be seen.

I apologize for my earlier comment but the frustration is borne from the lack of importance towards the direction the things that are breaking the most... after all with out Multiplayer, it wont matter if the suns rays show or the chair casts a shadow because no one will be playing to see it.. hope that makes a little more sense :) And to me thats the last thing i want to see happening.

Terox, and or BIS .. im sure by private invitation between the people in this thread and trusted friends of, we could muster 50 players to fill a server to prove and hopefully get to the bottom of these/this issue. I know my clan mates and associates would be happy to oblige. Just a thought... Anything to sort it and be productive

Champy_UK

[windies 11]

no, i clealy stated he should pick one where it's happening and upload it to ticket ...

you can't expect us to try randomly pick some mission and HOPE it will replicate ...

The issue isn't about specific missions, but rather about the performance of multiplayer as a whole and the scaling of server and client resources in regards to multiplayer, even in reference to singleplayer. All you need to do is replicate the variables, the environment is less important.

[own3d75 10]

Hi Dorf ;]

I rly love yr server [hud] arma-3.dk domi...

but as y said, sometime its impossible to play, coz fps drop...

I hope that BI will help y/us with that issue

otherwise lot of ppl will drop instead fps - game

its 2013 [soon 14] mp is most imported aspect of gaming 2day, so plz folk, just accept that fact

cheers

Gj Dorf!!!

[hellfire257 3]

Missions like Domi tend to munch a lot of resources. Using something like that for repro is crazy because of all stuff running. It would be nearly impossible to nail down the issue.

[Alcatraz968 10]

Missions like Domi tend to munch a lot of resources. Using something like that for repro is crazy because of all stuff running. It would be nearly impossible to nail down the issue.

You ever try a server with a "Gasp" Headless client? Alot better FPS on those.

[galzohar 31]

I run a server with the following specs

Dual Xeon X5570, 16 GB ram

...

When i did get a simulation cycle rate of 0 FPS, the cpu was only running at 12%

A X5570 (or i7) running at 12% is basically just a single thread running at 100% since it can run 8 threads (so 100% / 8 = 12.5%, and task manager rounds the display up/down). Or did you mean you have 2 of those CPUs on a single machine? In any case, it will likely not help anyway, as the game just isn't multi-threaded enough, especially in its problematic areas that seem to cause the CPU bottleneck. That is, it doesn't matter if a lot of stuff can be done in parallel, if there is one thread that takes a lot of time to complete and all the other threads are stuck waiting for it to finish simulating a frame.

Getting a faster CPU or overclocking helps a lot more than getting more cores, but obviously if high-end CPUs have a hard time only in multiplayer then something is messed up. I just can't believe netcode can be that demanding on any reasonable CPU, and my guess is just a bug in the simulation that only occurs with high player counts. It does seem like missions with the same number of AI and units do run slower when more players are present, though this could use more testing and verification by making sure it's tested on missions that don't scale the number of AI and vehicles based on the number of players.

Regarding scripting, most of the "bad scripting" you can do should not affect performance at all as it have been tested that no matte how much you try to force the game to run scripts (with some rare exceptions that generally only happen when you intentionally trigger them), it simply won't let you spend much CPU power on the scripts themselves. Of course, a script can still run a lot of commands that create objects that require simulation, which of course can result in more CPU power being required for simulation. However, it's still not CPU time spent on actually running scripts!

Missions like Domi tend to munch a lot of resources. Using something like that for repro is crazy because of all stuff running. It would be nearly impossible to nail down the issue.

The real problem is though the difficulty of the engine to deal with "a lot of stuff".

Scripting is unlikely to have anything to do with it. While I can't test it for you due to lack of players (lose of interest, not necessarily because of performance but it definitely did help convince them to quit), it should be pretty easy to test...

[progamer 14]

this is planned too (after the dedicated data server package)

Look forward to playing on the first official BI hosted server! :)

Hopefully some good optimization data will come from it.

[Lokyi 10]

It's pretty simple really, even running a HC you'll find both processes top out at around 2Gb RAM each (so you'll only really struggle if you have a server with 4Gb RAM or less), but CPU utilisation is bugger all, we hit about 30% across one server instance + HC (and that's only running an E3-1225v2).

The primary bottleneck with Arma is simply network bandwidth. Always has been. Ours has gigabit backend *allegedly* but from what I can work out, that's not both ways, upload (the important bit) tops out around 15Mb/s. Sometimes we'll get 25Mb/s during off-peak periods.

And of course, the more players you add to the server that more you have to split that bandwidth up between them...

And on top of that, there's network performance of the clients is some consideration too. Although UDP does negate some of the problems, such as error checking (or rather lack of on the network layer) etc.

Edit: Actually the only other thing I could suggested is somehow streamlining/cleaning up the memory usage/memory recovery. It take a couple of hours, but once the server does hit the 2Gb wall, that's when it seems to slow down significantly regardless of the amount of clients. But either way, with over 30 clients connected, it does chug pretty hard.

Edited November 12, 2013 by Lokyi

[trnapster 12]

Hi,

I'm really sorry, but this does not really help us to identify the problem.

We really need something more specific. Some steps how to reproduce it.

Once we have a better optimised mission in place we'll experiment with player vs ai counts and we'll be able to give some useable feedback

We just have to eliminate the flaws of the mission first

[terox 316]

am actually working on a 50 player mission, with params that can set AI numbers from 100 to 600

This is gonna be pretty clean, as in minimal scripting, albeit a logging script to the rpt

I should have it by tonight

It should be more than adequate for stress testing this particular issue and giving the players something to do instead of being bored waiting for the server admins to log their backend data.

I would also suggest that any admins who is interested in helping with this issue gets Fred's Arma3 server monitor installed

http://forums.bistudio.com/showthread.php?155881-Arma-Server-Monitor-(very-small-but-useful)

Am not sure if this logs data yet (Does it?)

I will also put together a simple "What to do" for any admins out there so we all follow the same testing procedure and therefore the data we send back can be compared

The rpt log script at present is

a 60 second continuous loop.

Each pass it logs

• 
  Time
  No. of players
  No. of server side AI
  Average FPS (over last 16 frames)

The admins are going to have to manually log data in and out using #monitor (Unless anyone knows how we can log this automatically

Should we all get wireshark logs ?

and if so what filters should we use

What other data can we collect that will be useful ?

Edited November 12, 2013 by Terox

[Hud Dorph 22]

Pls remember when testing this - the most important issue will be having 20+ clients on and have the session running for some hours.

[Llano 11]

this is planned too (after the dedicated data server package)

Have you done this to any other game (Arma 1/2 or OFP)?