TEST: 128 players? possible at Arma3? Sunday, 28.7.13

[Blue1 10]

Hey Guys, *RESULTS ADDED*

I would like to know if it is possible to get 128 players PLAYING on "my" CTI (pvp, not AI command style) Map.

Also i want to test / compare to one or two other (also vanilla) maps/missons.

I'd love to see whats happening if around 100+ ppl connect to my server (enough power & bandwith for actually every other Game on the Market^^)

right after start of a map, what perfomance it will be... and what is happening over time.

EDITED POST STARTS HERE:

It will be 2 different Maps, as we would like to compare a "nearly Vanilla King of the Hill" i made today, with our actual Teetimes Warfare Release v7.0

The "King of the Hill" will just have one Zone, triggered / occupied by who is in it, respawn delay of 10 seconds and "all standard Editor Infantry-Classes".

So it is just Infantry PVP contesting one Zone, without buyable Stuff or anything.

We are doing this to compare "Vanilla" 128 Players "alive and kicking asses" to our current and optimized build of TTWF.

AND OF COURSE - i want you to leave feedback in this Thread - about the Performance you get (please add Hardware- and Connection-Infos of your PC)!

RESULTS:

Blue1's 128Players Loadtest, Sunday 28.7.2013

Teetimes Warfare, reworked & optimized by Blue1 and Fred41 - Version 7.1

Measurements:

• name of running instance

• name of running mission

• number of missionObjects (OBJ)

• amount of allocated physical memory (MEM)

• simulation cycles per second average (FPS, also called Server-FPS)

• simulation cycles per second minimal (FPSmin)

• condition evaluations per second (CPS)

• number of alive players (PL#)

• number of alive AI units (AI#)

Server:

CPU: i7-3770 @ 3.4GHz

RAM: 16 GB

HDD's: SATA 6GB enterprise drives

Connectivity: 1 GBit on 100Mbit

I've used the unchanged 64-Players CFG-Settings from my usual Server

(as there were never Desyncs/Lags).

MinBandwidth=78643200;  		//75Mbit   
MaxBandwidth=104857600;	// 100Mbit 104857600

MaxMsgSend =512;
MaxSizeGuaranteed = 640;
MaxSizeNonguaranteed = 384;

MinErrorToSend = 0.0035;
MinErrorToSendNear=0.00901;

MaxCustomFileSize=160000;
//1600000 is 160k

adapter=-1;
3D_Performance=93750;
Resolution_W=0;
Resolution_H=0;
Resolution_Bpp=32;
Windowed=0;
serverLongitude=12;
serverLatitude=48;
serverLongitudeAuto=12;
serverLatitudeAuto=48;

Battleye enabled

verifySignatures = 2;

-------

Regarding to this Thread from Terox, WIKI Infos and "what is known":

Quotations:

Frames vs Simulation cycles

The Client machine measures the speed of the simulation in Frames per second

(Basically how many pictures are recreated on the screen every second.

*edit: thats your Personal Computers FPS

The Server however does not run any graphics, so it has a

unit of measurement of Simulation cycles per second.

The maximum the server is allowed to run at is 50 Simulation cycles per second

regardless of hardware

For ease of understanding I will refer to this as "Server FPS"

The actual Server FPS is dependant on

• The number of players connected

• The number of scripts running and the weight those scripts are placing on the cpu

• The number of AI that are in the mission

• The hardware capabilities, CPU,RAM Hard drive I/O and bandwidth settings

• The state of play (Combat causes greater load than non combat stages of the mission)

For each Server Simulation cycle, apart from actually running the simulation, the engine attempts to check the status of objects in the mission, compares that to the previous check and updates according to the minerrorsend, sending the required data to the clients in "packets" of data and also receiving data back from them.

If there are too many checks to be done or it cannot send the data out quickly enough or receive the data quickly enough, the Server-FPS will reduce.

EDITED/commented from HERE, regarding to v7.1 of TTWF by Blue1&Fred41

If it reduces below 20, the clients will start to notice a slightly deterioration in Gameplay.

At an Server-FPS of around 15 it is still very playable, but you may recognize visible "hops" sometimes.

At an Server-FPS of 10 the mission is STILL playable - but you can "feel" the reduced Performance of the Gameengine now, in "hectic" situations like fights or while driving/flying fast.

At an Server-FPS around 5 - its somehow playable but not very much fun in crowded / contested Areas.

Because every joining/leaving player adds more JIP-Lag/Desyncs which cumulate to a drastic Performancedrop to the Gameengine.

Also it seems, Steamticket-Checks can't work properly if the Engine is that Busy.

You should typically be aiming to achieve 25 to 30 FPS (*Edit: on our 128pCTI we are going for 15-20)

The aim of optimising your server is to try and maintain as high an FPS as possible.

To do this you need to take into account the following

1. Your hardware limitations

2. How many virtual servers you will be running consecutively on the same box

3. Your bandwidth limitations (Connectivity)

4. How many players each server will host for

5. The type of Missions you will be playing which includes the Features and

6. the weight of the scripting

Just to mention it again.

While most of the players reported very good CLIENT-FPS, even on crowded places or in heavy fights - a few complained about "lag" and bad performance.

If you encountered bad performance - please state your PC-Configurations and what "tweaking"-options you are already using etc.

Because - as this test has approved, the majority of players haven't faced any unusual problems or perfomancedrops even at high playercounts (70+)

(except the Gameengine on the Server had a big drop, see below) and could play very smooth!

Okay, after the Values are clear so far here the Results and conclusions.

At the beginning of the first Test i missed to configure the RAMDISK correctly,

which leads to more, unwanted HDD-Activity.

1)

74 Players - after some Time of playing

2) After a MAP-RESTART (#restart & #reassign), not full-restart of the server.exe:

Already 87 Players and 48 AI but the same Values as above.

This states the logical conclusion of:

- more Objects needs more calculations -> reduces Server FPS

- more players are not dropping Server-FPS as that much, as more Objects do

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128cti2.JPG (138 kB)

3)

Now, after a full-restart, again after some playing and on the correct configured RAMDISK

100 Players, 46 AI

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128cti3.JPG (180 kB)

4)

Few minutes later:

110 Players, more objects on Map, more MEM used

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128cti4.JPG (171 kB)

5)

A short while after another Full-Restart:

88 Players and first Screenshot showing the Networkusage: 32Mbit/s -> plenty capacitys left

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128ctiramdisk1.JPG (130 kB)

6)

Same, a few seconds later, 91 Players - for CPU-Usage in Details

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128ctiramdisk1cpu.JPG (166 kB)

7)

Same Game, a few minutes later.

Not 100% sure what happened - if it is a Script or Steamcheck causing this, as it doesn't occure in a special Timeframe.

As a Guesswork i'd say: Engine sums up everything and doesn't know what to do first ;)

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128ctiramdisk2cpu.JPG (173 kB)

8)

So - after locking the Server to prevent Join-Lags, a few seconds later Desync disappeared and everything was back to normal / playable!

*link to picture

9)

Another (full restarted) Game later that evening:

Heavy drops -> locked

10)

-> and luckily the automatic cleanup seemed to get in effect coincidentally too -> all good again -> unlocked and a few minutes later we had 102 Players playing STABLE and somewhat smooth!

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128ctiramdisk99-2.JPG (127 kB)

11)

But as the drops came back, again "out of nowhere", at 115 Players - i stopped the test to change to the Vanilla-Map:

https://raw.github.com/Blue1Arma3/Blue1Arma3/master/128ertest/128ctiramdisk99-3.JPG (130 kB)

12)

I wanted to compare a "nearly Vanilla" Map to our CTI.

Just a few Markers to define where the Action is, respawns, but no big scripts, features or other Stuff.

Just 128 Players PVP, Infantry Only - fighting with Standardequipments on a small Area.

Thanks again to Fred41 for this idea and everything =)

Link to Picture

13) 74 Players

Sadly we didn't reached more than 74 Players on the "Vanilla", and many people left too, because we missed to fix up a Bug in the Respawnscript - but i think the Results are clear.

At 54 Players - the Server-FPS was affected the first time!

Even 75 Players - nearly had no Effect.

That means to me - it should be possible!

With a well optimized Code, every Script checked for his particular Job and the effects on the Server/Client and as well good Loadbalance.

Assuming the dedicated Server runs on SSD or even better on RAMDISK and has a Headless-Client working (this will be one of our next Steps) !

BUT - as we can see on the CPU-Usage / Core-Balancing, i think there still needs to be some more work done by BIS, to allow Hosters really use their Servercapabilitys!

thanks for participating and cya soon for the next tests, as soon as we have a new - even more optimized version of our Mission.

best regards,

Blue1

-------------------------

continuing old OP

Of course i will provide result here and to BIS and stuff,..because...

I just think - in the Year 2013 it should be possible to release a Multiplayergame...

(and sorry, for me there is no reason to play Arma3 "offline" / SP, as it capabilitys are so big)

... thats abled to provide atleast 64 players stable and smooth MP-expirience which is NOT possible at the current status.

So, i want to help to find out what we or BIS can do against it.

Many provided very usefull hints - but i think it is time to proof / test what is really happening on a server.

Also i want you to suggest which tools to use / which values i could measure to proof that something is very wrong at the current state of MP.

And last but not least - i want to invite you to join this "loadtest" at Sunday, 28.7.2013. Afternon? Evening?

There will be restarts, mapchanges and everything you don't like if you would go "just play" ...

but i'd really like you to join me.

details will be announced if i get some response on that idea.

thanks and best regards,

Blue 1

Edited July 30, 2013 by Blue1

[Valken 622]

Why don't you just open up a public server with 128 slots and see how long it takes to fill it? Wouldn't that be easier?

Also, you need to list your timezone. This forum has a global presence.

I'm just curious for academic reasons. :D

[Blue1 10]

Hi Valken,

yes, you are right - timezones, thats something i haven't thought of :rolleyes:

it is Central European Timezone.

well, i think chances are better to get some results in a "managed test", while just opening a server and wait if it fills with 128 ppl, i guess,

won't work the same as having a time and date where everybody knows that it is just a test, several restarts / reconnects could happen and so on.

It also would make it easier to compare different settings / Maps and their results.

*PS: but as you are the first one replying after a few days - i guess this test will never happen anyway :(

okay - was just a idea^^

Edited July 20, 2013 by Blue1
*forgot somthing

[ric 1]

ill try to make it :)

[ratszo 17]

Most players don't read the forum.

128 slots sounds great.

Build it and they will come.

Good luck!

[ck-claw 1]

Most players don't read the forum.

128 slots sounds great.

Build it and they will come.

Good luck!

^^This -once you've got a few ppl on and the server is on the 1st page of the browser, it won't take long ;)

Edited July 20, 2013 by ck-claw

[sEi 10]

+1 im in

[fred41 42]

... great idea, i will be there too :)

[markocro 66]

count me in :)

[ylguf 1]

I see this test as being pretty meaningless unless certain conditions are met. right now you are trying to get a server full and see if it runs good or not regardless of uptime or population. first there needs to be a function that sends info from the server to client and back to server (preferably a know amount data). and this function should tracked with BIS_fnc_codePerformance, otherwise the only info you are acutally collecting is a best guess of if its running good or bad. then this function should be ran on lan with 1 client to get a accurate minimum time the code takes to run. so you track the function runtime/ server uptime/ and population and you actually have a numbers to graph population vs performance, and time vs performance. because without this how your server is running is subjective, it could be running good (in comparison to something that runs like crap). the important thing is the actual amount of data being processed and sent because even if the server is running bad with 128 people you could calculate the optimum population

Edited July 21, 2013 by ylguf

[dale0404 5]

Yeah, what he said!!! ^^ :confused:

If I am around I will jump on.

[Blue1 10]

*bump* & i am currently at checking the "vanilla" to 128 players - just to let u know ;)

[ratszo 17]

Just finished 2+ hours on your server..., near full and silky smooth gameplay. Helos were tracking perfectly, not a hint of lag-spiking.

Saw very few 'Null' kicks.

Is this 'Ram-disk' magic?

[ric 1]

Just finished 2+ hours on your server..., near full and silky smooth gameplay. Helos were tracking perfectly, not a hint of lag-spiking.

Saw very few 'Null' kicks.

Is this 'Ram-disk' magic?

how many players?

[ratszo 17]

how many players?

64. I joined at 62/64, very little join/disconnect scrolling. Seemed like server got full and people stayed in.

[Blue1 10]

Hi Ratszo,

nope, the Stable-server isn't on Ramdisk anymore, since Tuesday XD - it is some other Magic ;)

[Blue1 10]

*** but it will be, for tomorrows tests.

stay tuned!

[Blue1 10]

Okay Guys - 128 PlayerTest is on its way - see announcement in OP please =)

And of course, any suggestions are still much appreciated!

Cya Sunday =)

[Walt71 1]

nice , great idea :bounce3:

[tomclothier 1]

Ill see if people from my community are up for coming on :)

[Blue1 10]

Starting now =)

[GottyPlays 10]

I went on your server, god it was laggy as hell! :S vehicles were desyncing a lot, people warping. I would love to see a smooth 128 players server, but im afraid that it wll be hard to do on arma 3

[dale0404 5]

The test wasn't a success then?

[scorch_052 127]

Yeah I think 64 players is pushing it for most servers. In order to host a 128 player server youd have to have a beast of a server with lighning-fast internet. I'm talking like dedicated server CPUs (Intel Xeon) and 1-2GBps internet.

[Blue1 10]

tests were a full success - we found out a lot.

I will post screenshots / statistics and values and stuff later of course!

We could play WITHOUT ANY LAGS/DESYNCS and "nearly healthy" server-fps up to 90 people on our CTI-Map. (and yes, i will proof it!)

Problems occured mostly due to a lot of people repeatedly join/leave/join/leave ... and due to this, some performancespikes while over 100 players happened. After locking the server Desyncs etc. were removed and all went "well" (as WELL as it was possible with that numbers)

Yeah I think 64 players is pushing it for most servers. In order to host a 128 player server youd have to have a beast of a server with lighning-fast internet. I'm talking like dedicated server CPUs (Intel Xeon) and 1-2GBps internet.

Server wasn't anyhow busy at all ,...25% CPU-Usage and around 30Mb/s Network,.. but wait for the Screenshots and proven Facts.

I think, later this evening i will find time to add everything.