Server FPS Problems (Fixed with player Re-log)

[Rabid Squirrel 149]

Server FPS Problems (Fixed Briefly with player Re-log)

 

I've been encountering particularly poor performance on my group's Arma server over the past couple of weeks. We run in-house Co-op missions that we've tried to optimise as best we can, we try to not have more than 40 concurrently simulated AI at any one time.

Our Average Player Count is between 10 - 20, the example below we had 14 players attending the mission. Personally I feel the server should be performing much better than this under the circumstances, but it appears to be consistently terrible, and I'm running out of things to try. Generally I'll use spawn scripts in combination with DynamicSimulation for the AI, with the below example I pre-loaded all of the mission AI (this was an experiment as the past couple of missions we experienced some horrible desync with AI spawns), and prepped them for Dynamic Simulation. All the AI were garrisoned (placed in buildings with "path" disabled), Dynamic Simulation Distance was set to about 75 meters. In addition, as the fighting centered around the town of Zargabad, Server and Player viewDistance was set to 300.

 

 

This is a graph of server performance:
The orange line depicts AI number (although most of that AI would've not been simulated at the time).

The grey line depicts server fps

The blue line is number of players

 

This is the same graph minus the AI (gives a closer look at the performance versus player number).

 

Now what the Topic is actually about:
We observed a couple missions ago that a full Arma reset and re-log onto our server would fix the clientside fps problem. As you can see above we did this twice during last night's mission, in an effort to fix the problem. It worked somewhat, made the mission playable, and as such we completed it. What was strange was that the second one seemed to "jump-start" the server, frame rate was perfect and what I expected for the whole mission, except it was only during the last 30 minutes. It is important to note that the mission does not get restarted, we leave at least 1 person in the server during the re-logs.

 

I suspect that perhaps the server is caching some information for the player, and when the player performs a full Arma restart, that cache is discarded and a new one created.

 

So my question:
Has anyone ever encountered this behavior before? If so, is there a way to refresh the server without logging everyone out and back in again?

 

Additionally, players during re-log noted that Arma was using about 8 gigs of ram before the shut down (which seems to indicate a memory leak).

 

Additional Information:

Spoiler

Server Specs:

Xeon Processor @ 2.4GHz (Unsure Exactly Which, Attempting to Get that Information Now)
6GB RAM
200GB SSD

Arma running on the server - Linux 32-bit

Scripts Running:

A rudimentary condition check that was using waitUntil with a 10 second reiteration.

A script that spawns a flyby jet every 3 minutes, it flies over the town, and then deletes on the other side.
 

Mods Running: (Serverside)
@cba_a3\;
@cup_terrains_core\;
@cup_terrains_maps\;
@sa_pmc\;
@rain_tex\;
@rhs_afrf\;
@rhs_usf\;
@rhs_gref\;
@rhs_saf\;
@ryanzombies\;
@ace3\;
@task_force_radio\;
@asr_ai3\;
@ares_mod_achilles_expansion\;
@ace_compat_rhs_afrf\;
@ace_compat_rhs_gref\;
@ace_compat_rhs_usf\;
@hlc_core\;
@hlc_wp_aug\;
@hlc_wp_fal\;
@hlc_wp_g3\;
@hlc_wp_mg3\;
@acex\;

 

Thanks in Advance for the Help!

 

[Tankbuster 1747]

In my experience, Xeons are not good for Arma servers. Great webservers, but not nearly as good as a high clock i5 or even an i3.

[Rabid Squirrel 149]

3 minutes ago, Tankbuster said:

In my experience, Xeons are not good for Arma servers. Great webservers, but not nearly as good as a high clock i5 or even an i3.

Is there a reason for this? Or is it just one of those "Arma things"?

[Tankbuster 1747]

One of those things I'm afraid. Your 'extensive' mod list is a possible cause too

[Dedmen 2716]

5 minutes ago, Rabid Squirrel said:

Is there a reason for this? Or is it just one of those "Arma things"?

Can confirm. Your processor is absolutly not-optimal for Arma.
Arma mostly runs single threaded meaning over 95% of the processing to be done cannot be spread over more than a single CPU core.
Xeons usually have many "slower" cores. But Arma needs fewer "faster" cores. Like a Modern I7 in the upper range of 3-4Ghz single core clock.
I would recommend 3.8Ghz and higher with a modern CPU. You are on 2.4Ghz... That's almost half.

I am actually running all mods you are running plus some extra on my Linux.
On a Intel Core i5-2400@3.1Ghz on one and Intel(R) Core(TM) i3-2130 CPU @ 3.40GHz on my second server. And even they are non-optimal.
Our FPS rarely dips below 50.

[Tankbuster 1747]

By way of comparison, I have a fast clocked i3 CPU that comfortably out performs that Xeon as an arma server.

[Rabid Squirrel 149]

5 minutes ago, dedmen said:

Can confirm. Your processor is absolutly not-optimal for Arma.
Arma mostly runs single threaded meaning over 95% of the processing to be done cannot be spread over more than a single CPU core.
Xeons usually have many "slower" cores. But Arma needs fewer "faster" cores. Like a Modern I7 in the upper range of 3-4Ghz single core clock.
I would recommend 3.8Ghz and higher with a modern CPU. You are on 2.4Ghz... That's almost half.

 

2 minutes ago, Tankbuster said:

By way of comparison, I have a fast clocked i3 CPU that comfortably out performs that Xeon as an arma server.

 

Okay, thanks a bunch guys, part of me knew this was probably the case but I was hoping that I was just doing something silly. Guess it's time to start looking at other options.

[djotacon 190]

You have a perfect setup to run Arma 3 server forgot the coments like "Arma mostly runs single threaded", this is a false statement based in ignorance and the absence of authentic knowledge about how arma 3 server works.

 

You must focused on this: "poor performance on my group's Arma server over the past couple of weeks." prior of that your server works well and the problem isnt your server.

 

You mus test any installed mod and even your own O.S to discard any problem before making a hardware investment.

 

The Xeons are much better than a I3 or i5 to run Arma 3 server.

[freak69 5]

Disregard what people are saying about your CPU / Hardware let us have a closer thought about what comes into the server when a player joins.

 

any custom added eventhandlers on the player objects?

using "triggers"?
 

I'm asking because triggers as much as eventhandlers run unscheduled on each frame (at least triggers do, not sure if it counts for ALL eventhandlers too, but for some for sure.)

Then there is the thing with many locally run script functions which are supposed to be just that, local but they have a global effect. Best example -> createVehicleLocal. You can use locally created bombs to kill people or vehicle to "ram" into global vehicles.
 

Another thing I recognizes is any variation of "getPos" (position, getPosASL, getPosATL, object modelToworld, ".." and so on..)

Those seem to have a little to no "performance block" when using them once. But they have this tiny effect on everyone within the network.
This is a huge problem if you have triggers depending on position on anything. Because then every single player does that "tiny effect" onEachFrame and well you see where I am going?

Also; anybody looked into how dynamic sim "really" works? by any chance it is getting positions in a very fast loop or onEachFrame?

I mean that position thing is a really big problem as it seems like once you ask for the position of a remote object (not talking about getting the position of a local object like "player") it is not telling you the position which should be saved in your local memory as it was sent through the network already by some kind of scheduler. No it's asking for it on the network and therefore causing a delay + workload.

---

Could as well be a callExtension call done pretty often; depending on the extension and the number of calls 

 

[terox 316]

I also don't have any performance issues with xeons.

I cant compare a xeon performance with for example an I7 because I have never tried it.

I tend to buy 1u dual socket motherboard rigs, afaik I7 doesnt support this, so it has to be xeon.

Arma also utilises more than 1 core. AFAIK, the AI processing is on a dedicated core

 

I also run what is now a very old set of xeons, 2 x X5675 @ 3.9Ghz

We can run with something like 50 players and around 200 AI with that in a vanilla environment with no caching or Headless Client.

While running more servers (At least 3 sometimes 4)

 

So  this is going to be down to either

• Poor server configuration
• Inadequate bandwidth
• the template you are using for your missions
• maybe one or more of the addons
• issue with some of the clients (The clients can drag the server performance down)

 

When you have issues like this it would help if you listed hardware, connection and config files in your posts.

 

Good luck

[Rabid Squirrel 149]

Hey all,

Back Again, so we haven't switched servers as of yet (multiple reasons for this, one of which being the cost of high performance servers around here), anyway. I've been taking steps to try and boost performance, and for the most part it has worked fairly well, and the performance is reasonable on average. Take the below for example:

Same graph minus AI count for better readability:

 

As you can see, performance was quite decent for the majority of the mission (averaged as 32 server fps), client fps was even better for the most part.

 

Now the strange thing is that this problem specifically:

On 11/8/2017 at 10:49 AM, Rabid Squirrel said:

We observed a couple missions ago that a full Arma reset and re-log onto our server would fix the clientside fps problem. As you can see above we did this twice during last night's mission, in an effort to fix the problem. It worked somewhat, made the mission playable, and as such we completed it. What was strange was that the second one seemed to "jump-start" the server, frame rate was perfect and what I expected for the whole mission, except it was only during the last 30 minutes. It is important to note that the mission does not get restarted, we leave at least 1 person in the server during the re-logs.

 

I suspect that perhaps the server is caching some information for the player, and when the player performs a full Arma restart, that cache is discarded and a new one created.

didn't occur. Which means that that problem in particular is not related to the server itself, but the mission, mods, or players. Of course, that only narrows it down some.

 

Thanks to everyone for the suggestions so far, I am looking into all of it. If I do happen upon the particular error I will post my findings.

[Rabid Squirrel 149]

Solved

 

So it would appear we've found the solution to our problems, still early in testing, but it appears the performance degradation over time has stopped completely.

 

 

The cause: 

corpseManagerMode and

wreckManagerMode

 

We had our suspicion that this was causing problems, so I pulled both of them from our description.ext and wrote my own script for dead body removal and wreck cleanup. We've now had 2 full 2 hour missions and 1 shorter 1 hour Zeus mission, and the client frame rate did not degrade at all. Additionally, server frame rate was more than reasonable, with clients having between 30 and 40 fps on average.

 

I'm still unsure why these managers would've been degrading our server and client frame rate, or why the frame rate recovered on re-log.