constantly increasing memory usage

[dlegion 98]

EDIT AS SUGGESTED:

----------------------------------------------original message-------------------------------
hi guys!
i've noticed that in my MP dedicated mission, the initial RAM usage is about 1100Mb.
but checking it with " #monitor 1 " command....but its slowly and constantly increasing!!
it goes up about 1 Mb every 2-3 seconds, but keeps increasing until reach 2000Mb...and at that point server choke to death!!
any idea what can be or how i can track the trouble ?
thanks!
----------------------------------------------original message-------------------------------

PROBLEM DESCRIPTION:
dedicated server memory usage keeps increasing at a variable rate apparently influenced by number of AI in mission.
it keeps increasing more or less constantly and endlessly. on 32bit "arma3server.exe" when over 2000MB it start to choke, decreasing down to 2 FPS,
and ultimately crashing with htis error:   https://ibb.co/k30HBa

3D EDEN CREATED TEST MISSION from NEW (empty):
(lets ignore my mission for now, and focus on standard test): Altis, 6 WEST rifle quads (8 men each) 4 WEST combined tank platoon (4 tanks each), 3 playable units,
 all with a move waypoint and a cycle waypoint. (can double the numbers to accelerate the process).

WHAT I DID EXACTLY STEP BY STEP TO REPRODUCE THE PROBLEM:
launch TADST v3.0.0, leave default settings (for testing purposes) checked box "persistent battlefield" and "auto-init",
selected "arma3server.exe" and tried also with "arma3server_x64.exe". (different starting memory usage, but both were increasing).
selected previously created TEST mission. Launched the server. Opened the windows task manager.
keeps monitoring the "arma3server.exe" process memory usage. Notice slow, but pretty constant increase in memory usage.

WHAT OTHER TEST I DID:
i tested my mission with all my scripts (and they are many) except the ones spawning AI. in my mission all AI is spawned by script, so there were 0 AI.
server did not had increasing memory usage.
i tested an old mission version (more than 6 months ago), that i'm sure it was running fine for at least 24 hours
(i remember playing on it with friends one night, then forgetting server up, then the following evening playing again on it with friends).
the test now showed a memory usage increasing.
i tested other missions made in the past, i dont have any documented previous check on them about memory usage, but now its increasing at various rate,
apparently related to numbers of AI.

MY HARDWARE / SOFTWARE / CONFIG ecc...
i5 4440 3.10ghz, 12Gb ram, 64bit, NO-SSD, connection: ADSL 7Mb upload 0.4Mb download.
(its the same configuration i had at the time of the 24 hours server running with no problem)
i always used 32bit "arma3server.exe" except for the very latest test.
LINK to latest version of my mission (here problem is accentuated due to spawn-despawn units based on player distance):
https://mega.nz/#!KtZ2GLiQ!-PxDcaPxXqjeuAaSeUEk2YT1e-UDZf22TXr0dhrVS5E
 

Spoiler

-------------- T E S T S -----------------------------------------
DWAR 20/07/2017 NO SCRIPTS                   = 848 MB seems stable
DWAR 20/07/2017 with R3F_LOG & MAG_REPACK    = 849 MB seems stable
DWAR 20/07/2017  +   minedetector            = 849 MB seems stable
DWAR 20/07/2017  +   Dfps                    = 849 MB seems stable
DWAR 20/07/2017  + Dclean & mineMarker       = 850 MB seems stable    // another test : 888.260 at 12:58 -- 895.288 at 13:11 -- 897.788 at 13:58 -- 897.884 at 14:06 -- 897.760 at 14:14 -- 897.836 at 14:25

DWAR 20/07/2017  +   GMOhostages             = 875 MB slowly increased to 912 in 15 minutes (spawned 9 units + 1 tank)....when deleted (by script after 30 minutes) memory stays at 910Mb
DWAR 20/07/2017  +   HQBtroops+HQRtroops     = 928 MB increasing 3MB every 1 minute (spawned 4 APC with crew)
DWAR 20/07/2017  +   GMOreset                = 1000 MB increasing 1Mb every 10 seconds (spawned 51 soldiers + 1 tank + 1 helicopter)
DWAR 20/07/2017  +   Dvehspawner.sqf         = 1110 MB constantly increasing 1 Mb every 1-3 seconds (spawned 2 helicopters, 2 airplanes, 8 civs on foot, 8 civs in cars, 2 apc, 2 soldiers, 2 boats)

 

 

TEST MISSION WITH FEW AI PLACED FROM EDITOR (48 AI + 16 tanks):
908 MB at 08:16
919 MB at 08:50
921 MB at 09:00
924 MB at 09:36

------------------------------------------------------
DLEGION DWAR mission (MORE UNITS AND THEY RESPAWN):
1080 MB at 12:03 (in game time)   
1250 MB at 12:13
---2----------
1100 MB at 12:03
1256 MB at 12:13
1264 MB at 12:15
1490 MB at 13:12
1536 MB at 13:26
1611 MB at 13:41
-------3-----------
1240 MB at 00:24
1501 MB at 00:58
1546 MB at 01:03
1621 MB at 01:24
1662 MB at 01:35
-------4------------
1150 MB at 00:33
1208 MB at 00:38
1283 MB at 00:47
1349 MB at 01:00
... 6 hours later (or less), server crashed with "out of memory" error

----------------------------------------------------

AN OLD XENO DOMINATION mission (FEW AI COMPARED TO THE OTHERS, BUT STILL PRESENT):
1684 MB at 09:54
1759 MB at 10:03
1786 MB at 10.13
1800 MB at 10.40
1853 MB at 14:45

 

 

thanks for any help!

[HazJ 1289]

Try running tests on scripts / functions (code), etc...

https://community.bistudio.com/wiki/BIS_fnc_codePerformance

More information here:

https://community.bistudio.com/wiki/Code_Optimisation

Also, server specs? Network... Bandwidth, etc...

[dlegion 98]

thanks man !
well my PC specs (i use it also for dedicated server) :
i5 4440 3.10Ghz , 12 GB ram, Nvidia 660 2Gb, no SSD. connection is a 7Mb upload and 0.40mb download.
the point is that in the past i never noticed this trouble...i rememebr keeping the server up for the whole night, and in the morning it was running fine!
now in less than 1 hour it choke!!

i'm reading the WIKI...but not sure to understand how should i use or start it! plz make an example !
for info, i tried removing all my scripts....and it still increase in memory usage over time, but more slowly!
its the most tragic problem since i started to make missions...hope to get whats wrong and cure it!!

EDIT: in the meantime i've started on a clean configuration (mission without a single script and without AI) that seems stable (at least in 3 minutes no change).
i'm re-enabling one by one all my scripts....an writing the changes...

[dlegion 98]

huge amount of test....and i'm pretty sure its AI !!
every script adding AI cause an instant increase in MB usage,and thats obviously ok, but also a costant slow increase in MB usage over time!!
i tried also to spawn some AI from ZEUS, same result! MB usage slowly increases! here the results:
 

Spoiler

DWAR 20/07/2017 NO SCRIPTS                                  = 848 MB seems stable
DWAR 20/07/2017 with R3F_LOG & MAG_REPACK    = 849 MB seems stable
DWAR 20/07/2017  +   minedetector                             = 849 MB seems stable
DWAR 20/07/2017  +   Dfps                                          = 849 MB seems stable
DWAR 20/07/2017  + Dclean & mineMarker                 = 850 MB seems stable

DWAR 20/07/2017  +   GMOhostages                          = 875 MB slowly increased to 912 in 15 minutes (spawned 9 units + 1 tank)....when deleted (by script after 30 minutes) memory stays at 910Mb
DWAR 20/07/2017  +   HQBtroops+HQRtroops            = 928 MB increasing 3MB every 1 minute (spawned 4 APC with crew)
DWAR 20/07/2017  +   GMOreset                                 = 1000 MB increasing 1Mb every 10 seconds (spawned 51 soldiers + 1 tank + 1 helicopter)
DWAR 20/07/2017  +   Dvehspawner.sqf                      = 1110 MB constantly increasing 1 Mb every 1-3 seconds (spawned 2 helicopters, 2 airplanes, 8 civs on foot, 8 civs in cars, 2 apc, 2 soldiers, 2 boats)

 

its really tragic !

EDIT:
as ultimate test, i created a new mission, placed just 3 playable units, and 6 rifle squads (8 men each) and 4 tank platoons (4 tanks each). gived them a move waypoint and a cycle waypoint.
started around 810Mb...slowly but constantly increasing !! i'm getting desperate!!

[Grumpy Old Man 3546]

Try those tests with vanilla A3 and no scripts, then work your way up, scripts first, then mods.

 

Cheers

[dlegion 98]

totally agree, but i was not clear...i already did that!! 
 

1 hour ago, dlegion said:

EDIT:
as ultimate test, i created a new mission, placed just 3 playable units, and 6 rifle squads (8 men each) and 4 tank platoons (4 tanks each). gived them a move waypoint and a cycle waypoint.
started around 810Mb...slowly but constantly increasing !! i'm getting desperate!!

 

[Grumpy Old Man 3546]

What exactly do you mean by "server choke to death", does it get non responsive? Heavy fps drops?

Is this on local hosted MP server or dedicated?

 

Might be a guess out of the blue but did you check your RAM?

Other stuff on the server running fine?

 

 

Cheers

[dlegion 98]

Server "choke " because it end avaible RAM. After 2000mb it start to damatically drop fps .

Its dedicated (i didn t found a way to use "#monitor 1" on other.

I thinked it too (about ram check )...but only start eat ram with ai....without ai ram usage stays stable !

[dlegion 98]

...i left the server up and running for the night....this morning here is what i found:
https://ibb.co/k30HBa
i think the problem is way more serious than i imagined...

[Grumpy Old Man 3546]

So even with vanilla A3 and as you stated only 3 playable units, 6 rifle squads and 4 tank platoons with a few waypoints the same thing happens?

No mods and other scripts running?

Any other programs running?

 

 

Cheers

[dlegion 98]

yep, exactly !
at first i was thinking was a script, but here is a 100% pure and clean vanilla....and everyone can test this too !
just open windows task manager anche check increasing memory usage of the "arma3server.exe"  process!
i'm scared now!!

[kremator 1065]

There really has not been any serious explanation by BIS devs as to WHY spawning AI can cause this.  There are SO MANY RPT messages that really make no sense to us mortals about what is going on in the mission regarding spawning AI.

[dlegion 98]

!!! so its a "known" problem???
.....how can that be??? its a game-breaking problem!

[sarogahtyp 1108]

Most common solution is to restart server after a few hours.

I do it this way:

https://forums.bistudio.com/forums/topic/209091-server-fps-recommendation/?do=findComment&comment=3222062

[Grumpy Old Man 3546]

That's not what kremator said.

If this issue was common the devs would be all over it, but I really can't replicate it.

Been running local hosted MP with 6 rifle squads, 4 tank platoons and 3 playable units all with waypoints on cycle for 30 minutes, memory keeping stable at 1.6GB, no rises, no drops.

Dedicated was the same, using a bit more than 450MB ram. Still no usage fluctuation.

 

22 minutes ago, kremator said:

There really has not been any serious explanation by BIS devs as to WHY spawning AI can cause this.  There are SO MANY RPT messages that really make no sense to us mortals about what is going on in the mission regarding spawning AI.

 

Taking the opportunity to post a few of my favorite rpt messages, all vanilla units:

 

13:18:13 soldier[B_HeavyGunner_F]:Some of magazines weren't stored in soldier Vest or Uniform?

Very ambiguous.

 

13:18:16 Client: Nonnetwork object c35a0f00.

No idea what that's supposed to mean, lan network? Road network? What?

 

13:18:32 No speaker given for 

Huh?

 

Then thousands of lines similar to this:

13:21:18 Link to 9b0989ad (Obj-305,216:429) not released

 

Back on topic:

Check the hardware for errors or set up a virtual machine and do a clean install and see if the error persists.

Also like @sarogahtyp mentioned, a restart once in a while doesn't hurt.

Buddy of mine had performance issues back in XP times, after a bit of troubleshooting I asked him to type systeminfo to cmd console and tell me the system start time: was running for almost 2 months.

No idea how xp managed to not crap itself way sooner.

 

Cheers

[dlegion 98]

ahh good to know! thanks grumpy !!
well all i can do to start is format PC, reinstall everything and try, but how can i check if is hardware error ?
thanks again !

EDIT:
PS: just to be 100% sure, can you please try with my mission? doublecheck its always a good thing!
thanks!

https://mega.nz/#!KtZ2GLiQ!-PxDcaPxXqjeuAaSeUEk2YT1e-UDZf22TXr0dhrVS5E

[kremator 1065]

@dlegion it isn't a hardware issue.  It is typical Arma!  Restarting a server isn't a good idea if you have a long run planned.

 

@Grumpy Old Man those errors are really annoying I'll agree.  Wouldn't take very long for a dev to comment (and not the 'oh don't worry about them' message!)

[dlegion 98]

@kremator yep... my mission was planned to run constantly.... sure i planned to reboot soon or later...but should work lets say for a week without blowup! (at this rate it can't stand 8 hours !!)
just to be sure i'm asking right now a couple of friends to run theyr own dedicated to test that thing and be sure if its me or arma!

anyway...my mission is there....that has more AI plus that AI get deleted and respawn based on range from players...so it should accelerate the process and make it more noticeable!
if you wish to test it would be a double check for everyone!
thanks guys!

EDIT: last test:
my mission with all scripts EXCEPT the ones spawning AI (so 0 AI in mission):
test : 888.260 at 12:58 -- 895.288 at 13:11 -- 897.788 at 13:58 -- 897.884 at 14:06

[sarogahtyp 1108]

8 minutes ago, kremator said:

 Restarting a server isn't a good idea if you have a long run planned.

Therefore you can use databases like MariaDB together with extDB3...

[dlegion 98]

@Grumpy Old Man ...did you had chance to test with my mission too ?
i'm still waiting for test of my friends before format C: ...anyway feels really strange that happens only in presence of AI !
if possible i wish to have more confirmation that its just me having this problem!
thanks!

try doubling AI numbers, and check with windows task manager....its clearer from there!
PS i'm using TADST to launch dedicated server...may it interfere somehow ?

[HazJ 1289]

I could be wrong here, been a while ha... Going years back, I'm sure AI use a whole core (or two? one I think). Servers use two cores. I'm sure all cores are used when hosting and playing (not sure!). Keep in mind, you are hosting and playing on the same machine however, this shouldn't be a problem when placing a few groups and a vehicle group in the Editor. I have no idea why you are having this issue. No clue. Sorry if I have provided any false information. (:

Edit: Ah, just noticed you are using the arma3server.exe via TADST tool.

[dlegion 98]

yep, but the test have been made with no arma3 client (OFF) , just server.
and i rememebr times (6-9 months ago), where i started my mission at evening, played with friends, forgot server open, found the server already open the following evening, and played again without restart!
i didnt changed anything hardware or software!
@HazJ anyway....can you do a huge favour, and test this yourself? Altis, 6 rifle squads, 4 tanks platoons, on dedicated? and maybe my mission too?
really thanks !

[das attorney 858]

It's natural that the server needs more RAM as the mission goes along.

 

It needs to keep track of all the AI, their damage, ammunition etc etc, plus any vehicles spawned plus damage to houses, trees, opened door states, what the weather's doing and so on.

 

I'm looking at task manager on my mission and server is at 3.8 Gb.  When you say it chokes, do you mean crash or FPS comes to a standstill?

 

Arma uses paged memory and I notice you have no ssd so maybe it's paging from the HDD and slowing down.

 

Also, you mention hitting 2GB then you say it chokes, have you limited the server memory by  command line?

[dlegion 98]

On 8/8/2017 at 8:22 AM, dlegion said:

...i left the server up and running for the night....this morning here is what i found:
https://ibb.co/k30HBa
i think the problem is way more serious than i imagined...

server went down to 2 FPS, then crashed.

i'm getting really confused....
1 @kremator says its a know problem
2 @Grumpy Old Man says to have tested and memory stay still
3 @das attorney says its normal, and this explanation can somehow work with Kremator one.
4 i personally remember to have left server up with my mission for a whole night and day, and was running fine, about 6-9 months ago. i tested that old mission version now, and now has the same problem....
so honestly i think the simplest way is do like Grumy did, just test on different computers/connection....because on mine for sure this happens!

i use TADST to run dedicated, and i didnt changed anything about server....where i can look for that settings?
thanks for your help guys !

 

[das attorney 858]

Can you upload your server rpt to pastebin?