Arma3 and the /LARGEADDRESSAWARE flag (memory allocation > 2GB)

[ric 1]

It's me in a windowed client measuring my memory usage while I'm connected to a server. I don't have a way to measure server performance right now, though I would be very interested in seeing the memory usage of the dedicated server.

i was running a CTI mission on my ded server, the usage started out at 400MB and by the time the FPS dropped into the teens with just me on the server after and hour the total commit of RAM was 830MB give or take a meg

[windies 11]

i was running a CTI mission on my ded server, the usage started out at 400MB and by the time the FPS dropped into the teens with just me on the server after and hour the total commit of RAM was 830MB give or take a meg

Well in reality, a dedicated server shouldn't use a ton of memory because it has no assets to load like texture's and models. It doesn't have to stream the entire world on demand or anything.

I think the issue with multiplayer is that you can easily surpass 4gb commit because you are having to simulate EVERYTHING around EVERYBODY in the server as a client. Even in Single Player though on an empty map, you can see that ArmA 3 can easily surpass a 4gb memory footprint.

[ric 1]

Well in reality, a dedicated server shouldn't use a ton of memory because it has no assets to load like texture's and models. It doesn't have to stream the entire world on demand or anything.

I think the issue with multiplayer is that you can easily surpass 4gb commit because you are having to simulate EVERYTHING around EVERYBODY in the server as a client. Even in Single Player though on an empty map, you can see that ArmA 3 can easily surpass a 4gb memory footprint.

i am not sure if i understood you but from what i have read the way A3 works is that it is syncing individual simulations to provide a coherent environment hence why dsyncs can jump hundreds of frames. the funny part was that while i was observing performance i watched the server flogging 10 different PBO. at the same time as if it was playing the game not being a server.

now hile i was on my CTI server i noted that it was sending out a AVG of 480mbps while i was connected, another player joined and it doubled...which is not a problem if you have the pipe to accommodate the bandwidth need...but imagine the kind of stress put on a server when there a 90 players...that's a LOT of syncing...hence once again it shows that the faster and better the architecture of a single core will give the best results.

now windies I want you to get a nitrogen pot so we can do the some tests ;)

[windies 11]

i am not sure if i understood you but from what i have read the way A3 works is that it is syncing individual simulations to provide a coherent environment hence why dsyncs can jump hundreds of frames. the funny part was that while i was observing performance i watched the server flogging 10 different PBO. at the same time as if it was playing the game not being a server.

now hile i was on my CTI server i noted that it was sending out a AVG of 480mbps while i was connected, another player joined and it doubled...which is not a problem if you have the pipe to accommodate the bandwidth need...but imagine the kind of stress put on a server when there a 90 players...that's a LOT of syncing...hence once again it shows that the faster and better the architecture of a single core will give the best results.

now windies I want you to get a nitrogen pot so we can do the some tests ;)

I was talking more from the perspective of the client. The server would generally have to simulate everything for each client connected anyways. It's not so much crappy servers or crappy missions as much as it's the fact that the engine is not designed or made for these large scenario's in multi player anymore. You can obviously blame a mission for being too large or having too many assets or a server for being too poor for the game or the mission, but is it really the missions fault when it's simply a ported domination or evolution mission from a previous iteration of ArmA that ran fine before with the same amount of assets on the same server?

It's just showing that the engine doesn't scale very well for demand.

[ric 1]

It's just showing that the engine doesn't scale very well for demand.

that's the primary issue for ARMA period. It is incapable of using resource available causing a lot of the performance issues we see. so far from what i have read if the engine was at least in a state of %50 Parallelization you would see a 2x performance increase :)

here are some links if you want to bore yourself to tears :)

http://software.intel.com/en-us/articles/common-best-known-methods-for-parallel-performance-from-intel-xeon-to-intel-xeon-phi

https://www.sharcnet.ca/help/index.php/Measuring_Parallel_Scaling_Performance

http://cs.smith.edu/~thiebaut/transputer/chapter8/chap8-1.html

http://www.kornelix.com/lbench.html

http://en.wikipedia.org/wiki/Amdahl%27s_law#Speedup_in_a_sequential_program

[fred41 42]

@Windies,

i think, the stutters, you observed while flying a jet on altis, are mainly caused by IO (reads) and preprocessing of the data, coming to late and blocking the simulation.

This could indicate a weak (or not existing) player movement prediction.

It should be possible to calculate/predict, where a fast traveling unit, will be a few frames later.

If such a prediction, in an separate thread, would prefetch the data (loading data from file to memory and preprocess) enough frames ahead, such stutters should not exist.

Additional, a lot more relevant data could be hold in physical RAM (workingset of process + system working set), instead of paged on disk, at least if we have a 64 bit OS and enough RAM.

(the virtual adress space and the available RAM is not fully/optimal used on 64 bit OS systems, currently)

[windies 11]

@Windies,

i think, the stutters, you observed while flying a jet on altis, are mainly caused by IO (reads) and preprocessing of the data, coming to late and blocking the simulation.

This could indicate a weak (or not existing) player movement prediction.

It should be possible to calculate/predict, where a fast traveling unit, will be a few frames later.

If such a prediction, in an separate thread, would prefetch the data (loading data from file to memory and preprocess) enough frames ahead, such stutters should not exist.

Additional, a lot more relevant data could be hold in physical RAM (workingset of process + system working set), instead of paged on disk, at least if we have a 64 bit OS and enough RAM.

(the virtual adress space and the available RAM is not fully/optimal used on 64 bit OS systems, currently)

I totally believe it's I/O reads because I can see chunks of texture's and objects missing that are trying to be streamed in fast enough. This is on an SSD with 560mb sequential read speeds btw.

I think the only way to go is 64 bit from here if the program is seriously trying to work with this amount of memory. They can keep trying to maximize 32 bit process usage on 64 bit OS's but that's kind of a redundant operation don't you think?

[fred41 42]

@Windies,

i think i found the reason for the "stutters" while fast flying (moving).

It seems a video memory overload situation, causes a drastical recovery action (to free some GPU memory).

The falling edges in the GPU memory allocation graph, are identical with the "stutters" in game.

To verify, you can use process explorer (sysinternals/microsoft), double click on arma3.exe process and select tab "GPU Graph", graph "Dedicated GPU memory".

Greets,

Fred41

I totally believe it's I/O reads because I can see chunks of texture's and objects missing that are trying to be streamed in fast enough. This is on an SSD with 560mb sequential read speeds btw.

I think the only way to go is 64 bit from here if the program is seriously trying to work with this amount of memory. They can keep trying to maximize 32 bit process usage on 64 bit OS's but that's kind of a redundant operation don't you think?

Edited September 19, 2013 by Fred41

[tonygrunt 10]

@Windies,

i think i found the reason for the "stutters" while fast flying (moving).

It seems a video memory overload situation, causes a drastical recovery action.

The falling edges in the GPU memory allocation graph, are identical with the "stutters" in game.

To verify, you can use process monitor, double click on arma3.exe process and select tab "GPU Graph", graph "Dedicated GPU memory".

Greets,

Fred41

I have the same experience, today I was flying across Altis and my GPU memory would reach around 2.5-2.6 GB and then drop by 0.9-1 GB down. Sudden stutter and memory would start filling again till it reached the tipping point again.

[fred41 42]

... my video card is a 2GB VRAM model and the "recovering" occurs at ~1.6GB (horizontal red line).

1.6GB (2.6 for you) seems to be a alert level, forcing a radical freeing of all vertex/triangle/texture data and refill with a uptodate (smaller) data set.

This procedure needs a lot more time (500-1000ms) then a normal video frame (~20ms).

I thing this wrong behavoir could be fixed, by a fine tuned player movement prediction, combined with a smoother video memory usage monitoring and adequate buffer updating/clearing.

I have the same experience, today I was flying across Altis and my GPU memory would reach around 2.5-2.6 GB and then drop by 0.9-1 GB down. Sudden stutter and memory would start filling again till it reached the tipping point again.

Edited September 19, 2013 by Fred41

[windies 11]

Latest dev build has gotten rid of the stutters while flying. Not all of the stutters though while playing just the massive stuttering that was happening while flying, though memory usage is all still the same as what I posted before. @Fred, that same behavior occurs with ArmA 2 for example, if it hits a certain VRAM cap it flushes textures and reloads them. You can do the same thing I believe by hitting shift and - and then typing "flush", I believe thats the routine for it. I think it's so that they don't run into the old memory overflow or d3d overflow problem that they had in ArmA and I believe that the original ArmA was around the time they came up with that flush trick, then eventually added it as an automated routine in the engine to prevent the overflow issue.

[tonygrunt 10]

Latest dev build has gotten rid of the stutters while flying. Not all of the stutters though while playing just the massive stuttering that was happening while flying, though memory usage is all still the same as what I posted before. @Fred, that same behavior occurs with ArmA 2 for example, if it hits a certain VRAM cap it flushes textures and reloads them. You can do the same thing I believe by hitting shift and - and then typing "flush", I believe thats the routine for it. I think it's so that they don't run into the old memory overflow or d3d overflow problem that they had in ArmA and I believe that the original ArmA was around the time they came up with that flush trick, then eventually added it as an automated routine in the engine to prevent the overflow issue.

Never had this problem with Arma 2, my older gtx480 would top at 1.5GB(max) and stay around there. Sometimes opening & closing the map would trigger a memory flush but nothing like Arma 3 where I witnessed an almost 1GB drop while flying.

Maybe its graphics card/memory/driver specific.

[fred41 42]

Altis and especially kavala, looks really awesome with viewdistance/objectdistance >= 3000 and texture at "ultra".

It's wonderful and i don't know a fly-simulation looking that nice.

Also, this observed stutters is not really that annoying, because it is not very frequently (depending from travel speed and view/object distance settings).

But i think, there must be a simple reason for that data accumulation in VRAM, maybe just a weak buffer allocation/release strategy.

If i hoover 100m over kavala center, viewdistance 3000, and turn my view some times around, i see a VRAM level of ~1.0GB.

Now all requiered terrain and object data are placed in the VRAM buffers.

If i start moving now, some new data have to be transfered to the VRAM buffers, to satisfy my viewdistance settings continuously.

But there are still "old" data in the buffers, which are now outside my view distance radius and could theoretical be deleted (buffers released).

In real, some buffers should be hold for a while, maybe i change my fly direction and the old data will be needed again.

But here, it seems to much data are hold for a time period to long, which causes the accumulation (allocated buffers > released buffers).

I am sure, a agile player movement prediction, should be able to predict the needed data for the next frames and allocate/fill and release the buffers accordingly.

And we could enjoy this really nice altis just a very little bit more :)

Edited September 20, 2013 by Fred41

[ramius86 13]

"fred41 2 days ago tbbmalloc full virtual range, 0.5GB reserve"

your tbbmalloc.dll is changed on github, differences than before?

One thing, both client and server got 32 gb ram: so it's better maxmem at 8192 or not use this param ? :confused:

Edited September 21, 2013 by ramius86

[fred41 42]

... good find :)

There is a minor change in tbbmalloc.dll.

The change is related to armas internal memory manager. I observed, that arma periodically checks, if memory allocation via .dll is more than 1GB.

If .dll reports allocation above 1GB, arma starts to request .dll to trim cache to free virtual address space (each 50ms).

This makes sense, if virtual adress space is only 2GB (32bit OS), but on 64bit OS we have nearly 2GB more address space.

To "becalm" arma, i just report a lower virtual allocation value, as in real is allocated via .dll.

The reported memory allocation is now scaled to the whole virtual address space and a reserve of 0.5GB.

What means arma now starts trim requests not before only 0.5GB are left in the full virtual address room.

But dont expect a big performance gain, it is only a smal step in the right direction.

Arma still cuts maxmem to 2047MB, so a value of 8192 does'nt change armas memory managers behavoir, currently.

Greets,

Fred41

"fred41 2 days ago tbbmalloc full virtual range, 0.5GB reserve"

your tbbmalloc.dll is changed on github, differences than before?

One thing, both client and server got 32 gb ram: so it's better maxmem at 8192 or not use this param ? :confused:

Edited September 21, 2013 by Fred41

[ramius86 13]

Hi fred41, today with dev branch seems there's a change in tbbmalloc, when you have time to test them keep us updated ;)

PS just see your last malloc for 8 gb, I'm gona testing it right now!

PPS just tried, seems doesn't work, arma don't start at all even with administrator rights

Edited October 21, 2013 by ramius86

[fred41 42]

Hi fred41, today with dev branch seems there's a change in tbbmalloc, when you have time to test them keep us updated ;)

PS just see your last malloc for 8 gb, I'm gona testing it right now!

PPS just tried, seems doesn't work, arma don't start at all even with administrator rights

... yes, you have to set a user privileg additional, because of LARGE_PAGE support:

'Local Security Policy' -> 'Local Policies' -> 'User Right Assignment' -> 'Lock Pages in Memory', set it for your (administrator) arma launch user/group

Thanks for the hint, but are you sure tbb4malloc_bi.dll has changed today? Will check it out soon.

[ramius86 13]

I've done this, for make this sure I've done everyone. I will retry now

[EDIT]

Nothing again, http://i.imgur.com/8DUli4h.png I've selected ALL the users to make sure but nothing.. I've tried with server 2012 R2, win 8.1 and win 7 , game don't even start :(

Edited October 21, 2013 by ramius86

[fred41 42]

I've done this, for make this sure I've done everyone. I will retry now

[EDIT]

Nothing again, http://i.imgur.com/8DUli4h.png I've selected ALL the users to make sure but nothing.. I've tried with server 2012 R2, win 8.1 and win 7 , game don't even start :(

... i think you have to restart your machine, after setting/changing this privileg ... did you?

[ramius86 13]

Uh.. :rolleyes: I was thinking about that but I didn't give a reboot too much influence :p Now works, thanks! I will do some test, need to set maxmem now?

Edited October 21, 2013 by ramius86

[ramius86 13]

I still got many problems with arma3server.exe or arma3.exe to FORCE them to use more than 2 gb... avg 1,3-1,5 gb . Fred do you have some missions ( very heavy one ) to use to test this ?

[fred41 42]

I still got many problems with arma3server.exe or arma3.exe to FORCE them to use more than 2 gb... avg 1,3-1,5 gb . Fred do you have some missions ( very heavy one ) to use to test this ?

... you can have problems ;)

My experience is that armas RAM allocation increases slowly with time (for both server and client). So maybe you just have to play/host for longer time?

BTW: Arma is using ~1.7GB additional system RAM via file mapping api. This amount is not showed in the process virtual space allocation. I think this area is used mainly for internal map terrain/texture/object caching.

Edited October 25, 2013 by Fred41

[ramius86 13]

Ok, probably is due to the few time spent while testing. I've just restarted both servers with this malloc and I will let them alive for some days. Tonight I will see during a coop what my client will do and will monitor with procexp

[DaViSFiT 21]

with what params you start your dedicated ramius? I also just try to improve the server cause he is just using 1.2GB ram and very labile fps when getting under load.

[ramius86 13]

simply add -malloc=tbbmalloc in the params as stated in first fred41's post