Arma3 and the /LARGEADDRESSAWARE flag (memory allocation > 2GB)

[John Kozak 14]

You can see that Task manager only shows 1.5gb in use, but that's only the Private Working Set, so that's only physical memory allocated. Then you see the total commit charge in Performance Monitor and you see it's at 6.6gb and the big red bar shows 43% of my total commit charge in use. So ArmA 3 does use at least 4gb of memory and does fill the entire 32 bit addressing space.

To confirm this with 100% certainty, you can use Process Explorer (look for Virtual Memory) or VMMap (look for "Total") tools. Both tools are official Microsoft ones from Mark Russinovich and helped me not once to debug memory problems.

[dazhbog 10]

If i understand this: http://forums.bistudio.com/showthrea...=1#post2481817 correctly,

the recommendation for a modern system with 64bit OS and 8+GB RAM is:

"dont use maxmem and let arma 3 change internal limits automatically" ???

Yes, that should work very well in most cases (there might be some corner cases of course).

---------- Post added at 09:54 ---------- Previous post was at 09:46 ----------

@ramius,

reaching the 2GB border with altis map should really not be a problem.

Empty mission starts with 1.1GB (client), small missions starts with ~1.3GB, and dependend of the mission and player number it

normally slowly grows to above 2GB (max. seen by me 2.4GB). I use processexplorer from sysinternals for measuring.

Probably the easiest thing is to put a soldier on Altis near Kavala, set high object view distance and look around you in a circle. That will force the engine to load the objects and terrain into memory and based on the view distance it shouldn't be hard to reach and break the 2GB barrier. You can also jump through several places on the island which will force loading of even more objects into memory.

@Dwarden,

yes, thats exactly what i have to assume now.

I tested yesterday with both tbb 4.0 and tbb 4.2, more than 1 hour each (maxmem not used/set).

Both runs with perfect FPS normally between 40 and 50 (vsync on).

Max. memory ~2.2 for each of both tests (BECTI 0.9 on altis).

Also, there is no important difference between 4.0 and 4.2, maybe 4.2 is a tick better (just a feeling).

The difference i observed in the last days (problems disappear), seems to be only caused by improvements/changes in the engine (DEV branch).

PRELIMINARY CONCLUSIONS:

tbb4malloc_bi.dll works well enough, currently

dont use maxmem without a need (64bit OS, enough physical RAM)

Makes sense, I'm glad there are no anomalies :).

[NeoArmageddon 957]

Tested vanilla TBB 4.0 and TBB4.2 on our dedicated server last night. Three players, 180 AI's in approx 140 groups and a lot of waypoints in Kavala. Server starts with usage of 800mb anf 50 FPS. Players walking in towards the AI in Kavala let's the servers FPS drop to 25-40 FPS and memory usage quickly goes up to 1015MB but seems to stay at this level. No noticable difference between TBB4 and TBB4.2.

Also, there is no important difference between 4.0 and 4.2, maybe 4.2 is a tick better (just a feeling).

The difference i observed in the last days (problems disappear), seems to be only caused by improvements/changes in the engine (DEV branch).

As my conclusion I will confirm the above statement for dedicated servers.

P.S. I would be glad to help with more testing on my Win08 dedicated server!

Edited September 12, 2013 by NeoArmageddon

[windies 11]

To confirm this with 100% certainty, you can use Process Explorer (look for Virtual Memory) or VMMap (look for "Total") tools. Both tools are official Microsoft ones from Mark Russinovich and helped me not once to debug memory problems.

Without ArmA 3 loaded:

http://s13.postimg.org/hycur1qmv/Commit_Without_Arm_A3.png (145 kB)

With ArmA 3 loaded at the main menu:

http://s13.postimg.org/9qax61yqf/Commit_Arm_A3_Load.png (177 kB)

ArmA 3 at editor with Altis loaded:

http://s13.postimg.org/sjwq31wyf/Commit_Arm_A3_Editor.png (170 kB)

ArmA 3 while flying around an empty map:

http://s13.postimg.org/x38yoketz/Commit_Arm_A3_Flying.png (163 kB)

You can see that commit goes from 1.8gb to 6.0gb which is a 4.2gb increase.

[Dwarden 1124]

Without ArmA 3 loaded:

http://s13.postimg.org/hycur1qmv/Commit_Without_Arm_A3.png (145 kB)

With ArmA 3 loaded at the main menu:

http://s13.postimg.org/9qax61yqf/Commit_Arm_A3_Load.png (177 kB)

ArmA 3 at editor with Altis loaded:

http://s13.postimg.org/sjwq31wyf/Commit_Arm_A3_Editor.png (170 kB)

ArmA 3 while flying around an empty map:

http://s13.postimg.org/x38yoketz/Commit_Arm_A3_Flying.png (163 kB)

You can see that commit goes from 1.8gb to 6.0gb which is a 4.2gb increase.

what you most likely see is the engine utilizing mapping memory w/o address via OS API

http://www.bistudio.com/english/company/developers-blog/85-breaking-the-32-bit-barrier

Memory with No Address

And then one day, an idea came, as such ideas usually do, from nowhere. As usual with such ideas, once the system is implemented, it seems very obvious, but I never heard about it being used before, therefore I am quite proud to have invented it. I call this technique Non-addressable Memory Store.

The technique is based on using the File Mapping API. Yes, you hear well, the same API which caused problems with Flashpoint, but this time it is used a very different way, using it not to read the files, but as a way to allocate memory:

•on game initialization, enough memory is reserved and committed by using CreateFileMapping API. This consumes the physical memory, but no virtual addresses

•when storing or retrieving a page of data, temporary view is created using MapViewOfFile, which is destroyed again once access is finished. This uses very little of the virtual space (64 KB). Typically only a few pages are mapped into the memory this way at the same time.

Windows is handling this pattern very well, while the space can be backed up by the "page file" in theory, in practice it works in such a way that as long as there is enough of physical memory available, there is zero page file traffic and all information is handled in the physical memory only. The memory kept in this type of storage is not mapped to any virtual addresses at all, it is identified by the offset in the file mapping instead. Addresses are assigned only temporarily, when the content of the cache is read. This is possible thanks to the fact that the data stored in the file cache are location independent (do not contain any pointers).

The Barrier Is Broken

U.S. Navy Photo by Ensign John Gay This technique in theory allows even for a 32b application to use more memory than 32b address space allows - the size is limited only by the possible size of the file mapping, and by the free RAM, therefore with 32b OS it would be possible to manage about 3 GB of data, and with 64b OS even more. This is of not something we do in ArmA, experiments have shown having a file cache size larger than 512 MB brings very little improvement, as we are not reading that much data anyway, but for some other applications it might be a useful option. It is not limited to file caching, but it can be used to store any content, with one significant limitation - the application must never point into a memory allocated this way using ordinary permanent pointers, as the pointers would become invalid once the memory is unmapped.

What this brings to ArmA is that the internal file cache, which is sized between 100 - 500 MB depending on how much RAM you have in your system (or depending on what -maxmem command line argument you use), is now using almost no virtual space at all. Those several hundreds MBs seem to be enough to keep the rest of the system happy. So far it seems to

please note, this blog is from 2008 for Arma 1, so the numbers of MB are now in other 'ranges' ;)

[windies 11]

I know that Dwarden, but thank you for clarifying it some more. I'm not trying to figure out what uses the memory. What I'm trying to get at though is that the game, utilizes 4gb very easily and probably would utilize much more if given the addressing space. I think what we are running into is a situation where the data chunks being streamed are just too large or too numerous to try to be streamed through the file mapping API. With a larger addressing space, e.x. 64 bit addressing, you could pre-load a lot more data and then stream in smaller bits of it using the same non addressing memory mapping you use now or just paging it and calling it when needed using the normal virtual memory system.

[Dwarden 1124]

ye well i think lot of people missing the part that Arma engine uses memory of

directly mapped adressable space (2GB on 32bit OS /4GB on 64bit OS)

+

indirectly maped via OS API which is limited by OS itself ....

but i see the point you trying to claim, that the OS file mapping API might be the choke point (tho i doubt it, especially on Windows 7 / 8 with theirs optimized kernel and memory managers)

[windies 11]

Well there's something there that is causing the stutters and it's much more pronounced the faster you are traveling or rather the more data you are forcing through the pipeline. The only thing I can think of is that the non addressed data is trying to be read into virtual memory much faster than it can handle. It's the only link in the chain that I can see causing that issue, unless there's some sort of bug in the malloc or memory manager.

Another thing I noticed is CPU spikes of 5-10% during large stutters which signals to me that large amounts of I/O traffic is being processed as on an empty map with no AI or ballistics or scripts to calculate, there shouldn't really be anything spiking the CPU usage except file operations. The stuttering got so bad that at times I thought the game was going to crash. This is on an SSD with 8gb of ram and a Phenom II 965 @ 3.8ghz.

Edited September 13, 2013 by Windies

[Dwarden 1124]

may i just check something (usual verify to be sure) , what PCI express mode is you GPU in ?

what is your OS (version,x64/x32), CPU, GPU and chipset (mainboard)?

[windies 11]

PCI-E 2.0 GTX 480

Gigabyte GA-MA770-S3 Rev 2.0 AMD 770 chipset

Phenom II x4 965BE C3 revision @ 3.8ghz

Windows 7 64 Home Premium

[ramius86 13]

Stuttering also for me with max viewdistance and objects over 8000m with or without this dll.

With viewdistance no problem but with viewobjects things gona crazy, especially with jet on altis..

My game stop responding and restart after 5 seconds and so on.. also without change direction or view.. for what hw are made those specs? titan x 3 ? ram usage was always over 2-2,5 gb and gpu ram around 1.8

3930k@4.5

32g@2400

asus rampage iv extreme x79

Ocz revodrive 3 x2 pcie 2x

Sli gtx 680 pcie 3x

Win 8.1 pro

Maybe the game need more speed to read objects..

Tonight ill try restore ramdisk also here but my hdd isnt really slow :p

Edited September 13, 2013 by ramius86

[John Kozak 14]

Without ArmA 3 loaded:

http://s13.postimg.org/hycur1qmv/Commit_Without_Arm_A3.png (145 kB)

With ArmA 3 loaded at the main menu:

http://s13.postimg.org/9qax61yqf/Commit_Arm_A3_Load.png (177 kB)

ArmA 3 at editor with Altis loaded:

http://s13.postimg.org/sjwq31wyf/Commit_Arm_A3_Editor.png (170 kB)

ArmA 3 while flying around an empty map:

http://s13.postimg.org/x38yoketz/Commit_Arm_A3_Flying.png (163 kB)

You can see that commit goes from 1.8gb to 6.0gb which is a 4.2gb increase.

Systemwide commit memory is inconclusive, that's what I was trying to say ;-) For example, the system itself may reserve memory for e.g. drivers, especially Realtek ones, or some other applications (Steam UI, TrackIR, ...)

Using 4.2 GB of address space can't obviously happen for 32bit process even with LAA - only 4GB is possible even on 64 bit system.

Can you repeat the tests with the VMMap or Process Explorer, and look at virtual memory of the process itself?

Edited September 13, 2013 by DarkWanderer

[fred41 42]

@Windies + @Dwarden, thanks for your interesting questions and clarification.

Your discussion was induction for me, to read this:

http://www.bistudio.com/english/company/developers-blog/85-breaking-the-32-bit-barrier

again and this time without to overlook some important details :rolleyes:

@DarkWanderer,

please read the document linked above (if you did'nt already), this cleared something for me :)

(CreateFileMapping forces the OS to pre-occupy RAM without consuming virtual addresses from arma process,

later the arma process can access this RAM in smal windows, via MapViewOfFile)

Greets,

Fred41

Edited October 28, 2013 by Fred41

[John Kozak 14]

File mappings do not consume virtual memory of the process, yes. But they do consume system's total memory (system commit). Hence, measuring system commit is senseless. The results Windies has provided confirm that - because by them it seems like ArmA process eats 4.2GB of virtual space, which is not possible.

Limits for 32-bit process VM are:

• Without LAA or on "default" 32-bit OS: 2GB
  
• 32-bit OS with 3GB flag and LAA: 3GB
  
• 64-bit OS with LAA: 4GB
  

Basically, that's what Dwarden said:

what you most likely see is the engine utilizing mapping memory w/o address via OS API

I'd replace "most likely" by "definitely" :) To see effect of LAA flag, VM and only VM matters.

Edited September 13, 2013 by DarkWanderer
Editing out some parts. Misunderstood Fred41, sorry

[fred41 42]

... continued later ...

Edited September 13, 2013 by Fred41

[John Kozak 14]

... ups ...

@DarkWanderer, it was never my intention to "insult" you in any way, or to be "sarcastic" (really, why should i?).

Maybe i misunderstood your last post (this is possible because i am not very good in english).

No, it's me who must apologize here. After reading some especially "adequate" stuff, I've misread your phrase in a bad way :) My bad, sorry.

I got the impression, that you mean the following:

The large values for commited system memory (observed by Windies) are only caused by " for e.g. drivers, especially Realtek ones, or some other applications (Steam UI, TrackIR, ...)"

Because of that i assumed, that you didnt know about the linked document.

Thats all. Sorry again.

If you follow this thread, you could notice, that i did some wrong assumtions, related to the reason for the observed problems.

At least for me, such misstakes are normality and not really a problem.

The title of this thread is a few days old and i can't change him now, but i think we should not disrupt this very interesting thread, just because the title is obsolate?

Greets,

Fred41

Well, actually I've had a mention of memory files in that post, but removed it because was not sure about whether it actually takes virtual memory or not. Sure the discussion should continue. It just looked to me as if you tried to support Windies's point of view by pointing to this link.

In any case, the misunderstanding was due to my fault. I apologize.

-----------------------------------------------------------------------------------

Back on the track of discussion, I think it'll be interesting to check to which extent ArmA engine can utilize this "extra" mapping memory. Here Windies's observations are actually coming in handy - an indirect way to check such usage is to subtract the arma3.exe VM size from the total change of system commit (provided only ArmA 3 maps big files).

I've seen mentions that in some cases memory mapped files work slower that the regular file reads. As far as I know, Lock On/DCS series utilise regular file reads, not memory mapping (but still have 4GB file size limit - because of internal 32-bit counters :P). May this be the reason of stutters in some cases?

[fred41 42]

... thats what i am asking me currently too.

How can we determine, what amount of memory is currently "pre-occupied" via CreateFileMapping?

(hooking kernel32.dll was my first idea, but there should be a more reputable way :)

Back on the track of discussion, I think it'll be interesting to check to which extent ArmA engine can utilize this "extra" mapping memory. Here Windies's observations are actually coming in handy - an indirect way to check such usage is to subtract the arma3.exe VM size from the total change of system commit (provided only ArmA 3 maps big files).

I've seen mentions that in some cases memory mapped files work slower that the regular file reads. As far as I know, Lock On/DCS series utilise regular file reads, not memory mapping (but still have 4GB file size limit - because of internal 32-bit counters :P). May this be the reason of stutters in some cases?

[windies 11]

It's a best guess scenario. I'm no programmer by any means, but I understand some things from taking programming courses in high school and college. The way they do their memory management is weird because they would run out of addressing space very quickly if they did it in a normal way, I.E. Pre-loading all assets and paging non essential data and streaming from the page file. I'm sure pre-loading all assets would run into the 6-8gb range, so most likely what they do is then use the file mapping API to keep data ready to stream into memory on demand. This way they can run normal addressing for the process, which is why you only see around 2gb usage for the process but you see a commit charge of 6gb or more.

Also you have to remember that was a short flight, if you look at the screenshots and look at my max commit, I did some earlier testing and the commit charge almost hit 7gb. This is all with an empty map, no assets, just me in an aircraft flying around. I will test it out with a battle going on and see what the commit charge is like.

If this was drivers, why would commit charge increase gradually as I go through different stages of the game, I.E. main menu, editor, then loading into the game? I'm sure some of it is for the drivers, but I highly doubt you can explain it all away with that.

[John Kozak 14]

Well, as we found out already, it was not drivers etc... It was files mapped to memory. I've included it in my post initially, but later deleted - because I was not sure about their actual behavior.

While this is certainly not a good way to measure the LAA effects, you've managed to find a good approximation for measuring the total amount of RAM which can be potentially used by the process. I guess, it's a good starting point to look for other optimizations. Maybe there's a way to tweak OS file mapping behavior. I'll see if I can find anything on the topic.

Edited September 13, 2013 by DarkWanderer

[windies 11]

Here's a screenshot of a multi-player server and the current/max commit and process virtual usage for comparison to an empty map in the editor.

http://s10.postimg.org/8u2j84wex/Commit_Arm_A3_Multiplayer.png (161 kB)

It peaked around 7.4gb commit and the process virtual size was almost 2.7gb. That's around 5.6gb commit just for ArmA 3 when subtracting the base 1.8gb commit charge.

[fred41 42]

... thats interesting.

Just a question, you wrote "screenshot of a multi-player server" and a client is showed of the desktop and in the process explorer window.

Are there both server and client on the same system, or is it only client connected to a remote server?

Here's a screenshot of a multi-player server and the current/max commit and process virtual usage for comparison to an empty map in the editor.

http://s10.postimg.org/8u2j84wex/Commit_Arm_A3_Multiplayer.png (161 kB)

It peaked around 7.4gb commit and the process virtual size was almost 2.7gb. That's around 5.6gb commit just for ArmA 3 when subtracting the base 1.8gb commit charge.

Edited September 13, 2013 by Fred41

[windies 11]

It's me in a windowed client measuring my memory usage while I'm connected to a server. I don't have a way to measure server performance right now, though I would be very interested in seeing the memory usage of the dedicated server.

[fred41 42]

... you could start a dedicated arma server on the same system as your arma client, thats my current testing setup ...

If you do so, an entry in server.cfg, like this:

localClient[] = {127.0.0.1};

helps maxing out the network bandwidth.

But server performance is impacted a bit, because this two processes have to share the available CPU cores and physical RAM.

It's me in a windowed client measuring my memory usage while I'm connected to a server. I don't have a way to measure server performance right now, though I would be very interested in seeing the memory usage of the dedicated server.

Edited September 13, 2013 by Fred41

[windies 11]

If I ran them both on the same machine though I would have no conclusive way of measuring the amount of memory each program uses.

BTW DarkWanderer, DCS went 64 bit A LONG time ago, like back when DCS A-10C came out. In fact I don't even think they maintain a 32 bit binary anymore and they're system reqs list a 64 bit OS as a minimum requirement.

Edited September 14, 2013 by Windies

[fred41 42]

... yes, thats true too.

I checked the microsoft api's, but didn't found a reasonable way to determine the ammount of system memory, occupied indirectly via CreateFileMapping by a specific process.

But i think, it could make sense to monitor the (hard) pagefaults counter for the arma (server) process, to verify if a performance problem (lag) is related to memory allocation/paging.

Maybe i will soon add a counter PF/sec to the monitored processvalues in ArmaServerMonitor.

Greets,

Fred41

If I ran them both on the same machine though I would have no conclusive way of measuring the amount of memory each program uses.

Edited September 14, 2013 by Fred41