_blub

Hey guys, I made a custom memory allocator for Arma3 and I want to share it with you. Originally I made this for testing purpose only but maybe it can improve performance in some cases. Features Based on Intels tbbmalloc 4.4 Update 5 (https://www.threadingbuildingblocks.org/) Support for large pages Includes some special tweaks which can improve performance in some cases (opt in) (experimental stuff is not included anymore). Interface for reading memory statistics and modifying parameters on the fly Customizable via settings file Source code included. Feel free to modify :) Readme.txt for usage is included but there are also many answers in this thread. Download (based on "tbb44_20160526oss", battleye compatible) https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc.zip Download (based on "tbb2017_20161128oss", NOT battleye compatible yet...) https://dl.dropboxusercontent.com/u/103425066/CMA.zip Download 64-bit Version (based on "tbb2017_20161128oss", NOT battleye compatible yet...) https://dl.dropboxusercontent.com/u/103425066/CMA_2016_12_14.zip

There is a readme included :) Let me know if there is still something unclear

Hi dwarden, I dont know where to give you feedback about tbb4malloc_bi(_x64).dll (is there an official that for that?) but because its performance related I will try it here :) I just noticed that the binary is exporting multiple functions beside the needed interface for Arma3. Maybe its a good idea to remove those unnecessary exports to allow the compiler to do some more optimizations (e.g. inline some functions). Best regards.

Hi, Its documented here: https://community.bistudio.com/wiki/Arma_3_Startup_Parameters Regarding default allocation you can find this: So yes it should allocate more than 4GB by default if this is possible. I cant answer you the other questions and I am pretty sure only the developers can but maybe there is no fixed date for this :) Also there are other threads where your questions fit better and its more likely to get an answer.

Hi, sorry for my long absence :) No I did not but it seems those devs around dwarden are quite capable :P Give them some time to fine tune their new tbb, I am sure it will become awesome. I will set this project on hold for now because there is no need for this when we get official support. Also this way its possible to play online because the guys at battleye are thinking this project and the community around it are too small to get a whitelist entry... actually they whitelisted my first version but... well... But when there is need for an update (e.g. because of a new version of intels TBB or a feature request) I will do it.

@o5_ I assume you are using the stable branch right? Max memory has been increased to ~3GiB on x64 Windows. Do you get those freezes when using normal pages too?

Little differences are "normal". If you use CMA without large pages there should be no difference in performance if compared with stock tbb from BI. Keep in mind that Arma AI is calculated dynamically so it behaves every time a bit different which causes inconsistent benchmarks. The hard fps drop should not be related to my modifications. There are two possibilities. A) You are using large pages which can cause delays especially when your PC is running for some time. B) Arma3 x64 bit is still WIP. Never heared about it, sorry ;) I think you got something wrong :) AVX is just an extension on your CPU which adds some instructions. It does not modify/replace any "old" instructions. Most AVX instructions are for 3D vectors and such stuff but because CMA does no such things the chances are high that there are no AVX instructions inside CMA. Thats the reason why I set the SSE2 build as default. I think there are no benefits in the AVX2 build but I included it just in case. I dont plan to modify old builds in any way, sorry. xtbbmalloc is build on an older version of intels tbb. So there might be minor bugs inside. I will only update the newest version. Just imagine that xtbbmalloc is faulty and you dont want to find the little bug when you are near the end of an epic mission ;) Btw: Any reason for the w/o pagefile? Pagefile is quite important and could increase performance (available memory) in some cases. I implement only some modifications. The original tbb comes from Intel. I guess they did some minor changes which degrades performance a little bit on the build I used for v3... but as I said we have to live with it because I am sure they are aware of what they are doing and did not ****** performance without a good reason ;) By the way... did somebody test jemalloc? If so, how is the performance compared to stock tbbmalloc from BI? Have a nice day :)

Hi, Because A3 made the step to 64-bit I just added a compatible version of the allocator: https://dl.dropboxusercontent.com/u/103425066/CMA_2016_12_14.zip Use the cma_x64.dll when you run the A3 64-bit executable. Again not fully tested so let me know if there are any errors. Btw: Nothing new with battleye... no idea whats wrong this time because last time they were much faster... whatever lets wait a bit longer :)

Hi guys, sorry for my long absence :) I still have not much time for modding but today I had some minutes for this project. New release: https://dl.dropboxusercontent.com/u/103425066/CMA.zip Nothing really new. - Changed the placeholder name to CMA (CommunityMemoryAllocator), I know best name ever so feel free to rename ;) - Remove the experimental stuff to simplify usage. There is cma.dll which goes into the Arma 3\Dll directory and cma.ini which goes into the Arma 3 root directory. Same procedure as before, only the names changed. - Config file "cma.ini" has only one option left: UseLargePages. Set it to "1" to use large pages or to "0" if you dont want to. I will send this to battleye now so you should be able to use this online in a few days. Thanks and have a nice day! Btw: The pdb file is only for debugging. So you probably wont need it but I included it anyway. Oh and good work, nice guide :) Little update: The battleye guys are quite busy at the moment so the whitelist entry could take a while longer...

Hi, This is my config, I will go only over the performance related stuff: - CPU Count: unchecked - Extra threads: unchecked (there is no need to force anything I guess) - Memory allocator: as desired - Enable Hyperthreading: set (only set if your CPU supports hyperthreading) - System memory limit: This one can be interesting. If you believe https://community.bistudio.com/wiki/Arma_3_Startup_Parameters then the game should auto select someting between 512-1536MiB if you leave this box unchecked. But I could imagine that this page is not up to date any more... Launcher says valid input is from 256 to 2047MiB but Arma could use up to ~4GiB in theory. So I would force this setting to 4096. If it works then fine, if not... well... it should not matter. But maybe a dev (dwarden where are you :)) can tell you more. By the way... I am sure right after they released the new launcher the tool tip said someting about way more than 4GiB but that has been removed. Maybe the launcher devs dont know about the limitations of their game, no offense of course :D - Video memory limit: Launcher says anything over 2GiB may result in unforeseen consequences... I set this to 4096MiB... works. I think thats it... I am not sure if I understand correctly. But I think if you set a limit to 1024MiB than the engine would simply not request any more memory when it reaches 1024MiB. The engine would try to release "unused" (cached) memory first before it requests more. Hope this helps you :)

See initial post for update. New version: https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc_2016_10_09.7z I am short of time at the moment so I could not test this properly. If you encounter some bugs, tell me. Because of this I did not send this version to battleye yet. If you guys tell me there are no bugs or problems I will do that later.

Did you give your account the lock pages in memory privilege? If not, check this guide if you need help https://msdn.microsoft.com/en-us/library/ms190730.aspx I think this might be the problem :) Edit: A bit more time now... Give your local user account the "lock pages in memory" privilege. You can to this with secpol.msc oder gpedit.msc (check the link above) Then run Arma as admin.

This is not a dumb question :) Just choose the right (the one you want to use) configuration file (xtbbmalloc.ini) from the "Configuration" folder. Move only this file (xtbbmalloc.ini) into your Arma3 root directory.

You can use the old one (the one you find in the other download archives). But you are right, I will include it in the new archive too. ETA 5 mikes ;) Edit: Fixed :) Configuration file is now included.

I am not at home so I could not produce any logs :) Anyway, please download the following file (xtbbmalloc.dll). It is supported by battleye now. https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc.7z

A word? How about "pending"? :) I sent an email to BE with the file and the source. Lets see what they will do.

May I throw something in? https://forums.bistudio.com/topic/191816-xtbbmalloc-a-custom-memory-allocator-for-a3/ There are some guys which report performance improvement.

fred´s used intels tbb 4.2 Update 2. I am using intels tbb 4.4 Update 5. So those performance differences can come from changes between both versions. There is nothing I can do about that. I will update this allocator if there is a new release of tbb :)

Little update: v1.0.2: https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc/xtbbmalloc%201.0.2.7z Added some statistics and a new experimental configuration: "LockPages" (Check readme) Custom builds for SSE2, AVX, AVX2, with and without statistics enabled. Also added a demo mission which shows how to query and show xtbbmalloc´s statistics ingame. Did also some YAABs. I think I did not see the same battle twice :P So dont believe too much in those results... Used Dev build from 30.06.2016, no SDD 1) First run to allow Windows to cache some files... FPS: Did not check 2) tbbmalloc_bi (default allocator) @35.1 FPS 3) xtbb(+stats, LockPages) @36.5 4) xtbb(+stats, LargePages) @ 39.3 5) xtbb(+stats) @34.7 6) xtbb(no stats) @ 35.1 7) xtbb(no stats, LargePages) @37.0 8) xtbb(no stats, LockPages) @ 34.5 As you can see this benchmark is not really good for such tests :) Please let me know if you have ideas for some tweaks or if you want something to be included.

Thanks for testing pestbeule :) But because it is no big thing to change a little compiler flag... AVX build: https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc%20AVX.7z :) If you want or need some other features or statistics included please let me know.

xtbbmalloc compiled for AVX2: https://dl.dropboxusercontent.com/u/103425066/xtbbmalloc%20AVX2.7z No idea if AVX2 is even used but make sure your CPU can handle it (Haswell or newer for Intel, no idea about AMD). I dont think you will notice any difference but let the placebo effect do his job :)

@Bamse: Good to see its working :) @defunkt: Yeah saw that "feature" too. You are right, if you clear that flag, the image can not be relocated anymore. But there should not be a noticeable performance impact. So I am not sure what his idea was... @dwarden: Sorry I dont know fred41s non public release so I cant tell you all differences. I am not sure what you mean with "ability to set region sizes". Also I dont know what "MaxLargePagePrealloc" does. "preallocatedbytes" in xtbbmalloc is just allocating X bytes when loading. The idea is that you combine it with "HoldMemory". So you would allocate some memory at loading and hold it. When the game later needs some memory, it doesnt need to be allocated (which could take a little) because it is already there. As I said (hopefully) in the readme file. This is experimental. I wanted to see how it affects performance. But I didnt do any tests yet...mostly because I have no idea how to do a good benchmark. YAAB is not bad. But it is still dynamic and every time a little different :) The uploaded xtbbmalloc is compiled with SSE2 (which is default). I just forgot about that. I can compile it additionally for AVX2 but I dont think it will produce much different code... I know about AWE but I guess it needs to be supported by the game engine. Just another little hint which I should have included in the first post: When you use large pages make sure that you have lots of free memory and/or your PC is not running for long time. Otherwise it can take a long time for your OS to find memory regions which are suitable for large pages :)

Hey guys, first about fred41s tool (thanks derfunkt for the link). It sets the registry key HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\Arma3.exe\UseLargePages to 1 to ask the application loader to allocate large pages for the arma3.exe image file. There is not much documentation about this registry key. On msdn it says about UseLargePages: "Load image using large pages if possible". I guess this is just a "hint". Not a rule. Also I dont think this should provide much performance improvements (if it works at all). @Bamse: So xtbbmalloc.dll gets loaded but there is still no xtbbmalloc.txt in your Arma3 directory? How do you start the game and with which parameters?

@ineptaphid: One of the last updates changed the interface for custom allocators. There are now 3 additional functions required for aligned allocations. Those are not included in fred41s allocator. @pestbeule: I have no idea what armaLP.exe is :). Is it part of the performance build? I have never seen it before.

Thanks torndeco. I will do that in the future if it turns out that this allocator can be useful :)