xtbbmalloc - A custom memory allocator for A3

[DieselJC 196]

Question.. I get this an error..I run the PC as Admin but don't start the game as Admin..I start it from the launcher.

According to this the large pages isn't working..so how do I set that correctly? Do I need to start the launcher as Admin?

 

12:51:05: Init
12:51:05: Config:
User is admin: No
UseLargePages: Yes
SeLockMemoryPrivilege: Yes
SeIncreaseWorkingSetPrivilege: Yes
ForceMaxWorkingSet: Yes
HoldMemory: No
PreAllocBytes: 0.000
LockPages: No
12:51:05: Ext0SetPrivileges: 41, Error: 1300
12:51:05: MinimumWorkingSetSize: 4096.000 mb
12:51:05: Ext0SetWorkingSet: 0, Error: 0
12:51:05: Using normal pages
12:51:05: Init done

13:55:46: Exit...
13:55:46: ...ok.

[alinz 1]

To enable large pages; in your start menu search box, type "secpol.msc" and navigate to Local Polices > User Rights Assignment.

 

Double click "Lock pages in memory", Add user or group > Advanced > Find now. Select your windows account name from this list and click OK > OK > OK.

 

That should resolve your error.

[DieselJC 196]

To enable large pages; in your start menu search box, type "secpol.msc" and navigate to Local Polices > User Rights Assignment.

 

Double click "Lock pages in memory", Add user or group > Advanced > Find now. Select your windows account name from this list and click OK > OK > OK.

 

That should resolve your error.

Thanks..I have all that set from the first time using the allocator. Like I said I don't run the actual game or lau8ncher as admin..I run it normally then I canuse task manager to set A3 to high priority and steam to low for when I am gaming.

 

Just to clarify I have the .dll in the DLL folder and am using the Large Pages config settings file..what do I need to do with the xbbtmalloc.pdb file?

 

 

Thanks

 

Diesel

[goko- 14]

_blub thanks for the effort, this malloc saved me and my friends. Also I created aguide at steam here:  https://steamcommunity.com/sharedfiles/filedetails/?id=791321243so many people can enjoy better fps. thank you!

[larisagnia 0]

Hey guys, I've been trying to min max my FPS and performance on arma 3 for a VERY long time. I love the game, and I am very good at it but I still wish I had better frames. Sometimes Ill have 28 or so, and other times 55. But I have never done anything with this "memory alloctator" or large pages, etc. Can anyone, please tell me exactly what I need to do to get that fps boost I need in regards to the memory allocator and the large pages, and anything else that could help?

[ArtyCooL 0]

Hey guys, I've been trying to min max my FPS and performance on arma 3 for a VERY long time. I love the game, and I am very good at it but I still wish I had better frames. Sometimes Ill have 28 or so, and other times 55. But I have never done anything with this "memory alloctator" or large pages, etc. Can anyone, please tell me exactly what I need to do to get that fps boost I need in regards to the memory allocator and the large pages, and anything else that could help?

There is a good guide on Steam for how to tweak and setup everything including malloc and large pages. 

http://steamcommunity.com/sharedfiles/filedetails/?id=791321243

[_blub 28]

Hi guys, sorry for my long absence :)

I still have not much time for modding but today I had some minutes for this project.

 

New release:

https://dl.dropboxusercontent.com/u/103425066/CMA.zip

Nothing really new.

- Changed the placeholder name to CMA (CommunityMemoryAllocator), I know best name ever so feel free to rename ;)

- Remove the experimental stuff to simplify usage. There is cma.dll which goes into the Arma 3\Dll directory and cma.ini which goes into the Arma 3 root directory. Same procedure as before, only the names changed.

- Config file "cma.ini" has only one option left: UseLargePages. Set it to "1" to use large pages or to "0" if you dont want to.

 

I will send this to battleye now so you should be able to use this online in a few days.

Thanks and have a nice day!

 

Btw: The pdb file is only for debugging. So you probably wont need it but I included it anyway.

Oh and good work, nice guide :)

 

Little update:

The battleye guys are quite busy at the moment so the whitelist entry could take a while longer...

[_blub 28]

Hi,

 

Because A3 made the step to 64-bit I just added a compatible version of the allocator:

https://dl.dropboxusercontent.com/u/103425066/CMA_2016_12_14.zip

 

Use the cma_x64.dll when you run the A3 64-bit executable.

Again not fully tested so let me know if there are any errors.

 

Btw: Nothing new with battleye... no idea whats wrong this time because last time they were much faster... whatever lets wait a bit longer :)

[goko- 14]

Thanks _blub, testing right away.

update: there seems to be minor differences, again CMA (default x64build) superior against tbb4x64 at 64bit client. I've played gunship showcase with both mallocs, at certain moments (eg flying over town with lots of AI) CMA had better framerate. One thing though: with cma, I've had stutter once which caused fps to drop low 10's. I don't think this is related to malloc but, still worth reporting i guess. stability wise, it is stable as it gets. no issues. I'd be able to test it better if mp server had some players but, server is empty.

 

I have a question for you blub, I remember finding a memory allocator at arma2 days called "hoard". I am not a programmer, I'd build and test it myself If I can. Do you know about hoard, can it benefit us on arma3? http://www.hoard.org/download-hoard/

 

but, honestly..I don't think anything will make much difference until bi change something else fundamental in the engine. What makes me think like this is lack of difference between AVX and TBB4 builds. Theoratically avx should perform better, but I dont see difference (at least at my cpu - 2500k)

overall, 64bit makes the game smoother, especially having huge the draw distance without hiccup is fantastic. things can only get better after this step.

 

[Horus 83]

@_blub Can you pls make 64bit version of xtbbmalloc because that is best allocator for my system.

 

Intel Pentium G3258@4.4GHz

4x4GB DDR3 RAM@ 1886MHz

ATI R9 270@1050/1500 (Crimson ReLive Edition 16.12.1)

Win7 64bit

w/o page file

[STONEY-DSP 25]

i was wondering why the XTBBMalloc v2 SSE2 works better than the v3 and CMA with my CPU  AMD FX-4100 and also wondered if its easy for a nub like me to make 64-bit version of it )( trust me im a nub )

or would you take the effort and make  it a 64 bit of it?

[_blub 28]

On 15.12.2016 at 6:08 PM, archibald tutter said:

there seems to be minor differences, again CMA (default x64build) superior against tbb4x64 at 64bit client. I've played gunship showcase with both mallocs, at certain moments (eg flying over town with lots of AI) CMA had better framerate. One thing though: with cma, I've had stutter once which caused fps to drop low 10's. I don't think this is related to malloc but, still worth reporting i guess. stability wise, it is stable as it gets. no issues. I'd be able to test it better if mp server had some players but, server is empty.

 

Little differences are "normal". If you use CMA without large pages there should be no difference in performance if compared with stock tbb from BI. Keep in mind that Arma AI is calculated dynamically so it behaves every time a bit different which causes inconsistent benchmarks.

The hard fps drop should not be related to my modifications. There are two possibilities. A) You are using large pages which can cause delays especially when your PC is running for some time. B) Arma3 x64 bit is still WIP.

 

 

On 15.12.2016 at 6:08 PM, archibald tutter said:

Do you know about hoard, can it benefit us on arma3?

 

Never heared about it, sorry ;)

 

On 15.12.2016 at 6:08 PM, archibald tutter said:

Theoratically avx should perform better, but I dont see difference

I think you got something wrong :)

AVX is just an extension on your CPU which adds some instructions. It does not modify/replace any "old" instructions. Most AVX instructions are for 3D vectors and such stuff but because CMA does no such things the chances are high that there are no AVX instructions inside CMA. Thats the reason why I set the SSE2 build as default. I think there are no benefits in the AVX2 build but I included it just in case.

 

3 hours ago, Horus said:

@_blub Can you pls make 64bit version of xtbbmalloc because that is best allocator for my system.

I dont plan to modify old builds in any way, sorry.

xtbbmalloc is build on an older version of intels tbb. So there might be minor bugs inside. I will only update the newest version. Just imagine that xtbbmalloc is faulty and you dont want to find the little bug when you are near the end of an epic mission ;)

Btw: Any reason for the w/o pagefile? Pagefile is quite important and could increase performance (available memory) in some cases.

 

1 hour ago, STONEY-DSP said:

i was wondering why the XTBBMalloc v2 SSE2 works better than the v3

I implement only some modifications.

The original tbb comes from Intel. I guess they did some minor changes which degrades performance a little bit on the build I used for v3... but as I said we have to live with it because I am sure they are aware of what they are doing and did not ****** performance without a good reason ;)

 

 

By the way... did somebody test jemalloc?

If so, how is the performance compared to stock tbbmalloc from BI?

 

Have a nice day :)

[Horus 83]

I have better FPS with xtbbmalloc than with CMA on my system. I don't see any reason to use pagefile when I have 16GB of RAM.

[goko- 14]

Thanks for detailed answers, blub! I always thought SSE3 was better than SSE2 or in this case, AVX was superior to older instruction, more efficient maybe, or I imagined new to be somewhat better than old one hehe.

 

It is too bad there was a koth server with 80 slots, I was hoping it to get more populated towards weekend, nothing like multiplayer benchmark...but server is gone now, they probably took it down after 2 days of emptiness.. Only tests I could run was game itself showcases. my findings are; JEmalloc definately better @64bit. definately. cpu is sb 2500k. And yes JE malloc performed better against LargePages configuration of your latest build also (like I said , strange stutters keep occuring with LP enabled CMA build).

thanks and you too have a good one!

[St. Jimmy 272]

11 hours ago, Horus said:

I have better FPS with xtbbmalloc than with CMA on my system. I don't see any reason to use pagefile when I have 16GB of RAM.

You run out of memory very quickly with 64-bit if you don't have page file. 16GB of RAM couldn't handle the game much more than a minute when I tested. The game either requires at least very small page file no matter how much memory you've or 32GB of RAM is a minimum for not needing page file anymore.

[fred41 42]

14 hours ago, Horus said:

I have better FPS with xtbbmalloc than with CMA on my system. I don't see any reason to use pagefile when I have 16GB of RAM.

 

... from my (earlier) experience and in respect to large page usage, backing your physical memory pages by a pagefile (preferable OS recommended size, fix) is helpful/necessary for windows OS,

especially if you have to defragment your physical memory later to improve availability of large pages.

However, a very clean windows system (less background memory activity, no memory leaks) can survive without any pagefile and still have some large pages available, though.

 

BTW on linux huge page support is significantly better (transparent huge pages, huge page daemon, huge page file system ...)

[Horus 83]

@St. Jimmy @fred41  So far I haven't any problem but we'll see when 1.68 patch come out.

[goko- 14]

Hello @fred41

Do you think reason that arma performance is stuck because of CPU L3 cache policies since 2009's? cartel, Intel milking us since c2duos and only xeon or i7's have 15-20mb L3 cache, they keep releasing same cpus with different names and maybe some changes around memory configurations. btw AMD looks like back into the game with ryzen, maybe finally we see some actual progress.

[fred41 42]

10 hours ago, archibald tutter said:

Hello @fred41

Do you think reason that arma performance is stuck because of CPU L3 cache policies since 2009's? cartel, Intel milking us since c2duos and only xeon or i7's have 15-20mb L3 cache, they keep releasing same cpus with different names and maybe some changes around memory configurations. btw AMD looks like back into the game with ryzen, maybe finally we see some actual progress.

 

Hello @archibald tutter,

i am still on my good old 2500k@4GHz (bought used, btw), so i am not a victim of any cartel's :)

But yes, loocking forward to AMD's ZEN incarnation too.

 

As for arma's performance problems, i think it's memory bandwidth is still way to hi.

Cache locality is the first thing you have to care about on today's cpu's, for optimal performance, even since DDR4.

There is still a lot of potential lurking around, so buying more and more powerful hardware,

just to get 33 instead of 30FPS, is not the way, imho.

[goko- 14]

right on @fred41

I'm still rocking my 2500k @ 4.6ghz too!! kinda bad power consumption, exceeds 130w at load but 'tis good. still rocking radeon 5850 as well, undervolted. I feel like they really milked people all this time about GPU's too. We used to get top performance and only performance, suddenly for few years now, they want people to pay top dollar for same card/performance, only with few watts less consumption. And sad thing is, people accepted this policies without making any criticism. PC supposded to be about performance. Now everything is really overpriced and nothing to get excited about.

[o5_ 7]

Thanks for your contributions _blub! I just started using large pages the the gains are real! lol. When using XTBB and CMA (though the CMA seemed to hold out longer) I hit a point of no return that gets super slow twichy/freezing behaviour that if I don't immidiatly kill the process freezes my system . Arma takes ~3.5GB and with the additional load of the OS, puts me at about 6-6.5GB(ram) and ~12GB(pagefile) when this behaviour starts. I have 16GB of ram and 20+GB for pagefile so I'm not running out of memory. Any idea what's going on? I read they increase the mem usage to around 3.5 for the process right? is the process just capping out on mem? and if so why is it not dumping old memory? To reproduce I zoom around altis with the camera until all objects are loaded then go fight some AI with smokes and explosions and such. I'm on windows 10(64) and my system specs are as follows...

i7 4790, GTX 970

16GB DDR3 @ 2400

OS and Arma on their own SSDs 

 

cma.txt output...

20:10:06: Init
Config:
User is admin: yes
tSeLockMemoryPrivilege: ok
Using large pages
Init done

20:17:29: 
Dumping stats (Exit):
    PageFaultCount: 4134152
    PeakWorkingSetSize: 661.352 mb
    WorkingSetSize: 143.207 mb
    QuotaPeakPagedPoolUsage: 1.387 mb
    QuotaPagedPoolUsage: 498.672 kb
    QuotaPeakNonPagedPoolUsage: 113.422 kb
    QuotaNonPagedPoolUsage: 54.766 kb
    PagefileUsage: 1.869 gb
    PeakPagefileUsage: 3.041 gb
CmaPerfMappedMemory: 1793400832
CmaPerfCallVirtualAlloc: 5
CmaPerfCallVirtualAllocLP: 194
CmaPerfCallVirtualFree: 65
CmaPerfCallMemTotalComitted: 20514
CmaPerfCallMemTotalReserved: 0
CmaPerfCallMemFlushCache: 12400
CmaPerfCallMemFlushCacheAll: 5
CmaPerfCallMemSize: 385491283
CmaPerfCallMemSizeA: 948770
CmaPerfCallMemAlloc: 188372778
CmaPerfCallMemAllocA: 474384
CmaPerfCallMemFree: 188368469
CmaPerfCallMemFreeA: 474384

Exit

[_blub 28]

@o5_

I assume you are using the stable branch right?

Max memory has been increased to ~3GiB on x64 Windows.

Do you get those freezes when using normal pages too?

[o5_ 7]

Disregard... I have this issue using all allocators but, yeah stable branch. Strange I have never noticed. Again, Thanks.

Edited December 20, 2016 by o5_

[goko- 14]

Hello @_blub !

did you secretly employed at bohemia or what, man ? :] this latest tricks from performance build with tbb4 is impressive to say the least, still falling short compared to xtbb with largepages but, phenomenal steps for arma performance IMO.

check this post from dwarden: 

I am getting greater higher FPS with these latest TBB4, still it falls short compared to XTBB lowest minimum but, what they do here is phenomenal. when we think about L3 cache cartel of intel and all that in last 6-7 years, looks like tables are turning. And arma becoming a game what we imagined for so long.

exciting times.

[_blub 28]

Hi, sorry for my long absence :)

 

No I did not but it seems those devs around dwarden are quite capable :P

Give them some time to fine tune their new tbb, I am sure it will become awesome.

I will set this project on hold for now because there is no need for this when we get official support. Also this way its possible to play online because the guys at battleye are thinking this project and the community around it are too small to get a whitelist entry... actually they whitelisted my first version but... well...

 

But when there is need for an update (e.g. because of a new version of intels TBB or a feature request) I will do it.