fred41

... yes, basically what Dwarden wrote ... @frag85, even if this is not the windows inside forum here, i will try to answer some of your questions. First, your 3 last logs are recorded with swapfile on, right (you forgot to tell us how many GB)? This 3 logs are looking very good and one should assume, that this pagefile should solve your problem? Ok, a bit more detailed now. You probably know, that using large pages results in higher memory access speed (higher TLB efficiency). To use this advantage, 'tbbmalloc for arma' at first tries to allocate memory using large pages (VirtualAlloc with MEM_LARGE_PAGES flag). Only if this allocation attempt fails, allocator is forced to fallback to small pages (second allocation attempt). So if you see a fallback in your log, this just means, there are not enough CONTINOUS 2MB blocks in PHYSICAL memory available. Try to realize, that one single small page (4kB), inside a memory region of 2MB (512x4kB) can make the whole block unusable as large page!!!! It is important to understand this and to keep this in mind. Even if you have 512GB of physical RAM, it is very easy to make the whole memory unusable as large pages, by allocating each 512th small page (worst case of fragmentation). Do you still follow? Ok :) So it only depends of your OS memory manager, how clever it handles memory allocation (mapping) in phys. RAM, to prevent fragmentation and provide a maximum large pages. It seems, that the existence of a disk based paging file, makes memory managers life easier (at least yours and Dwardens observations seems to confirm this). Your interpretation of the log file colums are fully correct. Please realize, that each line in this logs, contains a memory region allocation (custom allocator requests RAM from OS). What you not see in the log, are upto MILLIONS of very small allocations PER FRAME, from arma to custom memory allocator. So, you can imagine custom memory allocator as an 'memory distributor'. It allocates (large) regions from OS memory pool and provides it as small pieces to arma. (such a memory distributor is called 'heap') Enough now :) If there are still questions left or new ones occure, please feel free to ask. Greets, Fred41

@Hurtz72, if i understand you correctly, you observed a FPS loss of 5.0FPS, without any real load? There is something wrong, of course. A first idea and question: Have you tried to change your power plan settings to 'high performance'. If not, try it out and let me know please. @Tankbuster, yes this custom command is executed local on server and should have access to all global variables in mission namespace. Just ensure, that the value returned from the full command line is a positive, discrete number (like 1, 2 ... 1000). If you are unsure, you can look in fn_ASM.fsm (and/or the source code of ASMdll.dll if you are talking 'C' too). Greets, Fred41

@frag85, many thanks for your effort to make that systematical and detailed tests. I will try to interpreting your logs: 1. enabling the paging file on your system, significantly improves the availability of 'large pages' and avoids/delays fragmentation of physical RAM 2. we can now exclude arma (even with higher settings) as source of phys. memory fragmentation 3. even if we didn't found the source of fragmentation on your system, we now have a solution to prevent such fragmentations Conclusion: Use a pagefile (and/or adapt the size) if you have fragmentation symptoms, to increase the availability of large pages (at least on windows 7) Thank you and thanks to dwarden again, you helped yourself out and other peoples with similar fragmentation symptoms too :)

@frag85, with 'pagefile' size, i meant the swapfile (ControlPanel->System->AdvancedSystemSettings->Advanced->PerformanceSettings->Advanced->VirtualMemory ...). I think it is worth a look/try, even if you have 32GB RAM. Related to the VRAM flush symptom, there was a short discussion about, a few weeks ago in this thread (started with post #58). This issue is still existing, but this is a problem outside of custom memory allocators scope/domain. Apart from this, there must be a source/reason for this heavy memory fragmentation on your system. Please try the following settings (just to exclude arma itself as a possible source): ViewConditions: 1600/1105/100 Textures: high Object: standard Terrain: standard Shadow: high Clouds: low PIP: deactivated I know this settings will not look very nice, but it would be interesting to see, if you run with this settings on a fresh started system, if there are still fallbacks to 'SP' in your log? Greets, Fred41

@Dwarden, this is interesting and i think a to small or disabled swapfile, could be the reason for some strange fallbacks on other systems too. @frag85, your observations and conclusions makes sense for me. Arma is using memory in many different ways, custom memory allocator is only one off them. I would still recommend to lower some setting, like view distance, texture resolution and object/terrain details, to less stress armas section object based cache (probably your current bottleneck). I am tuning my system always, by adapting my settings, so that i can get an average of ~70FPS in helos A3 Altis benchmark. This way i have mostly stable 50FPS in multiplayer sessions (vsync on). Dwardens success with a larger pagefile, lead me to the question, did you played with different pagefile sizes too? @Nuno Basto, thanks for feedback, please let me know.

... hmmm ... Dwarden, what is your experience: Does flushing the system file cache and clearing the standby list (before arma start) improve the availability of large pages significantly?

... that's where i have the idea taken from ... ;)

... thanks for your feedback and logs ... @NoPOW, your logs are still a good example for full large page availability and best allocation timings. @NumRollen, there is a new version available (with some improvements). @Kremator, i think you don't really need to run this tool for your system. It is mainly for this systems, where fallbacks to 'single pages' occures early, even if more than enough phys. RAM is installed (fragmentation). But it does'nt hurt to use it, because it could help to delay normal fragmentation a bit too. @ Bono_LV, your log file looks perfect. Improvement of 9% for a flyover-benchmark is a very good result.

@Kremator, yes. It helps to understand armas allocation pattern, in different scenarios and to adapt/implement some algorithms for better performance/usability. For everybody with early 'Alloc LP failed -> fallback' messages, i added a little experimental tool, for freeing some phys. RAM before arma start (flushing the system cache and empty the standby list). It may or may not solve this issue on your system, so just try it out. Please read the readme.md for usage: https://github.com/fred41/tbbmalloc_arma/tree/master/ClearMem If this tool should help, i could implement this directly in tbbmalloc for automatic memory clearing at arma start. Greets, Fred41

@Kremator, thanks for helping out with usefull hints. The 'MemFlushCacheAll request' ('MFCA' in newer builds) log message, just indicates that an full large object cache clear command is called from arma to the allocator (nothing wrong here). @Nuno Basto, your timings are suggesting that you use win 8.1? There is a newer build out now, which shows some info about your system and memory availability, at arma start.

... thanks, that are enough as a start :) Three related questions: Are all this logs from the same system? How much phys. RAM is present on your win 8.1 system? What windows 8.1 edition is it (home premium or starter)?

@kremator, i am glad to read this :) Can you please post some logs from your win 8.1 system, to compare the timings with logs from NoPOW? (Just want verify, that win 8.1 is the only reason for that impressiv LP allocation performance)

... just found this: http://www.askvg.com/how-to-enable-group-policy-editor-gpedit-msc-in-windows-7-home-premium-home-basic-and-starter-editions/ I did'nt tested it and can't support it here. So please use it only if you know what you do and at your own risk.

... a peacefully and happy new year everybody ... :o @frag85, thanks for your feedback. What you found about fragmentation is mainly related to heap and virtual address space fragmentation. But the availability of large pages is related to fragmentation in physical memory address space (unmapped, native addresses), because the memory blocks have to be continous and aligned to 2MB boundarys here, to be available as large page. This kind of fragmentation is especially caused by applications/drivers which frequently allocates/frees a lot of small pages. To locate the source of fragmentation on your system you could use a tool like RAMmap from sysinternals. This nice tool allow you to do a "Empty standby list" too. This discussion could be related to your observations too: http://forum.sysinternals.com/rammap-empty-standby-list_topic27297_page1.html Try this and let me know if it helps. Yes, reserving (preallocation) of large pages, is what in older microsoft documentation is recommended for large page allocations. This is a good way to avoid fragmentation, as long as you know how much memory your application will need and if your system have enough phys. RAM left. But for arma systems, it is difficult to predict the optimal ammount of memory to preallocate and keep in mind, that memory allocted as large pages, is locked to phys. RAM (not pageable). BTW: Windows 8.1 seems to be much more aware of the higher demand of large pages for todays systems/applications and protects the phys. RAM much better against fragmentation. See the allocation timings in NoPOW logs. This timings are ~20 x times better than in windows 7. This is of course a big advantage and an important step in the right direction.

... thanks for your feedback and log ... The usage of large pages is an optional feature of this experimental custom memory allocator, implemented to get the highest possible memory access speed. Two conditions have to meet to make this feature fully available: - the 'lock pages in memory' privileg have to be set for arma user (is already set correctly on your system) - enough large pages (continous 2M blocks in phys. RAM) have to be available This line in your log is not an error: 4.306s: 4.034ms 4096k at:0x00000000 Alloc LP failed -> fallback It just means, that your OS memory manager can't find enough large pages (2 in this case) to satisfy this current VirtualAlloc (large page) allocation attempt. In such a case, tbbmalloc is making a second allocation attempt (now with small pages, fallback) and disables the usage of large pages, for any further region allocation. It is very likely that your system memory is highly fragmented in such a scenario. On a fresh restarted system you have less fragmentation and most large pages available. Fragmentation proceeding depends on amount of available phys. RAM and system load. The usage of large pages is not the only advantage of tbbmalloc compared with the default one. So don't be worried, if your system can't satisfy all allocations as large pages. ---------- Post added at 15:24 ---------- Previous post was at 15:16 ---------- ... AFAIK, this Secpol.msc (or gpedit.msc) is only directly available for the prof. and up editions of windows, but there are several ways to make it available on home and standard editions too (i think i remember via registry tweaks) ...

@NoPOW, thanks for your logs. Still perfekt. @Meatball, your .dll installation is probably corrupted, please redownload from here: https://github.com/fred41/tbbmalloc_arma/blob/master/binary/tbbmalloc.dll (Press the 'RAW' button to get the file) UPDATE: - handled access violation counter added to log statistic (should be always zero, but just in case) - nolog version of binary added to github repository (the performance difference is very small) I am still interested in logfiles from different systems/scenarios. Especially logs from long running servers would be helpfull. The current statistic line, at the end of the logfile, looks like this: 5161.196s: regAllocTotalDur: 260.948ms LOCstat: 7707/10186:75.7% TLSLeft: 2 handledAccessViolations: 0 Especially if you found a (handledAccessViolations > 0) or a high TLSLeft value, please report it to me. Thanks, Fred41

... simple and solid, will add this hint to readme.md ... Thanks .kju [PvPscene] :)

@NoPOW, thanks for your logs (full logs upto process shutdown, would be better). Allocation timings listed in your logs are still impressing (win 8.1). Slow map loading is not related to custom allocator (just switch back to default allocator to verify this). My experience is that the first load is always very slow and the second a lot faster (all data loaded in section object based cache).

... hmmm, honestly i don't know how to hide addons in server browser :/ I decided to made it available as addon, just to save everybody the effort including it in each missions init.sqf manually. If you can find some detailed documentation about addon configuration, please let me know. Greets, Fred41

@Ezcoo, yes this good old one :) @everybody, thanks for feedback and log files UPDATE new tbbmalloc for arma build is out: - large page region allocation granularity changed to 4MB - arma thread creation/termination logged, to locate possible memory leaks caused by "ghost threads"

@Numrollen, please make a full ASM reinstall (see readme.md for instructions). All arma instances have to be down for .dll replacement. Details about ASM are here: https://github.com/fred41/ASM/ Very low CPS is mostly caused by overloaded VM (mission: too much/frequently spawns, execVM or endless loops in VM). Greets, Fred41

@Numrollen, thanks for your feedback and log file. The first line in your log, tells us, that the "lock pages in memory" privileg is NOT set for the user account you use to start arma. SeLockMemoryPrivilege: denied, huge pages disabled To get the whole advantage of tbbmalloc, please read the readme.md for info how to grant this privileg correctly. Only if your system has no fast SSD, a RAM disk for the whole arma addon folder probably can help a bit. Greets, Fred41

... thats exactly what i meant. Just asked, because this three object counting commands are customizable now and i was not sure, if you use defaults or your own .sqf commands here :)

... i am really glad i read that, because i am trying to improve my english, while posting and reading in this forum ... :)

Hi [HUD]Dorph, your screenshots are showing an interesting and important issue, it seems. Just two ASM related hints: - since objectcounting is customizable, it would be helpfull to know, how OC0 and OC1 are defined in asm.ini for this session - you could use the "timedivisor" feature to "compress" the whole scenario in one screenshot, for a better overview (popupmenu in value area, select: 2sec, 5sec or 10sec) Thanks for sharing your screenshots :) Greets, Fred41