Possible fix for the "3 FPS Drop" Issue

[claws01 22]

we keep losing player everyday

[bis_iceman 7384]

Hello, 

I guess it's pretty clear even from the replies, but it indeed seems that the issue has not been fixed with the new memory allocator. We have been carefully monitoring your answers. Thanks to everyone for participating on the tests. Although the solution was not found yet, we are still investigating the issue and will of course ask you for assistance. Thanks again for the help!

[Plainfield 2]

@bis_iceman I am putting my logs here with how the even occurred. 

https://feedback.bistudio.com/T120745

[Woody05 11]

I don't know if this information helps or not. I've noticed this happening the most whenever new units spawn in. For example we called in a large amount of reinforcements and it happened. Then last night an armored vehicle came at us. We shot a rocket at it and missed. However when it stopped and 6 troops jumped out it happened.

It's a pretty rough bug. I had to restart my whole computer because it locked everything up. Then when I came back in I had a corrupted user file. Had to re-add all my local mods.

[SpyInDisguise 0]

I don't know if this information helps or not. I've noticed this happening the most whenever new units spawn in. For example we called in a large amount of reinforcements and it happened. Then last night an armored vehicle came at us. We shot a rocket at it and missed. However when it stopped and 6 troops jumped out it happened.

It's a pretty rough bug. I had to restart my whole computer because it locked everything up. Then when I came back in I had a corrupted user file. Had to re-add all my local mods.

 

I am experiencing what you said but without corrupted user file. I mostly play on an Invade and Annex mission server (2 different servers) and this happens during a defense mission when units are spawned in. My fps drops and stays around 5 fps with random low texture flickering. I tried playing without mods (I only have JSRS) and fps still drops. Whilst playing on a KoTH server, it rarely happens but when it does it usually when i'm overlooking at a heli with people getting out. 

[.kju 3244]

@ Iceman

 

Please repeat the test and make people use the profiling binaries to catch the memory leak.

 

Note: it's not in performance binaries! It needs the debug layer on DEVELOPMENT/PROFILING binaries!

(Copyright by Dwarden)

[UK_Apollo 476]

Any update on when this might get fixed to Main Branch.. ?

 

We had a cracking mission ruined tonight by 75% of players getting FPS reduced basically to 2-3FPS rendering the entire troop combat ineffective.... :-(

As a follow-up to this, we (3CB) subsequently removed several 3rd party mods from our (extensive) modset and over the last few weeks none of our large scale missions (2-3 hours, 30+ players, 3 times/week) have had any problem with fps or client crashes at all. The improvement in stability was dramatic and conclusive. In addition performance on our 24/7 public server has been fine too.

 

We haven't yet narrowed it down to which mod / combination was causing the problems, but it wasn't vanilla Arma / memory allocators as perhaps was thought the case from when Serjames posted.

[Walterros 2]

As a follow-up to this, we (3CB) subsequently removed several 3rd party mods from our (extensive) modset and over the last few weeks none of our large scale missions (2-3 hours, 30+ players, 3 times/week) have had any problem with fps or client crashes at all. The improvement in stability was dramatic and conclusive. In addition performance on our 24/7 public server has been fine too.

 

We haven't yet narrowed it down to which mod / combination was causing the problems, but it wasn't vanilla Arma / memory allocators as perhaps was thought the case from when Serjames posted.

 

Hey there, something of note that might help you in your search. We've (IEU) have been running a few missions the past few weeks and eliminated the usage of RHS .50 cals on vehicles and noticed a reduction in crashes. It wasn't until tonight during our ALIVE op where ALIVE spawned a number of technicals (5) with RHS .50cals mounted (from Leights OpFor / ProOpF). They opened fire on our convoy simultaneously and crashed close to the entire server (30+ people). Hopefully that helps in eliminating which mods, if I have any further information I'll be sure to either post here or PM you directly.

[bis_iceman 7384]

Knowing a way how to reproduce the issue even with community made mods would be a bliss, be sure to keep us posted, please!

[scimitar 221]

I've tried numerous different things to try and solve this issue, including different binaries and memory allocators. I've also tried the most recent 1.66 release candidate and the different malloc files included with it, but with no luck. I noticed the readme.txt file in the Arma 3 DLL folder, where it mentions deleting all of the malloc files and trying the default system memory allocator. I'm still using the release candidate but deleting all of the game malloc files seems to have done the trick for me. Since switching to the Windows 10 system memory allocator, I've played several sessions ranging from 1-4 hours and I've not experienced the drastic fps drop again. For anyone experiencing the issue, it's worth a try to backup the game malloc files and then delete them from the DLL folder and give the system malloc a try. I can't say for certain that the fix is permanent for me but performance is good so far.

[domokun 515]

I've tried numerous different things to try and solve this issue, including different binaries and memory allocators. I've also tried the most recent 1.66 release candidate and the different malloc files included with it, but with no luck. I noticed the readme.txt file in the Arma 3 DLL folder, where it mentions deleting all of the malloc files and trying the default system memory allocator. I'm still using the release candidate but deleting all of the game malloc files seems to have done the trick for me. Since switching to the Windows 10 system memory allocator, I've played several sessions ranging from 1-4 hours and I've not experienced the drastic fps drop again. For anyone experiencing the issue, it's worth a try to backup the game malloc files and then delete them from the DLL folder and give the system malloc a try. I can't say for certain that the fix is permanent for me but performance is good so far.

How do you test? Do you run a particular benchmark?

How long do you benchmark before deciding that the memory leak has been fixed?

[scimitar 221]

I didn't run tests or benchmarks. I've just been playing and seeing how the performance is. I usually get the performance drop within 30 minutes to an hour. With certain malloc files I've gotten it within 10 minutes. I played a session for about 4 hours yesterday and didn't get the fps drop at all. When I deleted the malloc files I had been playing a particular mission right before doing so and I would get the bug within 15 minutes. I deleted the malloc files and went back and tried the same mission and was able to play it all the way through with no issues. I switched to the system allocator about 4 days ago and haven't had a problem since. It may just be a long running stroke of good luck but I suppose that time will tell.

[CNutter 24]

I'm curious, has BIS tried taking any surveys about this? I've heard several people say the problem happened more or less frequently depending on what mods they were running, some have said it started or went away with a hardware or driver change, and some have said upgrading (or even downgrading) their OS fixed the issue.

I think it would be quite helpful to finding the cause of the problem if BIS were to throw together a survey and toss a link to it in the launcher. If they could find a pattern, say certain hardware, mods, or peripheral combos being used among 3FPS victims that isn't present among the non-afflicted, they might be able to narrow their search for a solution.

 

Maybe even give players some kind of forum badge or in-game perk for submitting a survey, something to make them feel good about helping the devs...

[aceace33333 17]

I didn't run tests or benchmarks. I've just been playing and seeing how the performance is. I usually get the performance drop within 30 minutes to an hour. With certain malloc files I've gotten it within 10 minutes. I played a session for about 4 hours yesterday and didn't get the fps drop at all. When I deleted the malloc files I had been playing a particular mission right before doing so and I would get the bug within 15 minutes. I deleted the malloc files and went back and tried the same mission and was able to play it all the way through with no issues. I switched to the system allocator about 4 days ago and haven't had a problem since. It may just be a long running stroke of good luck but I suppose that time will tell.

 

scimitar, i'm eager to try your suggested fix, as I have been suffering from this glitch/problem for far too long.  I too have tried all kinds of malloc/.exe combinations, to no avail.  Quick question though... when you say you deleted all the mallocs in the Arma folder, did you then have to set arma to run the system malloc with a launch parameter? or will the game do that automatically.  I only ask because I thought I remember reading somewhere that if you delete the mallocs and run the game it will just re-install them.

 

So my basic question is, aside from deleting all of arma folder's mallocs, are there any additional actions that need to be taken?

 

 

P.S. @CNutter, I think the survey question might be a good idea.  And to answer what you had addressed, I've ran Arma 3 with no mods and no start-up parameters, with windows 10 OS (not from upgrade, my rig has had that OS since day 1) and i've still gotten the 3fps bug.  I'm not saying those other people haven't solved theirs by doing some of the actions you've mentioned (removing mods/upgrading OS).  But I am saying the glitch is deep enough in Arma atm that it occurs when running the most vanilla version of the game as possible.

[scimitar 221]

scimitar, i'm eager to try your suggested fix, as I have been suffering from this glitch/problem for far too long.  I too have tried all kinds of malloc/.exe combinations, to no avail.  Quick question though... when you say you deleted all the mallocs in the Arma folder, did you then have to set arma to run the system malloc with a launch parameter? or will the game do that automatically.  I only ask because I thought I remember reading somewhere that if you delete the mallocs and run the game it will just re-install them.

 

So my basic question is, aside from deleting all of arma folder's mallocs, are there any additional actions that need to be taken?...

 

Aceace33333, I took a look at the DLL folder and you are right that the malloc files have been downloaded again. I really can't completely explain the improved performance. At the time I deleted the malloc files, I also eliminated a few mods from my default SP playlist, so it's possible that I'm not exceeding the amount of used memory that seems to trigger the 3fps bug for some users. I really hadn't considered that until reading your comments and discovering that the malloc files had been redownloaded by Steam. I thought that it was necessary to verify the game files in Steam for any missing files to be replaced. The readme.txt file in the A3 DLL folder is a bit misleading as it says:

 

"The allocator is a dll placed in a directory named "dll", located next to the game executable. The allocator search order is:

* tbb4malloc_bi - based on Intel TBB 4, distributed under GPL v2 + RE (source code) based on tbb40_258oss

* jemalloc_bi - based on JEMalloc, distributed under BSD-derived license (source code)

* customMalloc_bi - not provided, feel free to plug-in your own

If no allocator dll is found, functions _aligned_malloc / _aligned_free (using Windows Heap functions) are used as a fallback. Important note: the Windows 7 allocator seems to be quite good, and it may therefore make sense for some users to delete all custom allocators on Windows 7 or newer).

You can select an allocator by deleting other allocators from the dll folder or via command line parameters."

 

​​​It doesn't say anything about Steam downloading the malloc files if they aren't found. I did notice the bit about command line parameters and did a search on that. Apparently it isn't necessary to do any file deleting to use the system malloc. You can just add "-malloc=system" to the launch parameters. At the moment I'm not going to add that parameter to my startup command line as things seem to be running fine. Should the issue return then I will give that a try. I apologize if I gave anyone false hope. What I did seemed to work but it may have had more to do with removing some mods from my playlist as I mentioned rather than anything else. However that indicates a possible temporary fix for anyone experiencing the issue that uses a significant number of mods. Try paring them down to only the ones that you consider essential and it may help alleviate the problem until BIS fixes it.

 

[aceace33333 17]

Aceace33333, I took a look at the DLL folder and you are right that the malloc files have been downloaded again. I really can't completely explain the improved performance. At the time I deleted the malloc files, I also eliminated a few mods from my default SP playlist, so it's possible that I'm not exceeding the amount of used memory that seems to trigger the 3fps bug for some users. I really hadn't considered that until reading your comments and discovering that the malloc files had been redownloaded by Steam. I thought that it was necessary to verify the game files in Steam for any missing files to be replaced. The readme.txt file in the A3 DLL folder is a bit misleading as it says:

 

"The allocator is a dll placed in a directory named "dll", located next to the game executable. The allocator search order is:

* tbb4malloc_bi - based on Intel TBB 4, distributed under GPL v2 + RE (source code) based on tbb40_258oss

* jemalloc_bi - based on JEMalloc, distributed under BSD-derived license (source code)

* customMalloc_bi - not provided, feel free to plug-in your own

If no allocator dll is found, functions _aligned_malloc / _aligned_free (using Windows Heap functions) are used as a fallback. Important note: the Windows 7 allocator seems to be quite good, and it may therefore make sense for some users to delete all custom allocators on Windows 7 or newer).

You can select an allocator by deleting other allocators from the dll folder or via command line parameters."

 

​​​It doesn't say anything about Steam downloading the malloc files if they aren't found. I did notice the bit about command line parameters and did a search on that. Apparently it isn't necessary to do any file deleting to use the system malloc. You can just add "-malloc=system" to the launch parameters. At the moment I'm not going to add that parameter to my startup command line as things seem to be running fine. Should the issue return then I will give that a try. I apologize if I gave anyone false hope. What I did seemed to work but it may have had more to do with removing some mods from my playlist as I mentioned rather than anything else. However that indicates a possible temporary fix for anyone experiencing the issue that uses a significant number of mods. Try paring them down to only the ones that you consider essential and it may help alleviate the problem until BIS fixes it.

 

good to know.  I'll still give it a try.

 

But as I said, i still get the 3fps bug with no mods running and no launch parameters.  And in case anyone thinks maybe its a really buggy/bad custom scenario, i've been able to produce the bug by just replaying the YAAB (Yet Another Arma Benchmark) over and over again.  Its hardly an intensive scenario, literally just a small firefight with scripted camera movements.  So if the bug can happen there with just vanilla arma running, I think there's a major engine bug BI needs to address during this "sweep" of theirs.

[aceace33333 17]

quick report back, I too deleted all the mallocs from Arma's DLL folder, and proceeded to run Arma for several hours playing various scenarios, with no "3fps" bug/drop.  Out of curiosity, I restored the DLLs (so arma defaulted back to the TBB4 malloc i assume) and ran arma again... just like clockwork, the 3fps drop happened about 50-60 minutes into playing.  I think it's safe to say BI still needs to find the memory leak with their DLLs.

 

This was done on the 1.66 RC branch, and actually as i was running these tests is when BI released 1.66 to stable.  So haven't replicated these tests yet with the new build, but i assume it will be the same result.

 

Also, its worth noting that running arma with all the mallocs deleted (in which arma chooses presumably chooses the "system" malloc) the performance is notably worse... about a 5-10fps drop on average, versus running the default TBB4 malloc.  But at this point, I'd rather have stable gameplay without the fear of "3fps bugging/crashing" than anything else.

 

Hopefully dwarden's profile/performance builds in 1.66 get to the bottom of this.

[scimitar 221]

quick report back, I too deleted all the mallocs from Arma's DLL folder, and proceeded to run Arma for several hours playing various scenarios, with no "3fps" bug/drop.  Out of curiosity, I restored the DLLs (so arma defaulted back to the TBB4 malloc i assume) and ran arma again... just like clockwork, the 3fps drop happened about 50-60 minutes into playing.  I think it's safe to say BI still needs to find the memory leak with their DLLs.

 

This was done on the 1.66 RC branch, and actually as i was running these tests is when BI released 1.66 to stable.  So haven't replicated these tests yet with the new build, but i assume it will be the same result.

 

Also, its worth noting that running arma with all the mallocs deleted (in which arma chooses presumably chooses the "system" malloc) the performance is notably worse... about a 5-10fps drop on average, versus running the default TBB4 malloc.  But at this point, I'd rather have stable gameplay without the fear of "3fps bugging/crashing" than anything else.

 

Hopefully dwarden's profile/performance builds in 1.66 get to the bottom of this.

 

Thanks for reporting back with your results as they are very interesting. Did Steam not automatically restore the malloc files? When I checked my dll folder earlier, I didn't pay any attention to the file dates but now they are dated 12/1 after todays update. I did notice that when I switched from the 1.66 RC build to the official 1.66 build that there was no large download, so there is apparently not much difference between the builds. Like I mentioned in my earlier post, I didn't experience the fps bug after deleting the malloc files so I really don't know if it was defaulting to the system malloc or a Steam restored BIS malloc. I didn't notice much change in performance other than a few possible instances of stuttering but my fps is usually very good so a drop of 5-10 fps might not have been noticeable. I haven't had a chance to play yet with the official 1.66 release so I don't know yet if anything has changed performance wise. At least now I know that the system malloc can be forced with a command line parameter so I will try that if the fps bug returns.

[aceace33333 17]

The only reason I notice the 5-10fps performance difference is because I run RivaTuner OSD for fps and such, and have tracked the difference between the mallocs.

 

And suprisingly, for me, since deleting the mallocs, they haven't been restored/re-downloaded yet.  I've played arma multiple times, turned computer on/off, somewhat anticipating them to eventually re-appear.  But they haven't.  So that's a good sign i guess for this fix.

[scimitar 221]

The only reason I notice the 5-10fps performance difference is because I run RivaTuner OSD for fps and such, and have tracked the difference between the mallocs.

 

And suprisingly, for me, since deleting the mallocs, they haven't been restored/re-downloaded yet.  I've played arma multiple times, turned computer on/off, somewhat anticipating them to eventually re-appear.  But they haven't.  So that's a good sign i guess for this fix.

 

I've been thinking about the files being restored. I now believe that some time after deleting them and before the 1.66 stable branch release that there was an RC branch update that restored the files. I'm not certain but I do remember an update over the last few days. I'm going to leave them for now and hopefully I'll get a chance to play some more over the next few days. If the bug returns, then I'll try deleting the files again or just use a command line parameter to use the system malloc.

[aceace33333 17]

I left the Malloc files from 1.66 as they were and played for a while last night.. no fps bug for several hours... was really optimistic.. but then it happened after a couple hours in. I always have RivaTuner OSD running, and unnoticed that the ram and pagefile useage had reached a little bit higher before the 3fps bug triggered. So I guess this means the memory isn't as bad now... I'm not quite sure.

But nevertheless it still happened, so I think I'm gonna try the system malloc for a bit and see if that fixes it.

[aceace33333 17]

I noticed*

[succo 1]

Did it work for 1.66? I'm willing to try it, but I'm not very intimate with the game files

 

Can you explain a little more in depth where these ddl files for the malloc are? 

[aceace33333 17]

Didn't work. Got the 3fps bug on both the default malloc and system malloc.... sigh

[scimitar 221]

Did it work for 1.66? I'm willing to try it, but I'm not very intimate with the game files

 

Can you explain a little more in depth where these ddl files for the malloc are? 

 

I haven't tried using the system malloc with 1.66 yet but I can help with the file locations. This default Steam/Arma 3 location is mentioned in the first post in this thread:

 

"C:\Program Files (x86)\Steam\steamapps\common\Arma 3\Dll"

 

Of course the actual location can very, depending on your OS, what drive you may have installed Steam to and whether you specified an install location other than the default location. There should be three files in the Dll folder (JEMalloc_bi.dll, readme.txt and tbb4malloc_bi.dll). Delete or backup the *.dll files and the game should default to the system malloc. If the system malloc doesn't perform well, you can either restore the backed up *.dll files (if you backed them up) or verify your Arma 3 files in Steam and they will be restored.