Server x64 crashes?

[nuvix 0]

Hi all, since the ARMA 3 x64 update we're experiencing a lot of problems with ARMA 3 crashing on the server side. Just wondering if anyone here also has the same issues.

Main issue is that at random times (usually between 1, 1,5 hour of playing we get this error:

Exception code: C0000005 ACCESS_VIOLATION at F7626C5B this is a server side crash..

Lots of client side crashes as well, also ACCES VIOLATION something something something..

To all people who are hosting servers, do you experience these problems as well? If so what did you do to solve it? So far I tried setting: -malloc=system or -malloc=tbb4malloc_bi_x64.dll disabled BattleEye.

Or we get on the server side messages like this:

2017/03/23, 19:54:57 Server: Network message 117e91 is pending
2017/03/23, 19:54:57 Server: Network message 117e92 is pending
2017/03/23, 19:54:57 Server: Network message 117e92 is pending
2017/03/23, 19:54:57 Server: Network message 117e92 is pending
2017/03/23, 19:54:57 Server: Network message 117e92 is pending
2017/03/23, 19:54:57 Server: Network message 117e93 is pending
2017/03/23, 19:54:57 Server: Network message 117e93 is pending
2017/03/23, 19:54:57 Server: Network message 117e94 is pending
2017/03/23, 19:54:57 Server: Network message 117e94 is pending
2017/03/23, 19:54:57 Server: Network message 117e95 is pending

And this:

2017/03/23, 19:55:07 Client: Nonnetwork object 18000640.
2017/03/23, 19:55:07 Client: Nonnetwork object 2beb4800.
2017/03/23, 19:55:07 Client: Nonnetwork object 18000640.
2017/03/23, 19:55:07 Client: Nonnetwork object 2beb4800.
2017/03/23, 19:55:07 Client: Nonnetwork object 18000640.
2017/03/23, 19:55:07 Client: Nonnetwork object 2beb4800.
2017/03/23, 19:55:07 Client: Nonnetwork object 18000640.

 

Info about the server: 32 GB of RAM, 2 CPU's, Win2012R2, 10 player slots and playing the scenario Antistasi with mods ACE, RHS, XLA and CBA.

Anybody has seen this?

-nuvix

[nuvix 0]

RPT file#1: http://pastebin.com/9siHNDPE
RPT file#2: http://pastebin.com/U3kiBShn

 

Als it's 24GB of RAM not 32. :)

[terox 316]

Looks like mission issue..

Quote

2017/03/23, 18:11:00 Call extension 'ace_medical' could not be found
2017/03/23, 18:11:00 [ACE] (medical) INFO: Extension ace_medical.dll not installed.
2017/03/23, 18:22:23 Warning Message: You cannot play/edit this mission; it is dependent on downloadable content that has been deleted.
rhs_c_troops
2017/03/23, 18:32:44 Warning Message: You cannot play/edit this mission; it is dependent on downloadable content that has been deleted.
rhs_c_a2port_air
2017/03/23, 18:40:02 Warning Message: You cannot play/edit this mission; it is dependent on downloadable content that has been deleted.
rhsusf_c_uav

Following looks like an issue with hit eventhandler/handledamage eventhandler system , Could be ace medical or a revive system

Quote

2017/03/23, 19:12:44 Error compiling '' in 'HitPelvis'
2017/03/23, 19:12:44 Error compiling '' in 'HitAbdomen'
2017/03/23, 19:12:44 Error compiling '' in 'HitDiaphragm'
2017/03/23, 19:12:44 Error compiling '' in 'HitChest'
2017/03/23, 19:12:44 Error compiling '' in 'HitArms'
2017/03/23, 19:12:44 Error compiling '' in 'HitLegs'

 

[vabene1111 2]

We are having similar issues.

Our two Server crash around 2-6 Times a Day, this only started happening after the x64 Update an never happend before, mostly during prime time with high loads on the Server (~70-80 Players).

 

Next Time it crashes i will include some RPT files.

 

Also coming from the Life Community i know that many other Server are experiencing the same Problem and are desperately waiting on a Fix from Bohemia, as there is broad consent that this must be on bohemias side (i myself haven't analysed the logs so i can only repeat what i hear/read).

 

EDIT: 

To have an Idea of the frequency: Our server ran the whole day and than, when the load rose, started crashing : 17:50, 18:16, 20:11, 20:36

 

Here are the RPT Logs just before the crashes occurred https://drive.google.com/uc?id=0B_TmB6GeWD7PUlh0ZS1jbzBpaWM

The Logs are pretty small, as Server generally run better with the -noLogs Parameter, i might turn the Logs on this afternoon to generate a few complete sets if needed.

 

As i am not sure if the mdmp, bidmp files contain sensitive Information, i will not post them publicly, but if any BI DEV is interested in taking a look please send me a Message.

 

Edited March 24, 2017 by vabene1111

[nuvix 0]

Yeah, but none of those are game breaking at all. And mods will always generate errors. The error that entirely stops the game is the ACCESS_VIOLATION.

[vabene1111 2]

Yea thought so, but if BI says they need it i will provide it .. right now i would send them a copy of the server or let them look via TeamView (or whatever they need) because these crashes are really annoying, for both our admin Team (constantly restarting & explaining to / helping players) and for the players (since they cant really play normally.

[terox 316]

Well I can confidentally state that our milsim servers do not have this issue, therefore it is most likely specific to Altis Life, and guessing at a Database, addon, DLL related issue

You could try and prove a vanilla server to verify this

[nuvix 0]

We are running a scenario called Antistasi (which is very cool btw check it out here: http://www.a3antistasi.com/)

 

Vannilla or with mods, we had the same crashes. However I did turn off the following -loadMissionToMemory OFF and using this as well: -malloc=tbb4malloc_bi_x64.dll server has now been stable for 4-5 hours. So far looking good, gonna start stress testing now and see if it holds up.

[Buddard 0]

nuvix thanks for the link! started to have headaches searching for it. no, seriously. i even have used an Analgin because of the headaches.

[terox 316]

On 24/03/2017 at 3:31 PM, nuvix said:

We are running a scenario called Antistasi (which is very cool btw check it out here: http://www.a3antistasi.com/)

 

Vannilla or with mods, we had the same crashes. However I did turn off the following -loadMissionToMemory OFF and using this as well: -malloc=tbb4malloc_bi_x64.dll server has now been stable for 4-5 hours. So far looking good, gonna start stress testing now and see if it holds up.

 

 

So am assuming you weren't using the default memory allocator then ?

 

We almost always ask for the rpt file, it is full of information that can help us identify issues, so please if you have any issues in future, always copy the contents of all the rpt into a pastebin link

 

The very start is really important, see the example below.

It shows what mods you are actually running, the branch and version and............... the memory allocator being used

Quote

 

=====================================================================
== D:\Games\ArmA3\A3S_No1\arma3server_x64.exe
== "D:\Games\ArmA3\A3S_No1\arma3server_x64.exe" -port=2302 "-profiles=D:\Games\ArmA3\A3S_No1" -config=Config_No1.cfg  -enableHT -loadMissionToMemory  -filepatching -servermod=@ZCBA_A3;@ZServer1

Original output filename: Arma3Retail_Server_x64
Exe timestamp: 2017/03/16 10:23:13
Current time:  2017/03/30 18:28:53

Type: Public
Build: Stable
Version: 1.68.140908

Allocator: D:\Games\ArmA3\A3S_No1\Dll\tbb4malloc_bi_x64.dll
PhysMem: 24 GiB, VirtMem : 8192 GiB, AvailPhys : 19 GiB, AvailVirt : 8192 GiB, AvailPage : 25 GiB

 

 

[eggbeast 3684]

we have the exact same problem running evolution mp co-op missions with unsung mod.

 

there were zero server crashes before the update using the same mod version and same 3 missions in rotation for the past 3 months

now, approximately every 2 hours we get a crash.

 

we have been using dwardens updated exe's etc and still no luck fixing it.

 

we believe it is caused by a mixture of clients running 32 bit and 64 bit exes. the server is running 64 bit.

 

it is impossible to manage which exe a client runs on a popular, busy public server. but i can say that once we told everyone to switch to 64 bit, we are getting maybe 3-4 hours of game running before some random public joins and bam! the server goes down again.

 

therefore, we are certain that:

1. it is not the mission (happens on a range of them)

2. it is not the server configs - never happened before the update

3. it is not the mod (ran stable every single day for 3 months+ prior to update)

4. it is the latest 64 bit game update from BI

5. it is an instability issue caused by random JIP clients

6. it is a memory leak issue (Access violation is always EXACTLY THE SAME in every rpt file)

 

we have submitted a bunch of rpts and mdmps to BI, still unfortunately no solution after 3 weeks of hoping.

We are all playing other games until this is fixed

it has destroyed our regular unit ops, recruit training and daily gaming in a pretty devastating way.

a lot of regulars message me daily asking if it is fixed yet. we are powerless to do anything, and just re-send BI the log files every week or so, usign the latest exes.

 

 

[eggbeast 3684]

to anyone wit h these crashes - have you tried running your server (and thus forcing clients) on DEV branch?

I think we will try this next. as a group of us updated to the latest exe's we got from dwardens posts, and we played fine for 4 hours or more last weekend, then another random public joined and bam! server crashed. so perhaps forcing all players to use different exes by going to dev branch may help clear up the issue... worth a try eh

 

[Dwarden 1125]

i got the crashdumps, we will take look

[eggbeast 3684]

thanks dwarden, we'll run some tests as discussed into next week and send yo ua set of dumps based on certain conditions:

32 bit server

64 bit server

latest trial exe on server and all clients in closed session

dev branch - all clients and server

etc

[Justin Waters 0]

Was there a verdict reached with this error?