How to battle server crashes

[Homer Johnston 0]

Hey there, lately we're getting a ton of random server crashes - I *suspect* it's new since we began using 1.14... we can pretty much count on playing an hour, maybe 2, and then get a yellow/red chain... then look at the server and find that the console window is missing... and that there's a crash entry in the server RPT/minidump files.

Is there anything I can actually do with these RPT entries?

Is there anything I can do to combat these crashes? Maybe we'll have to play an old arma version? (:()

The server has no addons, and is not running Battleeye... it's playing some fairly large-scale coops (new question - if server isn't powerful enough and runs at low FPS for a long time, does this increase odds of crashing?)

Thanks for any input... Hopefully I'm not the only one experiencing this (or actually, hopefully i AM the only one...)

[Hud Dorph 22]

Ok server have no addons, what about clients ? Are u alowing people to join with whatever addons they like ?

I have been running a dedicated server (not rented) for a long time now and have had servercrashes all the time.

Finaly i decided to allow only certin signed addons to join my server, and guess what....... no servercrashes ever since.

[nobrainer 0]

The server has no addons, and is not running Battleeye... it's playing some fairly large-scale coops (new question - if server isn't powerful enough and runs at low FPS for a long time, does this increase odds of crashing?)

This is what I have experienced at least. I know others do, but I don't recommend playing when server FPS goes belove 15, then there is a certain chance of hanging the server.

But there are other people here that knows more about servers than me, but I run 1 win xp 32-bit sp-3 server and 1 win 2k3 32-bit server. The first one is a pc just designed for ArmA dedi with Intel Core 2 Duo and the last one is a HP 380 dual xeon server. The first one runs much better than the Xeon. But both crashes when server FPS gets too low.

I hope u can use some of this.

Best retard NoBrainer

[Auss 208]

2 things you can do, use verify signitures OR use Doolittles system and prevent certain mods from being used. I have blocked 4 mods and found a massive increase in server reliability. I've blocked XAM, SLX, ECS and compact fix.

[deady 0]

Still get server crashes here too occasionally. I don't allow any mods.

Sometimes it crashes, othertimes it will just lockup and a game won't begin until I shutdown the server and restart it again.

[Dwarden 1125]

send all server crash dumps to BIS support email ...

[Auss 208]

Quote[/b] ]send all server crash dumps to BIS support email

If we all did that the amount of traffic to that mailbox would crash their mail server.....

[oktane 0]

I sent a message to them last week, and dump files.. no response.

We keep getting crashes in the same two places. I cannot figure out what is causing them.. I crc checked all the files, running vanilla, even took out as many server/profile/etc settings as I could. Almost always, the crashes occur in these two code locations:

The crashes occur at the two following locations:

Fault address:  00530C26 01:0012FC26 C:\Servers\ArmaPublic\arma_server.exe

Fault address:  005E5CA9 01:001E4CA9 C:\Servers\ArmaPublic\arma_server.exe

If these two functions would stop crashing so much, I could deal with the other occasional odd crashes. Can anyone else check their servers arma_server.RPT file to see if you get these too?

[Auss 208]

I have sent my server crash logs to BIS support. I sent at least 4 emails with the server specs and detailed list of how the server is setup. This was the reply back.

"We dont know what is causing the problem, we have sent your logs to the developers" .......

That was 2 months ago I havent heard anything back, I haven't bother to continue as they seem to not really care. I hope you have more success than I did.

[mr.g-c 6]

Having still frequently crashes as soon as we are getting above a certain level like ~20 Players.

Its always Segfaulting.... Kern-log is full of them at every point it crashes:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">Nov  9 20:15:20 arma-rpg kernel: [4236679.001871] server[14151]: segfault at 4 rip 8b6892c rsp ffc479d0 error 6

I hope they fix that for the next patch - its getting really annoying.

[oktane 0]

the hex address at RIP, is it often the same?

PS: Also no reply from BIS.. idea support did the same, forward to BIS but no reply.

[mr.g-c 6]

No, the adress always changes.

But its always erro6....

72Hrs of CPU-Burn test + 24 Hrs of Memtest86 showed no errors in the hardware...

Its Ubuntu X86-64bit 8.04 Server LTS btw...

[mr.g-c 6]

And another one... any input?

Quote[/b] ]Dec 1 21:41:15 arma-rpg kernel: [5905669.374743] server[20253]: segfault at 0 rip 8ba50f1 rsp f743f8c0 error 6

[ck-claw 1]

Does anyone know if turning 'verify signatures' on (from your expierence) increases stabillity?

EDIT:-

Nvm -Testing Doolittles thingy!

[rip31st 98]

In my opinion running a server - Improper server settings such as min/maxbandwidth, MaxSizeGuaranteed miscalculated. Amount of AI in a mission or objects a server tracks. Type of connections players have to the server are all relative to server stability. Mods can also hog up bandwidth and cause instability.

Missions can cause it:

-Too many ai or objects=crash.

-Too many dead bodies, objects, vehicles not getting cleaned up=crash.

-Sloppy lose ended code within the mission=crash.

Imagine a server where dead bodies/vehicles are scattered or not cleaned up efficiently in a timely manner. Many bodies start stacking up. People come into contact with the dead bodies here and there. The server has to continuously track them and make sure players can see them while performing other vital tasks. Eventually too many bodies/vehicles whatever are resident in the Servers memory...Kaboom! Server goes down.

OR imagine a server with a high amount of AI. The server does well when tracking the normal player VS player or player traffic, but when 80 AI and AI operated vehicles all the sudden are brought into the mix, the server starts to bog down. Now multiply this factor as players branch out to different towns activating AI in other areas. Then add tracers, sound effects, bullets the server has to track, then you get problems. Server eats it, goes down.

Player connections can cause it:

-Too much desync with a combo of high network traffic=crash.

Say for example 20 people with a low ping, good connection are on the server. Now someone with a high ping low bandwidth jumps in. Now all the clients have to wait for the server to distribute all the messages equally. No big deal. If their ping is 250 that's only a quarter second difference advantage the person with the bad connection now has. Combine that with a few more people joining the game with not so great conns and there will be issues. The amount of messages stacking up on the server lowers the FPS and the server eventually crashes because there is no longer room to store all these messages and it can't discard them because it's waiting for a client to either send or receive messages. People joining a server with a broad spectrum of pings will wreak havoc every time.

Ever get the red link and AI look like they are running in place or people can't move, or they can't enter or control vehicles?...Exactly. Sometimes the server recovers, other times you're not so lucky. It can be frustrating to wait 5-10 minutes for the person with the bad conn to leave or admin kick him and the bad network messages that got backed up get cleared out. Other games deal with this issue differently. They discard unanswered messages resulting in players appearing in one place and then 100 yards away 1 second later. Sure the server doesn't go down, but game play is ruined.

Mods can cause issues:

-Too many different type of mod data being tracked by the server can cause issues.

Say for example 5 people are playing on a server running Vanilla ArmA. For those that don't know what Vanilla is. It's ArmA out of the box unmodified. Then someone with a sound mod/effects jumps into the game. Those Messages use to send and receive data about the sounds or effects are enlongated because thier sounds/effects differ from the original sounds and may be longer or shorter. The messages begin to stack up waiting for a response from the other players without the sound mod because their system already responded by sending the default data but all the sudden the server tries to send the same event twice. When the responses go unanswered, they get stored. Another guy with a mod that modifies the AI behavoir enters the server. His AI respond differently then the other players in a unexpected manner. Sometimes they take too long to perform a task or take a unexpected route or engage other players or other AI a lot sooner then expected at farther distances. The server doesn't expect the results. His AI can see and engage targets further away. The server receives messages that it's AI or other players are being killed before expected. The messages are stored waiting for reception from the players without the mod and eventually bog down the server. Resulting in weird unexpected results like people dying without hearing a shot or dying unexpectedly. Yhen someone with a mod that modifies burning vehicles and smoke effects enters the server. His vehicles burn for 20 minutes before being deleted. The other players vehicles burned for 5 before they were deleted. The guy with the mod sends his message finally his vehicle has been deleted to the server. Server send the message again. Because the message has already been answered by the client, the client ignores the message. The messages are stored on the server. Compound that by a factor of the amount of other players. Eventually it bogs down the server.

My advice:

-Play without the mods, or limit them to the same mod with the same files. Run the server with equalModRequired=1

-Limit the amount of AI or objects being tracked. Run a script to delete the dead bodies and objects shortly after they are dead. IF you're not running the latest quad core xeon server don't be putting 40-80 AI in a warfare/domination/evolution town. Get rid of AI and objects within the mission and limit them. OR just run player VS player missions.

-Check signatures to avoid players coming into the game with different versions then the server. Unfortuneatley some source files aren't the same like Atari, Steam or the store bought boxes.

-Server admins need to calculate bandwidth properly. Find out the true connection of your server to the internet. It's pretty common to find a server that has a 1mbit connection running a game that says "100mbit connection!!!". Then limit the amount of players that can join after you find your servers bandwith.

-Players - Avoid joining servers that give you a high ping. If it's yellow that means caution in real life. If its red don't bother.

-Admins - Have ingame admins to monitor connections. Players with too high of a ping/desync or bandwidth problems, kick them out. No hard feelings.

Anyway, goodluck!

[mr.g-c 6]

Unfortunately, nothing of what you are writing is any real solution for the core problem.

The Dedicated Server is simply improper coded/bugged (i'm not alone with that opinion, lol).

You actually should await a crash-free dedicated server, in every other game i hosted servers in the past (quite a few), it was running without any problems (at least after 2 years and many patches) - only Arma is the exception as usual...  

Nothing against you Rip31st, but you seems to be very new in this dedicated server thing around Arma/OFP Engine, while many of us already hosted Dedicated Servers back in OFP1 times and already have quite some experience with it.

Anyway, there are a few things which will be fixed in the upcomming patch and fixes some of your mentioned (and already knowed since long times) issues... read it here:

http://www.flashpoint1985.com/cgi-bin....t=74051

and:

http://www.flashpoint1985.com/cgi-bin....1268528

[ck-claw 1]

Well so far, after a day running Doolittles little program stopping SLX,ECS,XAM and a few other things.

Server been full most of the day and only one crash so far!  

But tbh when monitoring the server you'll be surprised how many ppl don't run any mods other than QG.

[Otter 0]

ArmA. For those that don't know what Vanilla is. It's ArmA out of the box unmodified. Then someone with a sound mod/effects jumps into the game. Those Messages use to send and receive data about the sounds or effects are enlongated because thier sounds/effects differ from the original sounds and may be longer or shorter. The messages begin to stack up waiting for a response from the other players without the sound mod because their system already responded by sending the default data but all the sudden the server tries to send the same event twice. When the responses go unanswered, they get stored. Another guy with a mod that modifies the AI behavoir enters the server. His AI respond differently then the other players in a unexpected manner. Sometimes they take too long to perform a task or take a unexpected route or engage other players or other AI a lot sooner then expected at farther distances. The server doesn't expect the results. His AI can see and engage targets further away. The server receives messages that it's AI or other players are being killed before expected. The messages are stored waiting for reception from the players without the mod and eventually bog down the server. Resulting in weird unexpected results like people dying without hearing a shot or dying unexpectedly. Yhen someone with a mod that modifies burning vehicles and smoke effects enters the server. His vehicles burn for 20 minutes before being deleted. The other players vehicles burned for 5 before they were deleted. The guy with the mod sends his message finally his vehicle has been deleted to the server. Server send the message again. Because the message has already been answered by the client, the client ignores the message. The messages are stored on the server. Compound that by a factor of the amount of other players. Eventually it bogs down the server.

I admit I'm pretty new to scripting and mods, but I'm not sure this is correct, sound mods etc. should only run on the client PC and the message the client sends to the server would be exactly the same as the "vanilla" version.

[Rommel 2]

5 and half Days and still running. Our own mods plus the allowance of DMSmokeEffects/FX mods and Sound mods.

Around 400 Ai spawning on and off during gameplay. I reckon thats pretty bloody good.

[ck-claw 1]

Very good Rommel!  

I just gave up in the end and put on -No Mods

No crash in 2 days,which is a good improvement

[xeno 234]

Well, with 1.15 crashes are gone. We are used to play with a lot of addons and with 1.14 we had up to 4 or 5 crashes each day (or even more).

Xeno

[ck-claw 1]

Thats nice to know Xeno.

I did some what get the crashes down to 1/2 a day with DooAcs.

Then started to climb a bit so put to no mods.

Which alot of our regulars appreciated.

So roll on the full 1.15  

[Rommel 2]

Thats nice to know Xeno.

I did some what get the crashes down to 1/2 a day with DooAcs.

Then started to climb a bit so put to no mods.

Which alot of our regulars appreciated.

So roll on the full 1.15  

This was on 1.14 mind you, can't wait to see how 1.15 handles and improves things.

Should blow my mind.  

EDIT: New record: 6.7625days

[code34 248]

do you know when the linux server 1.15 will be avaible ?