DediCated Servers Crash

[aussie dave 0]

it has been running 18+ for the last couple of hours now.

last nite it was full for hours on the 16 player mode.

jump on now as it has 21 players in it. see if we can crash it.

[FinnTux 0]

We are using the base start line so that no addons are running on the server however are allowing client side addons on our server. Our server has been running with no crashes for over 24 hours now.

We've been running ours without any mods on the server and yet we get frequent crashes.

[FinnTux 0]

Just wondering if it would be possible to coordinate a test to try to solve this issue? All we need is a server and ~30 players willing to help a bit. We could test different scenarios: No mods at all+warfare32, all mods people can think of etc.

What do you think?

[[kh]jman 49]

Our dedicated server (Windows Server 2003 x64) is usually very stable and as a clan we have spent many hours working on fine tuning it. It's been available to the community for over a year and a half now so I know what I'm talking about in relation to arma server's daily use. I also have commercial experience in dedicated server administration.

In the last two days we have had five crashes. On reading the .rpt files the two critical crashes yesterday were caused by mission script loops that consumed all the of servers available memory until the server itself could no longer operate. In those two circumstances I had to terminal service in and manually restart the service (we use Firedaemon). Usually if the arma server crashes Firedaemon automatically restarts it after a 3 minute timer but in these cases Firedaemon could not function since the server was out of memory. The mission being played on both these occasions have been played before and contain scripts that are used on other missions that are 6-8 months old. I've contacted the author of the mission and passed on the .rpt files. After his testing he agrees with me that missions scripts that used to work no longer function correctly since 1.14.

We have had a further three crashes today all of which are presently unexplained since the .rpt files for these give limited information. They have all occurred roughly one hour into the arma server's uptime.

I do not agree it has anything to do with client mods. We run with signature checks on, we always have and do not let anything on which will potentially alter the server runtime. We presently run with BE on, VON on and custom faces/snd disabled.

If anyone wishes further information i.e. to exchange rpt files please visit our forums.

[FinnTux 0]

Just for the record...Signature checks on, only FDF_Sounds allowed and not a single crash in 24 hours. We had around 10 players during the early evening and then we played one whole mission of warfare and had constantly >20 (IIRC almost 5 hours) players.

I'll buy a virtual beer to the person who pinpoints the mod/addon causing all these problems...

[aussie dave 0]

well i have had no crashes yet.

But when we removed Warfare missions from the warfare.pbo the test server cashes as soon as you join it.

anyone got any ideas on how to remove the Warfare missions?

Warefare + client mods = server crashes.

thats what it looks like so far with our testing.

[[kh]jman 49]

Ok I'm baffled too. Our server has not crashed yet on Warfare and it's been on plenty.

15:57pm - 1:54am. 9 different Coops completed. No crashes and a very busy and full server 30 -35 players all night. I cannot see much of a pattern emerging.

[evilnate 0]

Ok I'm baffled too. Our server has not crashed yet on Warfare and it's been on plenty.

15:57pm - 1:54am. 9 different Coops completed. No crashes and a very busy and full server 30 -35 players all night. I cannot see much of a pattern emerging.

How many mods/custom addons are you running on your server 1.14?

Do you allow clients to have unsigned addons?

Do you allow clients to run with mods?

[cross 1]

Maps do not contain addons but we allow certain addons/mods, list which can be found in our website.

Signature check on.

Battleye on.

[FinnTux 0]

Isn't it obvious? At least one spesific client side addon/mod causes this. Just need to find out which one.

[Nick (SS) 0]

I can say with out a doubt there are ALLOT of poorly scripted client side addons that do nothing but cause server crashes, what makes it worse is the way ppl mix these addons, it really does drive me nutz, ArmA should only load addons the server is using, this would also get rid of some of the client side cheats

win win

[mattxr 9]

Im confused too, i can only see it being a Patch 1.14 problem as everyone was fine without server crashes before the patch and now after the patch everyone is getting totaly RANDOM crashes which are varing all day every day.. so i come to the concluson

Patch 1.14 somewhere is casuing these issues.

[lt.chris 0]

What are your dedi specs lads?

this will help me draw some results with the testing as well, im just going though all the possbilties

[Hud Dorph 22]

Mine are Core2 Duo E6600 @3100Mhz 4GB ram on a 25/25Mbit

Running nothing but Arma-server and HLSW

[aussie dave 0]

Server Spec's:

Windows Version Microsoft Windows Server 2003 Enterprise Edition Service Pack 2 (Build 3790)

GamingSA, located in the ADNAP datacentre in the heart of Adelaide with fibre optic high speed veins branching out to provide high-speed access for all.

thats it

[aussie dave 0]

I haven't had any server crashes from 0.000: NetLogger: start - Sat May 24 09:45:51 2008

The last Mission that was played:

mission="Warfare v1.0M";

i have noticed that the arma_server.exe keeps using the CPU about 2-3% after Warfare has been played and the server is empty.

I have tested:

ArmA Effects mod by Maddmatt

Extended Eventhandlers by Solus & Killswitch

and they are O.K.

but today this player joined a empty server with these addons and the server crashed:

Quote[/b] ]

90753.201: Ch(1):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90763.369: Ch(0):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90769.257: Ch(1):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90769.431: Pe(1):rCtrlB(124.185.94.60:60733,182,ccca1e12)

90769.431: Pe(1):ask(124.185.94.60:60733,'Oneil',1074542653,1074542653,65,0)

90769.434: Ch(90):open(202.136.100.194:2302 <-> 124.185.94.60:60733)

90769.434: Ch(90):acc(124.185.94.60:60733,'Oneil',1074542653,1074542653)

90769.434: Ch(90):sInst(3.00,6.00,1.00,0.00,12)

90769.445: 9:37:21 Oneil uses modified data file - CA;@ArmAEffects;@TrueMods;@Extendedevents;@RUG_DSAI;AACFSOUND;Six_tracer;SIX_We

90769.445: 9:37:21 Player Oneil connecting.

90769.510: Ch(90):uRTT(75.90,2,1,75930,27,75.90)

90773.058: Ch(90):sInst(3.62,3.55,0.27,0.00,0)

90773.147: Ch(90):uRTT(76.42,2,43,89233,11604,77.63)

90773.268: Ch(90):sHB(0.21,0.12,0.21,0.00,0)

90773.466: Ch(90):sHB(0.41,0.32,0.20,0.00,0)

90773.489: Ch(90):st(43,60,115119,60,38,35,2,3,5/40,36,0.02,16/8,9,0.02,32,0)

90773.665: Ch(90):sHB(0.61,0.52,0.20,0.00,0)

90773.871: Ch(90):sHB(0.81,0.72,0.21,0.00,0)

90774.069: Ch(90):sHB(1.01,0.92,0.20,0.00,0)

90774.270: Ch(90):sHB(1.21,1.12,0.20,0.00,0)

90774.469: Ch(90):sHB(1.41,1.32,0.20,0.00,0)

90774.669: Ch(90):sHB(1.61,1.52,0.20,0.00,0)

90774.871: Ch(90):sHB(1.81,1.72,0.20,0.00,0)

90775.074: Ch(90):sHB(2.02,1.93,0.20,0.00,0)

90775.269: Ch(90):sHB(2.21,2.12,0.19,0.00,0)

90775.469: Ch(90):sHB(2.41,2.32,0.20,0.00,0)

90775.669: Ch(90):sHB(2.61,2.52,0.20,0.00,0)

90775.873: Ch(90):sHB(2.82,2.73,0.20,0.00,0)

90776.069: Ch(90):sInst(3.01,2.92,0.20,0.00,0)

90776.163: Ch(90):uRTT(76.53,2,58,94085,17293,76.79)

90776.279: Ch(90):sHB(0.21,0.12,0.21,0.00,0)

90776.494: Ch(90):sHB(0.43,0.33,0.22,0.00,0)

90776.722: Ch(90):sHB(0.65,0.56,0.23,0.00,0)

90776.954: Ch(90):sHB(0.88,0.79,0.23,0.00,0)

90777.185: Ch(90):sHB(1.12,1.02,0.23,0.00,0)

90777.412: Ch(90):sHB(1.34,1.25,0.23,0.00,0)

90777.604: Ch(90):sHB(1.54,1.44,0.19,0.00,0)

90777.633: Ch(90):st(42,59,115119,59,2,2,2,3,20/0,2,0.03,1/46,49,0.03,15,0)

90777.800: Ch(90):sHB(1.73,1.64,0.20,0.00,0)

90777.997: Ch(90):sHB(1.93,1.83,0.20,0.00,0)

90778.189: Ch(90):sHB(2.12,2.03,0.19,0.00,0)

90778.384: Ch(90):sHB(2.31,2.22,0.19,0.00,0)

90778.584: Ch(90):sHB(2.52,2.42,0.20,0.00,0)

90778.796: Ch(90):sHB(2.73,2.63,0.21,0.00,0)

90779.120: Ch(90):sInst(3.05,2.96,0.22,0.00,0)

90779.219: Ch(90):uRTT(77.98,2,73,98311,16944,81.37)

90779.345: Ch(90):sHB(0.22,0.13,0.22,0.00,0)

90779.465: Ch(0):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90779.662: Ch(90):sHB(0.54,0.44,0.20,0.00,0)

90779.855: Ch(90):sHB(0.73,0.64,0.19,0.00,0)

90780.038: Ch(90):sHB(0.92,0.82,0.18,0.00,0)

90780.217: Ch(90):sHB(1.10,1.00,0.18,0.00,0)

90780.441: Ch(90):sHB(1.32,1.22,0.22,0.00,0)

90780.620: Ch(90):sHB(1.50,1.40,0.18,0.00,0)

90780.840: Ch(90):sHB(1.72,1.62,0.22,0.00,0)

90781.022: Ch(90):sHB(1.90,1.80,0.18,0.00,0)

90781.229: Ch(90):sHB(2.11,2.01,0.21,0.00,0)

90781.412: Ch(90):sHB(2.29,2.19,0.18,0.00,0)

90781.602: Ch(90):sHB(2.48,2.38,0.19,0.00,0)

90781.637: Ch(90):st(42,59,106557,59,2,2,3,3,20/1,2,0.03,2/81,89,0.03,16,0)

90781.785: Ch(90):sHB(2.67,2.57,0.18,0.00,0)

90781.974: Ch(90):sHB(2.85,2.76,0.19,0.00,0)

90782.161: Ch(90):sInst(3.04,2.94,0.19,0.00,0)

90782.270: Ch(90):uRTT(78.54,2,89,108339,28507,79.83)

90782.342: Ch(90):sHB(0.18,0.07,0.18,0.00,0)

90782.532: Ch(90):sHB(0.37,0.26,0.19,0.00,0)

90782.722: Ch(90):sHB(0.56,0.45,0.19,0.00,0)

90782.904: Ch(90):sHB(0.74,0.63,0.18,0.00,0)

90783.095: Ch(90):sHB(0.93,0.83,0.19,0.00,0)

90783.286: Ch(90):sHB(1.12,1.02,0.19,0.00,0)

90783.490: Ch(90):sHB(1.33,1.22,0.20,0.00,0)

90784.212: Ch(90):sHB(2.05,1.94,0.72,0.00,0)

90784.234: 9:37:35 Player Oneil connected (id=XXXXXXXX).

90784.439: Ch(90):sHB(2.28,2.17,0.19,0.09,0)

90785.476: Ch(1):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90785.717: Ch(90):st(42,59,98191,59,2,1,4,3,12/2,1,1.27,2/61,65,1.37,9,0)

90785.898: Ch(90):sMerS(1074542653,4818,5)

90788.458: Ch(90):sInst(6.30,6.19,0.08,0.00,0)

90788.534: Ch(90):uRTT(77.71,2,138,75802,29,75.77)

90789.952: Ch(90):st(63,89,85062,89,63,66,3,3,2/60,64,0.10,18/2,2,0.18,45,0)

90791.388: Ch(90):sHB(2.93,2.85,0.49,0.49,0)

90793.361: Ch(90):sInst(4.90,4.83,1.97,0.00,0)

90793.437: Ch(90):uRTT(77.30,2,169,76361,27,76.33)

90793.724: Ch(90):sHB(0.36,0.29,0.36,0.29,0)

90794.010: Ch(90):st(63,89,77842,89,1,1,2,3,4/4,5,0.28,7/0,1,0.20,3,0)

90795.563: Ch(0):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90795.704: Ch(90):sHB(2.34,2.27,1.97,0.00,0)

90796.068: Ch(90):sHB(2.71,2.63,0.36,0.36,0)

90797.679: Ch(90):sInst(4.32,4.24,1.61,0.00,0)

90797.757: Ch(90):uRTT(77.48,2,173,77935,25,77.91)

90798.070: Ch(90):st(62,89,64910,89,1,1,2,3,3/2,1,0.18,4/2,1,0.10,5,0)

90798.399: Ch(90):sHB(0.72,0.64,0.50,0.43,0)

90800.375: Ch(90):sHB(2.70,2.62,1.97,0.00,0)

90800.737: Ch(90):sInst(3.06,2.98,0.36,0.36,0)

90800.820: Ch(90):uRTT(77.30,2,178,76902,28,76.87)

90801.631: Ch(1):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90802.099: Ch(90):st(62,87,55174,87,1,1,2,3,3/5,1,0.85,4/4,1,0.75,7,0)

90802.701: Ch(90):sHB(1.96,1.88,1.45,0.00,0)

90804.142: Ch(90):sInst(3.40,3.32,1.44,0.00,0)

90804.217: Ch(90):uRTT(76.76,2,185,75560,39,75.52)

90805.033: Ch(90):sHB(0.89,0.82,0.51,0.00,0)

90806.161: Ch(90):st(62,87,40288,87,1,1,2,3,4/3,1,0.72,5/1,1,0.65,6,0)

90807.374: Ch(90):sInst(3.23,3.16,0.93,0.00,0)

90807.452: Ch(90):uRTT(76.98,2,192,77550,48,77.50)

90810.046: Ch(90):sHB(2.67,2.59,0.62,0.53,0)

90810.217: Ch(90):st(62,87,36149,87,1,1,1,3,3/5,1,0.17,6/0,1,0.09,6,0)

90811.659: Ch(0):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90812.042: Ch(90):sInst(4.67,4.59,0.61,0.00,0)

90812.120: Ch(90):uRTT(77.13,2,201,77514,30,77.48)

90814.268: Ch(90):st(61,85,32419,85,1,1,1,3,1/5,1,0.84,7/0,1,0.76,4,0)

90814.381: Ch(90):sHB(2.34,2.26,0.95,0.00,0)

90817.035: Ch(90):sInst(4.99,4.91,0.60,0.52,0)

90817.119: Ch(90):uRTT(77.49,2,210,78352,27,78.33)

90817.839: Ch(1):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90818.311: Ch(90):st(61,85,29043,85,1,1,1,3,4/5,1,0.86,6/0,1,0.79,6,0)

90819.037: Ch(90):sHB(2.00,1.92,0.60,0.00,0)

90821.375: Ch(90):sInst(4.33,4.26,0.51,0.00,0)

90821.452: Ch(90):uRTT(77.39,2,217,77181,28,77.15)

90822.366: Ch(90):st(61,85,25983,85,1,1,1,3,2/6,1,0.23,7/0,1,0.91,5,0)

90824.026: Ch(90):sHB(2.65,2.57,0.58,0.50,0)

90826.035: Ch(90):sInst(4.66,4.58,0.52,0.00,0)

90826.116: Ch(90):uRTT(78.66,2,226,81648,30,81.62)

90826.408: Ch(90):st(61,85,23214,85,1,1,1,3,4/4,1,0.37,7/0,1,0.29,6,0)

90827.754: Ch(0):st(31,31,0,31,31,0,0,3,0/0,0,-1.00,0/0,0,-1.00,0,0)

90828.376: Ch(90):sHB(2.34,2.26,0.94,0.00,0)

90978.878: Ch(90):sInst(152.84,152.76,149.36,0.00,19)

90978.880: Ch(90):sStar(0.00,152.76,0.00,0.00,0)

90978.880: Ch(90):sStar(0.00,152.76,0.00,0.00,0)

90978.881: Ch(90):sStar(0.00,152.76,0.00,0.00,0)

90978.881: Ch(90):sStar(0.00,152.76,0.00,0.00,0)

90978.881: Ch(90):sStar(0.00,152.76,0.00,0.00,0)

[[kh]jman 49]

For information. Our server just had it's first crash on Warfare v1.0M 32 Players. Server was full at the time.

[Knight31 0]

Our crashes are becoming less. Was on for 7 hours last night and had 1 lock up on Warfare v1.0M 32 player. The server was packed and it locked up when about 4 guys joined at once. It really wasnt a crash though. It just locked up so I had to restart the server. It seems as though the server is crashing however Im not online when it happens. According to squad tracker not very many people are in the server when it happens.

FDF sound mod seems ok.

<S>

Knight31

[[kh]jman 49]

I wonder if we could get a comment from the developers. This is becoming a real problem. I'd be happy to send them the crash logs.

[SlipperyJim 0]

So many server crashes. I can't play longer than 2-4 hours without the server reseting.

BIS how come 1.14 is such a big step backwards?

[aussie dave 0]

So many server crashes. I can't play longer than 2-4 hours without the server reseting.

BIS how come 1.14 is such a big step backwards?

Its the player addons causing the problems.

like i said in another thread to you, play on servers that run BattlEye and signature check.

Our GamingSA.com PvP Only server has not crashed since i stopped people using addons(which was last Saturday). As we test more addons we will add their keys to our server.

[SlipperyJim 0]

So really what you are saying is that the 1.14 patch broke all the addons.

[cross 1]

There may be some changes in 1.14 that result in incompatibilities with existing addons/mods or scripts in missions. I guess each addon maker should check compatibility of their addons/mods and confirm that working or issue an update. Until then i guess we should not allow addons/mods at all.

[aussie dave 0]

So really what you are saying is that the 1.14 patch broke all the addons.

No

But i'm working though the signed addons only, so far i have allowed only these addons on to GamingSA.com PvP.

so far the list is:

ArmA Effects mod by Maddmatt

Extended Eventhandlers by Solus & Killswitch

AACFSounds v2.3

[maddogx 13]

I can definitely say that it's not a good idea to allow addons that add any kind of custom content that would be required on all clients (including the server) to be displayed.

Sound mods and other effect replacement mods are most likely all okay, because they only change local stuff - but as soon as one client uses a mod that uses a new kind of vehicle/weapon/animation/etc. it should not be allowed on a normal public server.

The SLX mod is a prime example of a mod that should not be allowed on a standard public server. At the very least, it adds new custom animations and a new class of vehicle used for dragging. Anyone connected to the server who is not running SLX mod will thus encounter problems if someone is using it, not to mention that the behaviour of the server will be unpredictable.

Judging from my experience in modding ArmA, the following mods should be okay:

- most likely all sound mods

- Maddmatts ArmA effects

- extended event handlers

- dynamic view distance

- Kegetys lowplants

- the stand-alone flares countermeasures (though they would add an unfair advantage unless everyone has them)

The only thing is, when you allow the key with which the addon in question is signed, you're also allowing all other addons from that author which are signed by the same key, so that could add another problem.