Linux Dedicated Server feedback

[mAiN131 10]

I have since the 1.18 update the problem that my server not showe any mods or extensions. In SSH i see it loading but ingame i cant see anything of this.

[patxitron 1]

@Fred41, Good idea. I will try it.

I hope I get it by next monday (maybe sooner if I can elude some social compromises :p)

Best regards

Edit:

I just changed glibc-2.19 with both my initial change and your suggestion (selected by a define) and it compiles OK.

As I said, I hope I can release it soon (I must upload this to the server, set it the environment and make some tests).

If anyone want to test it I've created a chroot environment by following the LFS (Linux From Scratch) book. and then I've chrooted into it and I've modified and recompiled the glibc. I've generated a patch with the changes:

diff -ur glibc-2.19/malloc/malloc.c glibc-2.19-tweak/malloc/malloc.c
--- glibc-2.19/malloc/malloc.c    2014-02-07 09:04:38.000000000 +0000
+++ glibc-2.19-tweak/malloc/malloc.c    2014-05-02 17:06:58.136999541 +0000
@@ -4650,6 +4650,7 @@
struct mallinfo
__libc_mallinfo ()
{
+#ifdef USE_GLIBC_DEFAULT_MALLINFO
  struct mallinfo m;
  mstate ar_ptr;

@@ -4667,7 +4668,34 @@
      ar_ptr = ar_ptr->next;
    }
  while (ar_ptr != &main_arena);
+#else
+# ifdef RETURN_ZEROED_MALLINFO
+  struct mallinfo m;
+  memset (&m, 0, sizeof (m));
+# else
+  static struct mallinfo m;
+  static size_t count;
+  if ((count++ % 1024) == 0)
+    {
+      mstate ar_ptr;

+      if (__malloc_initialized < 0)
+        ptmalloc_init ();
+
+      memset (&m, 0, sizeof (m));
+      ar_ptr = &main_arena;
+      do
+        {
+          (void) mutex_lock (&ar_ptr->mutex);
+          int_mallinfo (ar_ptr, &m);
+          (void) mutex_unlock (&ar_ptr->mutex);
+
+          ar_ptr = ar_ptr->next;
+        }
+      while (ar_ptr != &main_arena);
+    }
+# endif
+#endif
  return m;
}

Edited May 2, 2014 by patxitron

[fred41 42]

@patxitron, very interesting appoach at all.

Thats linux, nothing is impossible (i am currently reading the LFS book, awesome stuff :) )

[grubes 10]

We've finally got a headless client connecting to our A3 Linux dedicated server and handling AI for our group. Just did some stress testing tonight with 12 AI rifle squads on each side of a firefight. Server frames never dropped below 30 FPS according to #monitor. Gameplay was incredibly improved. While it's unfortunate that the Linux build essentially requires the HC for reasonable performance, we're confident it will catch up over time.

I'll be putting together a set up guide showing how we've integrated the HC into mission building some time in the near future.

GRUBES

[black Sh33p]

http://blacksh33p.net/

[patxitron 1]

Ok, I've got it.

I've tested a chroot environment on a virtual machine on a server with @a3mp and @task_force_radio addons with a mission on chernarus with about 30-40 AI and 8 human players and, according to #monitor, the server was consistently running at 40-50 fps with some (very rare) drops to 30-35 fps.

The same setup without the chrooted environment used to run at a maximum of 30-35 fps with very frequent drops to 3-4 fps.

In the chroot environment, the glibc mallinfo is patched to provide a cached mallinfo struct refreshed every 1024 calls as Fred41 suggested (thank you).

To use it, download the chroot environment from this link, unpack it in your home folder:

tar xvf lfs-chroot-with-modified-glibc.tar.xz

mount some filesystems:

mount -o bind /dev /home/johndoe/lfs-root/dev
mount -t proc none /home/johndoe/lfs-root/proc
mount -t sysfs none /home/johndoe/lfs-root/sys

chroot to the environment using its "games" user:

chroot /home/johndoe/lfs-root sudo -u games \
       HOME=/home/games \
       PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin \
       TERM=$TERM \
       PS1='\u:\w\$ ' \
       /bin/bash --login +h

and then you can download steamcmd and install the Arma 3 server (wget and nano are provided in the environment to download utils and edit configurations).

Alternativelly you can copy your entire Arma3 server directory tree into the home/games of the rootfs environment and then chroot into it and launch the server.

You can exit the environment with the "exit" command.

Note that all above steps must be done as root (or using sudo), but when entering the chroot environment you drop the privileges to a normal user in order to minimize security concerns.

The sources needed to build the chroot environment are available here. My glibc patch is this one.

Best regards.

---------- Post added at 11:37 ---------- Previous post was at 11:21 ----------

Thats linux, nothing is impossible (i am currently reading the LFS book, awesome stuff :) )

Indeed, the Linux From Scratch book is a hard and painful method to get linux running, but a very interesting and educative one.

Best regards

Edited May 4, 2014 by patxitron

[fred41 42]

@patxitron, wow, very smart °°.

Do you think, it would be possible, to make intels tbbmalloc available for arma server on linux, using the same chroot aproach?

(http://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html)

Greets,

Fred41

[patxitron 1]

I think it's doable. I'm light years away from being an expert to this low-level of glibc using but from the link you provides I guess that the hook functions only can be used by modifying the server source code (which is, obviously, out of our reach).

However it is possible to patch malloc and related functions in glibc with the tbb ones or make them call the tbb ones.

The first approach could trigger a dependency hell and ending in including a non-trivial part of the tbb into glibc itself. The second approach implies linking the glibc library with the tbb one which must be done in the autoconf/makefile system, which will be a non-trivial work also (at least for me because I have no experience with the autoconf stuff).

I think the little potential performance gain doesn't pay the effort. But I can probably be wrong and someone knows how to do it easyly. Thats the great advantage of open source.

Best regards.

[k0rd 3]

Hey guys - I met someone on the ticket system who is going about this a different way - instead of setting up the chrooted environment, he is making use of LD_PRELOAD to substitute costly functions. V. cool, ya'll should check it out.

http://feedback.arma3.com/view.php?id=18487#c70219

http://feedback.arma3.com/view.php?id=18556

[patxitron 1]

I've not even tryed the LD_PRELOAD trick since in http://feedback.arma3.com/view.php?id=18556 the reporter asserts that trying to preload intel-tbb, crashes the server and he must inject some asm to do a succesful preload. I thought that preloading fails at all, but it seems that a simple dummy mallinfo lib could do the job in a simpler and more elegant way without my chroot trick.

Best regards.

Edited May 5, 2014 by patxitron

[fred41 42]

... thanks k0rd, for sharing this info.

If i understand it correctly, LD_PRELOAD works basically with arma server, so that Nics solution fixes the mallinfo() problem very efficient.

But other bugs in arma server, are preventing a succesfull replacement of malloc functions via LD_PRELOAD, so that preloading tbbmalloc, for example, don't work.

Nics did a great job, by locating this heavy bugs and it should be a lot easier for BIS devs now, to fix it finally and permanently.

[nics 10]

"To be able to post links or images your post count must be 1 or greater. You currently have 0 posts." Are you serious?

---------- Post added at 11:39 ---------- Previous post was at 11:38 ----------

Sorry guys for not sharing my information, but I simly haven't been aware of this thread. Thanks to k0rd for pointing me here.

@patxitron: In one of my issues, I indeed asserted the assumption that LD_PRELOAD might not work at all. But this had only been an assumption and I've been proven wrong. Sorry for your extra work.

For those interested: Every shared obect which calls 'pthread_key_create()' either directly or indirectly from its '.init' will crash arma3server if LD_PRELOADed. The Intel TBB allocators call 'dlsym()' to get access to glibc's malloc and the dl-framework in turn calls 'pthread_key_create()'.

To give you some status:

My first ticket, http://feedback.arma3.com/view.php?id=18487, simply complains on that costly 'mallinfo()' beast getting called far too often. 'mallinfo()' as implemented by glibc walks all memory-chunks the application ever 'free()'d and thus, consumes a lot of CPU cycles in general.

Substituting 'mallinfo()' by a NOP, I observed a performance gain of up to 1000%, especially at mission loading and AI spawning time.

The other two tickets, http://feedback.arma3.com/view.php?id=18556 and http://feedback.arma3.com/view.php?id=18706, refer to bugs which stand in the way of LD_PRELOADing the Intel TBB allocators. The Intel TBB allocators, substituting a NOP-'mallinfo()' by themselves, provide an extra performance boost of approx. 10% or more over the NOP-'mallinfo()' substitution alone, at least in my very quick tests I did with 1.16. The first of the two tickets can be regarded as being solved "in principle" in the sense that it is only waiting for BIS to apply a fix. However, it got assigned today, so chances are good that this happens in a timely manner.

The second issue on the other hand is harder: While it is easy to reproduce, these kinds of bugs related to unitialized memory are hard to track down in general. I will do my best, but without any access to source code, it is even harder :P

What I already know is, that this bug is located somewhere in some parser code and it might happen that solving this could solve some other Heisenbugs which also occur if nothing is LD_PRELOADed at all. We'll see.

[fred41 42]

@nics, nice to meet you here and thanks for all your work, to make arma server running on linux.

Edited May 6, 2014 by Fred41

[nics 10]

If it is possible to LD_PRELOADing a dynamic library like dummy_mallinfo to intercept calls to mallinfo(),

do you think it could be possible too, to use the same technique to intercept calls to malloc()/free() and call frontend functions,

like scalable_malloc()/scalable_free() in tbbmalloc library directly (without the proxy functionality provided by intel).

Perhaps this way the dlsym() and the resulting pthread_key_create() could be avoided?

The answer is no, since it's not the proxy calling dlsym, but libtbbmalloc already. But good try ;)

Furthermore, calling dlsym() is not the only problem. We still have the unitialized memory usage...

Edited May 6, 2014 by nics

[k0rd 3]

... thanks k0rd, for sharing this info.

Thanks to k0rd for pointing me here.

All the thanks go to you guys and patxitron for sharing your research.

Edited May 6, 2014 by k0rd

[fred41 42]

@nics, ok and thanks for clarification.

(I just now found, that pthread_key_create() is the unix equivalent of TLSAlloc() in windows and this is indeed essential for tbbmalloc to work.)

[stevos758 10]

Not sure what any of this means but it looks like you guys are doing some fantastic work! Thanks!

[jonekone 1]

^^ yeah thx for tracking bugs so vigorously

[dr death jm 117]

this server is in testing now, "bootcamp" nothing is tweeked running 30/40 fps

Its my friends set up ,he lets me run servers from it.

he'll be futher testing and tweeking tomorrow ill post everything i can if all is good.

Intel Xeon W3680, 24GB RAM, Raid10

Datacentre 100mbit

[griffz 1]

seriously bohemia , wake up ! since december 2013, we dont have a usable A3 dedi server . not even usable.

my gaming computer is the only one running windows

but hosting dedi & client on the same box implies a severe performance drop down.

ps: i just tested virtualbox on my debian, with kastenbiere cti. it's working yes , but my antique Q6600 doesn't offer decent performance in this way.

edit: Any news DR death JR ?

Edited May 10, 2014 by griffz

[leroyjenkins 10]

Im currently trying to install A3 on Ubuntu 12.04 64. When I get to the part of installing via steamcmd nothing installs, and Im given "Error! App 233780 state is 0x1 after update job". I followed a tutorial, and it stated to make a new steam login. I tried that, then tired my steam account that has a3 on it and it made no difference. I looked further and seen that the appid was updated a few days ago. I then tried the newly listed 495460 in which I was given the "no subscription error" on both accounts.

Am I missing something here? I've messed with this for hours now, and google has been of no help.

Thanks for any input.

**Update** Was sorted out by a member, not entirely sure what the problem was but its solved.

Edited May 11, 2014 by leroyjenkins
Update

[nsh4rpy 10]

Hi,

Im having issues setting it up on Centos i keep getting a error "./arma3server

./arma3server: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by ./arma3server)

./arma3server: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /root/Arma3/libsteam_api.so)"

How do i go about fixing this?

[killswitch 19]

Unfortunately, the Arma 3 server is compiled against a newer version of the C++ runtime libraries than what CentOS has. Fixing this can be done by

1 - using a Linux distro with a newer libstdc++.so.6 (CentOS won't have this until RHEL 7 / CentOS 7. For example, recent Fedoras, Debian 7 and Ubuntu 12.04 have it)

2 - looking at post #67 in this thread.

3 - building a newer set of runtime libs for your CentOS and use that when starting the Arma 3 by way of LD_LIBRARY_PATH. This is not for the beginner.

4 - finding someone who has done #3 and has a new enough libstdc++.so.6 you can use. I am someone. Here: libstdcpp-for-arma3-centos6.tgz.

Unpack that archive into the folder where Arma3 is installed. Then set LD_LIBRARY_PATH accordingly. For example:

export LD_LIBRARY_PATH=/path/to/arma3/lib

PS. Don't run the A3 server as root.

Edited May 13, 2014 by Killswitch

[griffz 1]

the malloc fix seems to work . my crcti start quite fast, and server sustain 18 20fps. thanks guys.

[nsh4rpy 10]

Unfortunately, the Arma 3 server is compiled against a newer version of the C++ runtime libraries than what CentOS has. Fixing this can be done by

1 - using a Linux distro with a newer libstdc++.so.6 (CentOS won't have this until RHEL 7 / CentOS 7. For example, recent Fedoras, Debian 7 and Ubuntu 12.04 have it)

2 - looking at post #67 in this thread.

3 - building a newer set of runtime libs for your CentOS and use that when starting the Arma 3 by way of LD_LIBRARY_PATH. This is not for the beginner.

4 - finding someone who has done #3 and has a new enough libstdc++.so.6 you can use. I am someone. Here: libstdcpp-for-arma3-centos6.tgz.

Unpack that archive into the folder where Arma3 is installed. Then set LD_LIBRARY_PATH accordingly. For example:

export LD_LIBRARY_PATH=/path/to/arma3/lib

PS. Don't run the A3 server as root.

Got it past that but now im getting a new error:

./arma3server

AppId parsing successful. Using appId=107410

Cannot register unknown string STR_CONTROLS_TOOLTIPS_CURATOR_COLLAPSE_PARENT

Cannot register unknown string STR_A3_BUY_DLC

Cannot register unknown string STR_USRACT_CURATOR_COLLAPSE_PARENT

Updating base class ->Base, by a3\dubbing_radio_f\config.bin/CfgHQIdentities/PAPA_BEAR/

Updating base class ->Base, by a3\dubbing_radio_f\config.bin/CfgHQIdentities/AirBase/

Updating base class ->VScrollbar, by a3\ui_f\config.bin/RscDisplayDebug/Controls/G_Expression/VScrollbar/

Updating base class ->HScrollbar, by a3\ui_f\config.bin/RscDisplayDebug/Controls/G_Expression/HScrollbar/

Updating base class ->RscText, by a3\ui_f\config.bin/RscDisplayCapture/controls/TimeLines/

Updating base class RscShortcutButton->RscButtonMenu, by a3\ui_f\config.bin/RscDisplayCapture/controls/ButtonAverages/

Updating base class RscShortcutButton->RscButtonMenu, by a3\ui_f\config.bin/RscDisplayCapture/controls/ButtonSavePreviousData/

Updating base class RscShortcutButton->RscButtonMenu, by a3\ui_f\config.bin/RscDisplayCapture/controls/ButtonPreviousData/

Updating base class RscControlsGroup->RscControlsGroupNoScrollbars, by a3\ui_f\config.bin/RscDisplayMain/controls/ModIcons/

Updating base class RscPicture->RscPictureKeepAspect, by a3\ui_f\config.bin/RscDisplayMain/IconPicture/

Updating base class RscListBox->RscCombo, by a3\ui_f\config.bin/RscDisplayCustomizeController/Steepness/

Updating base class ->RscStandardDisplay, by a3\ui_f\config.bin/RscDisplayControlSchemes/

Updating base class ButtonOK->RscButtonMenuCancel, by a3\ui_f\config.bin/RscDisplayControlSchemes/controls/ButtonCancel/

Updating base class RscButton->RscButtonMenuOK, by a3\ui_f\config.bin/RscDisplayControlSchemes/controls/ButtonOK/

Updating base class RscPicture->RscPictureKeepAspect, by a3\ui_f\config.bin/RscDisplayFileSelectImage/controls/OverviewPicture/

Updating base class RscShortcutButton->RscButtonMenuCancel, by a3\ui_f\config.bin/RscDisplayFieldManual/controls/ButtonCancel/

Updating base class RscButton->RscButtonMenuCancel, by a3\ui_f\config.bin/RscDisplayPublishMission/controls/ButtonCancel/

Updating base class RscShortcutButton->RscButtonMenuOK, by a3\ui_f\config.bin/RscDisplayPublishMissionSelectTags/controls/ButtonOK/

Updating base class ButtonOK->RscButtonMenuCancel, by a3\ui_f\config.bin/RscDisplayPublishMissionSelectTags/controls/ButtonCancel/

Updating base class ->RscSubmenu, by a3\ui_f\config.bin/RscMainMenu/

Cannot update non class from class a3\ui_f\config.bin/RscCallSupport/Items/

Cannot update non class from class a3\ui_f\config.bin/RscRadio/Items/

Updating base class Optics_Commander_02->Optics_Armored, by a3\armor_f_epb\apc_tracked_03\config.bin/Optics_Commander_03/

Initializing Steam Manager

unable to load subscribed content list. list will be updated from steam

unable to load published content list. list will be updated from steam

unable to load cached items meta info. save and update functionality will be broken

Steam Manager initialized.

Unknown hit sound type in bin\config.bin/CfgSurfaces/GdtVRsurface01.soundHit

22:00:31 Dedicated host created.

PhysX3 SDK Init started ...

PhysX3 SDK Init ended.

String STR_A3_dlc_loading_screen_text not found

String STR_A3_dlc_loading_screen_text not found

String STR_A3_dlc_loading_screen_text not found

22:00:32 BattlEye Server: Initialized (v1.190)

22:00:32 Host identity created.

[s_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.

CreateBoundSocket: ::bind couldn't find an open port between 27016 and 27016

Arma 3 Console version 1.18 : port 2314

^CString STR_A3_dlc_loading_screen_text not found

../lib/Network/networkServer.cpp ClearNetServer:NOT IMPLEMENTED - briefing!

String id 2130 is not registered

22:00:46

[pyroarchy 10]

Q: where does the linux ds keep its rpt logs?

A: ~/.local/share/Arma 3

just for reference, as i hope it helps some

We talked on IRC some =] Its not making any RPT log files there or systemwide, is anyone else having this problem? I have -nologs removed from the launch options, and it was never in there to begin with :( Its a hassle to troubleshoot missions without it. Thanks in advance for your responses.