Node.js Extension for Arma 3 (sock.sqf, sock.dll, sock-rpc)

[micovery 2]

Hey All,

I am the developer of the jni.dll extension, that brings support for the JVM in Arma 3. Check it out here if you have not already done so.

While working on that extension, I felt the pain of the slow development cycle ... having to compile the Java code, rebuild jar files, restart the game, etc.

I wanted to develop with something less "enterprisy", with faster turn-around, and more in-tune with the online community of developers.

And so, I decided to bring Node.js in into the picture ... what is Node.js you ask?

Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network

applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and

efficient, perfect for data-intensive real-time applications that run across distributed devices.

http://nodejs.org/

If you simply want to use JavaScript with your missions, then you should use Sima's JavaScript for Arma (@JS addon).

With Node.js, on the other hand, you get the benefit of an extensive (and growing) set of community provided packages.

The packages are available through the Node Package Manager (see https://npmjs.org). In fact, one of the main components of this release, is implemented as a node package itself.

How it works

---

It's just a simple client-server model between SQF side (client), and the Node.js side (server).

On the SQF side, there is a library called sock.sqf

On the Node.js side, there is a node package (sock-rpc), that allows you to setup an RPC (remote procedure call) server.

The glue between these two, is a C/C++ extension called sock.dll.

The extension transparently manages a TCP/IP socket connection to the remote server.

It acts a middle man relaying the requests originating from the SQF code, and serving back the response of the "remote" side.

Now, when I say "remote", I mean outside of the game. The extension is not intended to communicate with a truly remote (in another machine) Node.js server. If that's the case, then the network overhead can become unacceptable. As long as your "remote" side sits on the loop-back address (::1 or 127.0.0.1), then the communication never goes out of the TCP/IP stack.

Prerequisites

---

1. Visual C++ Runtime (http://www.microsoft.com/en-us/download/details.aspx?id=26999)

2. Node.js (http://nodejs.org/download/)

Components

---

1. sock.dll - This is a C/C++ extension that allows communication between SQF (using SOCK-SQF protocol), and a "remote" server (using SOCK-TCP protocol).

2. sock-rpc - This a Node.js module that implements the RPC server side, on top of the SOCK-TCP protocol

3. sock.sqf - This is small library that implements the RPC client side, on top of SOCK-SQF protocol

What are those protocols: SOCK-SQF, and SOCK-TCP? ... you may be asking yourself.

I made them up :), to describe the communication between the components.

If you want to read more on them, jump to the documentation: SOCK-SQF Protocol, SOCK-TCP Protocol.

Extension setup

---

Place the sock.dll file in the Arma 3 directory (C:\Program Files (x86)\Steam\steamapps\common\Arma 3\)

The extension reads configuration parameters from the Arma 3 process command line arguments.

There are two required command line arguments:

1. -sock_host

You must set this to the host name, or IP address of the "remote" server side (e.g. ::1)

2. -sock_port

This is the port number of the "remote" server side.

Server side model

---

This can be any Node.js application using the "sock-rpc" module, to expose methods that can be invoked from SQF.

The following example shows a simple RPC server with a "getDate" method, and an "echo" method.

"use strict";

var rpc = require('sock-rpc');

/**
*  Echo back all arguments passed.
*  echo(...,callback);
*/
rpc.register("echo", function () {
 var args = Array.prototype.slice.call(arguments, 0);
 var callback = args.pop();
 callback(null, args);
});

/**
*  Get date (no arguments)
*/
rpc.register("getDate", function (callback) {
 callback(null, new Date().toString());
});

rpc.listen("::1", 1337);

Note that a callback mechanism is used for sending responses back to the SQF side. The "callback" function is always passed as the last argument. The signature is: callback(err, result).

It is important that the methods being executed always invoke the callback, otherwise the client side may wait indefinitely for a response.

A future improvement for the RPC server will be to include a time-out mechanism. For now, make sure to always invoke the callback. (use try-catch blocks, as needed).

Client side model

---

From the SQF code, you can use the function "sock_rpc", to send requests over to the "remote" Node.js side.

    
  //Example for using the sock_rpc function

   private["_method", "_response", "_params"];  
   _method = "getLocalTime";
   _params = ["PST"];

   _response = [_method, _params] call sock_rpc;

   if (isNil "_response") exitWith {
     player globalChat format["Error occurred while invoking the %1", _method];
   }
   else {
     player globalChat _response;
   };

Data mapping

---

Underneath it all, the sock.sqf library takes care of serializing your request into proper JSON that can be evaluated on the "remote" Node.js side.

Also, on the way back, the sock-rpc Node.js module takes care of serializing the responses into SQF that can be compiled by the game engine.

For most cases, SQF and JSON can be mapped back and forth pretty nicely (except when mapping objects).

Here is the full details of the data mapping logic:

//SQF to JSON mapping

* SQF array, is mapped directly to JSON array
* SQF number, is mapped directly to JSON number
* SQF string, is mapped directly to JSON string (double-quotes escaped, i.e. "\"")
* SQF nil, is mapped to JSON null
* SQF objNull, is mapped to JSON null
* SQF objects, are mapped to JSON object with the following fields: {"netId": "0:1", "name": "player1"}

//JSON to SQF mapping

* JSON array, is mapped directly to SQF array
* JSON number, is mapped directly to SQF number
* JSON string, is mapped directly to SQF string (double-quotes escaped, i.e. """")
* JSON undefined, is mapped to SQF nil
* JSON null, is mapped to SQF nil
* JSON objects, are mapped to SQF code-block with the following structure: {[["key1", "value1"], ["key2", "value2"]]}

Source Code

---

Source is available in the following repositories:

1. Git repo for the sock.dll extension

https://bitbucket.org/micovery/sock.dll/

2. Git repo for the Noe.js sock-rpc server module

https://bitbucket.org/micovery/sock-rpc

3. Git repo for the sock.sqf RPC client library

https://bitbucket.org/micovery/sock-rpc.mission

Demo Node.js Extension & Mission

---

I have put together a mission to demo the setup of the Node.js sock-rpc server, and the sock.dll extension.

You can read the full instructions for the setup, in the git repository for sock.sqf library itself.

or, watch this video:

Additional documentation

---

The README files on the git repositories for each component contain:

1. Full API documentation

2. Protocol information,

3. How to enable debug, and logging levels

Summary/Reflection

---

Keep in mind that all the code is fairly new, and there will be bugs ... please report them, and I will try fix it on a timely manner.

Or better yet, you can fix them and contribute back.

What would you want to do with it? Let me know in this thread ...

All of the code is under MIT license, enjoy.

Cheers,

micovery

Edited September 29, 2014 by micovery

[dr death jm 117]

nice? will this work with linux?

[micovery 2]

nice? will this work with linux?

Not as it is right now. The C extension has some Windows specific code for benchmarking, error handling, and socket initialization.

The rest is mostly just posix, so it should be straight-forward to port it for building an *.so.

UPDATE/EDIT:

Created an enhancement ticket over at the repository.

If you are interested, I can work on it, but will need an environment to test the extension.

Or at least some assistance on how to setup the game server in Linux.

https://bitbucket.org/micovery/sock.dll/issue/1/add-support-for-linux

Edited May 28, 2014 by micovery

[AJCStriker 10]

First off I just want to say that I think this is fantastic!

However, I wanted to ask whether the system takes care of oversized returns or if we would have to implement that ourselves?

[micovery 2]

First off I just want to say that I think this is fantastic!

However, I wanted to ask whether the system takes care of oversized returns or if we would have to implement that ourselves?

Thanks! The sock.dll extension, and the sock.sqf library manage that transparently for you ... I have documented the internals at https://bitbucket.org/micovery/sock.dll#markdown-header-sock-sqf-protocol ... You can pretty much retrieve any size response (or rather, however long is the maximum you can fit inside an SQF string)

Edited May 30, 2014 by micovery

[Dahlgren 19]

Really cool extension, could be used to create some nice stuff :)

I'm also interested in Linux server support later down the road once BIS fixes the fps issues.

[cuel 25]

FPS is fixed now :)

This is really cool! Sadly our server is running linux but I'll check this out.

[defk0n_NL 2]

i think thers a bug with javascript arrays being converted back to sqf arrays.

as in: it doesnt work.

never mind, it works beautifully. if this would run on linux you could use any Database abstraction possible without Arma2Net. like MySQL, MongoDB, Redis etc

10/10

Edited September 9, 2014 by defk0n_NL

[micovery 2]

never mind, it works beautifully. if this would run on linux you could use any Database abstraction possible without Arma2Net. like MySQL, MongoDB, Redis etc

10/10

Great that you got it working!

I've never setup Arma 3 server in a linux box, but I at home programming in linux, and sticking to POSIX.

I'll give these instructions a try this weekend: https://community.bistudio.com/wiki/Arma_3_Dedicated_Server#Instructions_.28Linux_o.2Fs.29

If it works well, then you can have a sock.so lib soon :)

Edited September 12, 2014 by micovery

[micovery 2]

I have ported the library to Linux.

If you want to build it yourself, on Ubuntu 14.04 (64bit):

sudo apt-get update
sudo apt-get install git build-essential g++-multilib
git clone [url]https://micovery@bitbucket.org/micovery/sock.dll.git[/url]
cd sock.dll
make

If you want to use the pre-buit binary, on Ubuntu 14.04 (64bit):

sudo apt-get install lib32stdc++6 libc6-i386 lib32gcc1
wget https://bitbucket.org/micovery/sock.dll/raw/v0.0.2/Release/sock.so

It's nearly the exact same code-base as the windows version. (just added a few #ifdef __linux here and there).

I did a bit of testing on the Linux version to make sure it actually worked, but that was pretty much it.

Enjoy

[micovery 2]

New versions of both the sock-rpc Node.js module, and the Arma 3 sample mission are now available. (v1.0.0, and v.0.0.2 respectively)

Server side changes

---

On the server side, the callback mechanism for the JavaScript RPC functions has changed. The callback function is now passed as the last argument.

If you are expecting variable arguments, in your RPC function, you must pop the last argument to get the callback. e.g.

rpc.register("echo", function () { 
 var args = Array.prototype.slice.call(arguments, 0); 
 var callback = args.pop(); 
 callback(null, args); 
}); 

Also, notice when invoking the callback, the signature is now:

  callback(err, result) 

This is more in-line with the callback standards used throughout Node.js.

Client side changes

---

On the client side, I have improved the sock.sqf library to allow calling the sock_rpc function from client-side SQF as well.

Behind the scenes, it uses publicVariableServer, and publicVariableClient to pass the request, and response around.

The one caveat is that when invoking the sock_rpc function client-side SQF, you have to do it within a scheduled environment, as it uses SQF sleep to wait for the server's response.

e.g.

 [] spawn {
     private["_response"];
     _response = ["echo", ["arg1", "arg2"], false] call sock_rpc; 
     diag_log _response;
 };

If you are invoking the sock_rpc function on the server side, the "scheduled environment" restriction does not apply.

Edited September 30, 2014 by micovery

[AJCStriker 10]

I don't know if you still maintain this, but I was wondering if you could explain how to access keys in your keyvalue map datastructure on the sqf side? I am a little confused as to how it is supposed to work.

[Fusselwurm 13]

I'm currently in the process of building my own mission replay thingy, consisting of a NodeJS server, Redis, and a web client using T10's tiled maps with the Google Maps interface.

If anybody is interested, see here: https://github.com/gruppe-adler/ar3play-server

So, micovery: Thanks for this great extension! :)

Edited March 15, 2015 by Fusselwurm

[alexcroox 29]

Why would using a remote (external server) be unacceptable in terms of latency? Shouldn't it all be happening asynchronously over TCP? I'm looking to deploy this myself but I'd rather use a truely remote nodejs server to act as a middle man so random visitors to my site aren't making direct connections to my game server.

[stice 3]

Hey, first of all: Thanks for this awesome library.

 

I encountered a problem with JSON parsing where echoing output != input. 

 

The problem occurs with this function parameter: 

[["ItemMap","ItemCompass","tf_microdagr","ItemRadio","NVGoggles","H_HelmetB"],"arifle_MX_ACO_pointer_F",["","acc_pointer_IR","optic_Aco",""],"hgun_P07_F",["","","",""],"",["","","",""],"U_B_CombatUniform_mcam",["ACE_EarPlugs","ACE_fieldDressing","ACE_fieldDressing","ACE_morphine","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag"],"V_PlateCarrier1_rgr",["30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","16Rnd_9x21_Mag","16Rnd_9x21_Mag","SmokeShell","SmokeShellGreen","Chemlight_green","Chemlight_green","HandGrenade","HandGrenade"],"",[],[["30Rnd_65x39_caseless_mag"],["16Rnd_9x21_Mag"],[],[]],"arifle_MX_ACO_pointer_F","Single"]

After sending this array to the socket i get following error:

[SEVERE] [handler (rpc.js:90:14)] buf = RAW: (a lot of raw code, with two <null> in it. If you want the whole part, tell me)
[SEVERE] [ex2buf (sock.js:161:12)] RAW SQF request was not parsable as JSONUnexpected token <RAW SQF request was not parsable as JSONUnexpected token <
SyntaxError: RAW SQF request was not parsable as JSONUnexpected token <
    at Object.parse (native)
    at Object.exports.rawToJSON (\node_modules\sock-rpc\lib\sqf.js:223:19)
    at handler (\node_modules\sock-rpc\lib\rpc.js:83:23)
    at \node_modules\sock-rpc\lib\sock.js:205:7
    at data_machine (\node_modules\sock-rpc\lib\sock.js:102:9)
    at data_machine (\node_modules\sock-rpc\lib\sock.js:84:16)
    at Socket.<anonymous> (\node_modules\sock-rpc\lib\sock.js:132:17)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:163:16)
    at Socket.Readable.push (_stream_readable.js:126:10)

This can be solved by adding <null> to the replace.regex in sqf.js (text = text.replace(/(?:\{nil\}|nil|any|nothing|anything|<null>)/gi, "null") ;)

 

Now the JSON.parse works, but when i simply echo the array its different to what i sent in the first place.

 

Echoed from socket:

[["ItemMap","ItemCompass","tf_microdagr","ItemRadio","NVGoggles","H_HelmetB"],"hgun_P07_F",["","acc_pointer_IR","optic_Aco",""],"U_B_CombatUniform_mcam",["","","",""],"",["","","",""],"Single",["ACE_EarPlugs","ACE_fieldDressing","ACE_fieldDressing","ACE_morphine","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag"],any,["30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","30Rnd_65x39_caseless_mag","16Rnd_9x21_Mag","16Rnd_9x21_Mag","SmokeShell","SmokeShellGreen","Chemlight_green","Chemlight_green","HandGrenade","HandGrenade"],any,[],[["30Rnd_65x39_caseless_mag"],["16Rnd_9x21_Mag"],[],[]]]

Is this a known problem and is there a way to fix this?

 

The used save/get loadout library is https://forums.bistudio.com/topic/139848-getset-loadout-saves-and-loads-pretty-much-everything/, which uses empty arrays/entries to keep the correct order. Maybe those are causing the problem?

 

Thanks in advance

 

EDIT: Tested a little bit, following array causes the same <null>

[["a"],"a",["a"]]

Raw socket version:

[{},[[[[109,101,116,104,111,100],{}],[[108,111,97,100,76,111,97,100,111,117,116],{}]],[[[112,97,114,97,109,115],{}],[[[[97],{}]],<null>,[[[97],{}]]]]]]

EDIT 2:

Quick workaround: [str(_loadout)]

[TheLexoPlexx 0]

Hey there,
I just came across this amazing library and before I wanted to use it I was wondering if you are planning to update it for 64-bit.

 

Thanks in advance.

[Matthew10_28 4]

I was referred to this from here:

I don't have as strong of a development background as I need to understand this and run with it.  Would someone be willing to walk me through how I might be able to use this for my purposes?

[By-Jokese 3]

Could this extension work on rented servers like Nitrado, where you only have access to Arma 3 folder?

 

EDIT: Cant work as only has access to mod parameter, and cant add any other parameter for launch