JavaScript for ARMA

This addon adds JavaScript scripting support for ARMA addons and missions. The JavaScript language is powered by the (very) speedy Google’s V8 Engine. V8 implements ECMAScript as specified in ECMA-262, 5th edition.

Version: 0.7/3.21.11 (Addon/V8) - DOWNLOAD

Addon is open source with code available under GPL 3.

Be aware of the following limitations:

• You cannot use and call SQF commands from within JavaScript context (I have an idea how to overcome this callExtension limitation and will implement this feature before 1.0 release).
  
• The return value of JS_fnc_exec has a hard buffer limit (max 10kb output) – a limitation of the callExtension interface. You may have issues returning large strings or large arrays from JavaScript. Technically, I think I know how to bypass this limitation and make it transparent, so this issue is temporary and will be resolved in the upcoming updates. At the moment if you hit the output buffer limit - SQF will throw a "[OOB]" exception so you can still handle the issue with SQF code.
  
• Linux dedicated server is not supported at the moment (callExtension has no support for Linux shared libraries).
  

Since (at the moment) you cannot directly call SQF commands from JavaScript – this addon can be used to offload complex algorithms and computations that require fast execution and do not depend on ARMA/SQF state. Also, you can use the following native JavaScript language features:

• Prototype-based OOP
  
• Regular Expressions
  
• Fast Math, Array and String operations
  
• Date and JSON support
  

SQF functions and macros (API)

The following SQF functions are provided by this addon:

• JS_fnc_exec - Execute JavaScript code and return the result.
  
• JS_fnc_spawn - Execute JavaScript code in parallel (non-blocking mode, separate thread).
  
• JS_fnc_terminate - Terminate (abort) a spawned JavaScript script.
  
• JS_fnc_done - Check if a spawned JavaScript script is done/finished.
  
• JS_fnc_version - Get addon and JavaScript engine version information.
  

Please refer to the examples below. Detailed function documentation is still a TODO item yet.

The "exec" and "spawn" versions differ in that the "exec" version runs the JavaScript code and returns the result to SQF in a blocking mode, while the "spawn" will return immediately (with a script handle) and will run JavaScript code in parallel (in a separate thread).

An optional \JS\API.hpp header file is provided by this addon and can be included in your SQF scripts. This header file defines JS(...) pre-processor macro, which is an alias to JS_fnc_exec function. Note that the JS(...) macro skips SQF call command and is theoretically even faster than calling JS_fnc_exec directly!

To include the header file and JS(...) macro in your SQF script file, do:

#include "\JS\API.hpp"

JavaScript functions

The following functions are available for JavaScript code:

• sleep(seconds) - Suspend current background script for a given number of seconds. Note, that an attempt to use this function within a blocking code (executed with JS_fnc_exec) will raise an exception (just like with SQF sleep command).
  

FAQ

1. How are JavaScript errors handled?

If your JavaScript code has compile errors and you do not handle runtime errors yourself using JavaScript exception mechanism – the JavaScript engine will throw an unhandled exception. Once the unhandled exception is caught by this addon – it will be forwarded to SQF using SQF exception handling mechanism. Therefore, you can catch JavaScript errors in SQF with:

try {
"JS code with some errors" call JS_fnc_exec;
}
catch {
hint _exception; // [line 1] SyntaxError: Unexpected Identifier: "JS code with some errors"
};

2. Can I execute a JavaScript file instead of a code string?

Yes, using the SQF loadFile command:

// Load and execute "script.js" file:
loadFile "script.js" call JS_fnc_exec;

// Or if you have "\JS\API.hpp" header file included:
JS(loadFile "script.js");

3. What data type is returned from JS_fnc_exec call?

The results returned from JS_fnc_exec call are transparently serialized to SQF data types using the following scheme:

• JavaScript Number => SQF "SCALAR" type
  
• JavaScript Boolean => SQF "BOOLEAN" type
  
• JavaScript Array => SQF "ARRAY" type (including nesting)
  
• JavaScript null or undefined => SQF "VOID" type (nil)
  
• JavaScript String => SQF "STRING" type
  
• JavaScript Object with .toString() implementation => SQF "STRING" type
  
• Anything else => SQF "VOID" type (nil)
  

4. What about Unicode (UTF-8) strings?

Both SQF and JavaScript are UTF-8 native. You can throw at JavaScript UTF-8 strings (and run string operations there) and get back valid UTF-8 results to SQF.

Although be aware that SQF escapes quotes in strings using double quotes ("") while JavaScript does that with escape character (\").

5. How to dynamically detect @JS addon availability?

You can use configFile check to test if the JavaScript addon is available:

_hasJS = isClass(configFile >> "CfgPatches" >> "JS");

if (_hasJS) then {
 // Offload complex computation to JavaScript
}
else {
 // Use SQF
};

6. Can this work with ARMA 2?

Yes, with ARMA 2: Operation Arrowhead 1.61+ patch.

Note that if you are not using @CBA addon - you may need to add "Functions" module (F7) to your mission for the JS_fnc_* functions to be loaded and available (only required for ARMA 2).

7. Is JavaScript multi-threaded by design?

It is not. JavaScript memory model does not support parallel programming and does not have any synchronization mechanisms built-in (neither does SQF, really). The background scripts and parallel execution is implemented with constant V8 isolate (virtual machine) locking and unlocking. The locking/unlocking happens when you call JavaScript sleep function - therefore it is important that you use sleep within code having any (infinite) loops in it (so that other JavaScript scripts can get a window to execute).

While I do not have internal knowledge of SQF implementation - I suspect the parallel execution is using something very similar. The difference is that the context switching in SQF happens every time you execute a command (and basically everything in SQF is a command).

Examples

For improved readability the examples below will use the JS(...) pre-processor macro. Otherwise, you can replace JS("code") with "code" call JS_fnc_exec for the same effect.

Simple JavaScript statements and math:

_result = JS("2 + (2 / 2)");
// _result == 3
// typeName _result == "SCALAR"

SQF and JavaScript can then be mixed together:

if (JS("false") || {JS("4 > 2")}) then {
// true, SQF lazy evaluation works just fine
};

Assign and access global JavaScript variables:

JS("test = 123");
_test = JS("test");
// _test == 123

Call a built-in JavaScript function:

_randomNumber = JS("Math.random()");
_year = JS("(new Date()).getFullYear()"); // "2013"

Call your custom function with arguments:

_result = JS(format ["myFunction(%1)", _number]);
_result = JS(format ["myFunction('%1')", _string]);

Regular expressions:

_email = "my@email.com";
_emailValid = JS(format["/\w+@[a-zA-Z_]+?\.[a-zA-Z]{2,3}/.test('%1')", _email]);
// _emailValid == true

Background scripts:

// Execute a "dummy" JavaScript background script:
_script = "sleep(100)" call JS_fnc_spawn;

_isDone = _script call JS_fnc_done; // _isDone == false
_script call JS_fnc_terminate;
_isDone = _script call JS_fnc_done; // _isDone == true or false

Note: JS_fnc_terminate does not block and does not wait for the background script to be terminated - it just sends a signal. Therefore the last line in the example above could either return true of false (depends on timing).

Get addon version information:

_version = call JS_fnc_version; // An array: ["0.7", "V8", "3.21.11"]

TODO

• Allow calling of SQF commands from JavaScript (using sqf.* namespace).
  
• Add mission init script support: init.js, initServer.js, initPlayerLocal.js and initPlayerServer.js.
  
• Implement the workaround around callExtension output buffer limitation.
  
• Detect when ARMA is paused (suspend background scripts and use v8::V8::IdleNotification() internally).
  
• Add a second parameter to JavaScript sleep() function (to allow background scripts to continue when ARMA is paused).
  
• Do some JavaScript vs SQF performance tests (with startLoadingScreen to disable SQF throttling).
  
• Double check everything and do a final review of the code.
  

Version History

0.7 (2013-09-07)

• Updated V8 to 3.21.11.
  
• Fixed: Serialize JavaScript NaN value as SQF nil.
  
• Fixed: Serialize JavaScript Infinity value as SQF scalar infinity.
  
• Added a total of 32 unit tests (use execVM "\JS\Tests\_Run.sqf" to run).
  
• Other internal tweaks and optimizations.
  

0.6 (2013-07-07)

• Updated V8 to 3.20.2.
  
• Implemented background script handles (returned by JS_fnc_spawn).
  
• Implemented JS_fnc_done and JS_fnc_terminate functions.
  
• Added sleep(seconds) JavaScript function.
  
• Added a common header file (#include "\JS\API.hpp") with JS(...) macro.
  
• PBO and extension DLL files are now signed with bikeys (server-side key is included).
  
• Other internal tweaks and optimizations.
  

0.5 (2013-06-07)

• Initial release.
  

Nice release mate ! I'm sure this will be useful !

Very nice! Took it for a brief spin and it worked very well.

I'm not familiar with the callExtension interface, but would it be possible to call external services (eg. http) from within the javascript code?

A nice addition would be some sort of JSON support in sqf, but not really sure if that's possible.

Anyway, nice work!

I'm not familiar with the callExtension interface, but would it be possible to call external services (eg. http) from within the javascript code?

The core language does not have HTTP support, but that kind of API can be exposed to JavaScript as a library. Basically, I can take any C/C++ library and abstract it in a nice JavaScript API that then can be accessed using SQF. For example, to get proper HTTP support - the libcurl (a bit overkill) can be wrapped around JavaScript API.

Although I do not have immediate plans for that kind of functionality at least until 1.0 is ready. I also wonder why BIS never tried to do that natively in SQF? I take it there are some obvious security issues with this?

A nice addition would be some sort of JSON support in sqf, but not really sure if that's possible.

I think native JSON support is a bit tricky. JSON has both arrays and objects. SQF doesn't have a way how to represent JSON objects. But if your JSON is all made of arrays with numbers/booleans/strings, you can already use the following code to load JSON with this addon:

my.json file content:

[1, 2, false, true, "huh"]

Code to load my.json to SQF:

// Load JSON file in SQF
_json = format ["JSON.parse('%1')", loadFile("my.json")] call JS_fnc_exec;

// _json is now SQF array = [1, 2, false , true, "huh"]

Strings with quotes may need to be escaped in this example, I am yet to write a proper FAQ entry on SQF/JavaScript string escaping.

Looks great - will see if we can give it a test run for you! Well done!

[*]Implement the workaround around callExtension output buffer limitation.

So this is what you've been working on :) The output buffer is something like 16K is it not enough?

So this is what you've been working on :) The output buffer is something like 16K is it not enough?

In Arma 3 (at least in the DEV build) - it's 10kb.

In Arma 2 - it's 16kb (according to the beta patch logs, although I have not tested myself, it may have been changed)

I think for 99% of the cases you will never going to hit that buffer limit. But then there is that 1% - where things can get messy (right now SQF exception will be thrown). I want to eliminate this 1%: my solution is to do recursive callExtension calls with string/array concatenation until entire output buffer is returned. The trick is to make that buffer thread-safe (since you can run parallel JavaScript code with this addon) :)

In Arma 3 (at least in the DEV build) - it's 10kb.

In Arma 2 - it's 16kb (according to the beta patch logs, although I have not tested myself, it may have been changed)

I think for 99% of the cases you will never going to hit that buffer limit. But then there is that 1% - where things can get messy (right now SQF exception will be thrown). I want to eliminate this 1%: my solution is to do recursive callExtension calls with string/array concatenation until entire output buffer is returned. The trick is to make that buffer thread-safe (since you can run parallel JavaScript code with this addon) :)

Yeah, just checked in Arma 3 it is 10240 at the moment.

I understand what you are saying, but is it worth it? You will have to double the calls for 99% of cases just so that you cover 1% of cases. Personally I put it down as limitation and wouldn't sacrifice the speed. If you however make a DB app then it is different matter.

---------- Post added at 10:42 ---------- Previous post was at 10:34 ----------

BTW in Arma 2 it is also 10240

I understand what you are saying, but is it worth it? You will have to double the calls for 99% of cases just so that you cover 1% of cases. Personally I put it down as limitation and wouldn't sacrifice the speed.

The actual result (output string) and the available output buffer size are both available in C++ land - so there will be no overhead. It's a simple "if" statement comparing the output length to buffer size (probably a couple of CPU cycles). If the output can fit the buffer - just strcpy_s() the thing and be done with it, if not - I will use the advanced trick described above :)

BTW in Arma 2 it is also 10240

Thanks! If you have the time could you run a simple hint str("2+2" call JS_fnc_exec) test just to confirm the addon is working on Arma 2? (I don't have Arma 2 installed myself at the moment).

If you have the time could you run a simple hint str("2+2" call JS_fnc_exec) test just to confirm the addon is working on Arma 2? (I don't have Arma 2 installed myself at the moment).

yeah, it works.

This is great, been waiting for something like this for years. Any tips on getting C++ libraries implemented into the source and making it play nice with Java?

Good work on the release, but I'm curious about this statement "The JavaScript version is running at least 15 times faster (on my machine) compared to the SQF function (even with all the extra overhead of format command and the callExtension interface/protocol)." What code did you use to compare and how did you measure that?

Good work on the release, but I'm curious about this statement "The JavaScript version is running at least 15 times faster (on my machine) compared to the SQF function (even with all the extra overhead of format command and the callExtension interface/protocol)." What code did you use to compare and how did you measure that?

I used a simple SQF "for" loop with 1000 iterations measuring the time spent with "diag_tickTime" command.

This is rather a very raw performance test that does not represent true V8 benefits.

Also, what I am not entirely sure is if the SQF is still throttled even on the dedicated server? In any case - I think the dedicated server should be used for perf testing as there will be no rendering related overhead.

Release frontpaged on the Armaholic homepage.

JavaScript [ALPHA] v0.5/3.19.8.1

Do you think the release of official JAVA support for scripting could deprecate this addon?

I would suggest adding debug support as a todo. Perhaps even add the calling SQF function/line as content in the debug message. I don't have experience with v8 but I would first look into a implementation of a handler for v8::Debug::SetDebugMessageDispatchHandler() to do this.

Also, what I am not entirely sure is if the SQF is still throttled even on the dedicated server? In any case - I think the dedicated server should be used for perf testing as there will be no rendering related overhead.

It is. With that in mind, I'd be interested in some benchmarking done within startLoadingScreen which does disable script interruption as well as rendering.

[*]JavaScript Object with .toString() implementation => SQF "STRING" type

After playing around with this shortly, I don't think this is correct behavior. The framework should instead return a reference(assigning some sort of global named reference would probably be easiest, though I don't know if that would be best) that can be reused in subsequent calls to JS_fnc_exec in a manner similar to many other REPL implementations do for returned values.

EDIT: Of course this raises the question on how one would manage reference to primatives as well, a how to manage a system for call by value/call by reference(EDIT: Sorry calling by reference/value is a non issue as that is evaluated by v8, instead a system would be needed for returning references or values).

I would suggest adding debug support as a todo. Perhaps even add the calling SQF function/line as content in the debug message. I don't have experience with v8 but I would first look into a implementation of a handler for v8::Debug::SetDebugMessageDispatchHandler() to do this.

I am probably going to investigate better V8 debugging support in the future. In the mean time, I am working on a simple JavaScript log() function that will print to ARMA RPT file directly.

It is. With that in mind, I'd be interested in some benchmarking done within startLoadingScreen which does disable script interruption as well as rendering.

Nice, didn't know that. Will run some tests later.

After playing around with this shortly, I don't think this is correct behavior. The framework should instead return a reference(assigning some sort of global named reference would probably be easiest, though I don't know if that would be best) that can be reused in subsequent calls to JS_fnc_exec in a manner similar to many other REPL implementations do for returned values.

The toString() is actually part of the language. It's not used to serialize objects - but to return a string representation of that object (a good example is a Date object).

As for your suggestion - is there actually a good reason to return JavaScript objects to SQF land (by some sort of a reference)? Technically it would be possible to make a persistent V8 handle on the returning JavaScript object, hash it in some sort of a number representation and return that object as a specially encoded SQF string. But since you can't really do anything with this in SQF (can't transmit over the network, etc) - that would be pretty much useless. If you truly need to get a JavaScript object out of JS land into SQF - I would suggest getting a JSON representation of that object. Then you can transfer this JSON string over the network and even re-create it on another machine.

The toString() is actually part of the language. It's not used to serialize objects - but to return a string representation of that object (a good example is a Date object).

Serialization would probably better than toString() however I still don't think it is correct. When passing values into a REPL assignments should be transitive, passing a value returned from the REPL should result in the same value. The following sequence should be valid regardless of object's type(instead of an object it could be String, or a Long, or an Array) when calling by value, or by reference(ignoring issues of precision).

_expr1 = "new someObject" call JS_fnc_exec;
_expr2 = "someObject" call JS_fnc_exec;
//This should return true
(expr1 + " === " + expr2) call JS_fnc_exec;

As for your suggestion - is there actually a good reason to return JavaScript objects to SQF land (by some sort of a reference)? Technically it would be possible to make a persistent V8 handle on the returning JavaScript object, hash it in some sort of a number representation and return that object as a specially encoded SQF string.

Yes a named reference is exactly what I was refering to:

_value = "new someObject" call JS_fnc_exec" //Should return "expr1"(or some other token of your choosing) as a value
("_value+".toString()") call JS_fnc_exec //Should invoke toSting() on expr1

But since you can't really do anything with this in SQF (can't transmit over the network, etc) - that would be pretty much useless.

I don't understand how that makes references useless at all. references don't work across different VMs or networks in any other language without some sort of framework to serialize objects and send them for you in any other language, even SQF.

If you truly need to get a JavaScript object out of JS land into SQF - I would suggest getting a JSON representation of that object. Then you can transfer this JSON string over the network and even re-create it on another machine.

Serializing still wouldn't be correct, in the example above a new object would be constructed every time I pass a the value back to repl, instead of reusing the same object via a reference.

(snip)

Milyardo, I get what you are saying. A transparent way of passing in object references back and forth without serialization would be great - but is simply not doable right now in an acceptable way. The issue is with the following code:

_obj = "new Something" call JS_fnc_exec;
_obj = nil;

That would leak the JavaScript object as I am unable to detect when SQF reference has died. A nice solution would be for BIS to introduce new "resource" type for the callExtension to return/create. And somehow be notified when the resources are no longer referenced in SQF. Although I very much doubt this will be implemented any time soon.

The only workaround right now is to use serialization:

a) Implement toString() to serialize your object (hmm, or should I instead add support for something like toSQF() to not clash with the default JS toString() behavour?)

b) Provide some sort of unserialization method to restore your object in JavaScript land.

_obj = "new Something" call JS_fnc_exec;
_obj = nil;

That would leak the JavaScript object as I am unable to detect when SQF reference has died. A nice solution would be for BIS to introduce new "resource" type for the callExtension to return/create. And somehow be notified when the resources are no longer referenced in SQF.

That not a leak even in the loosest definition of word. There is still a reference to the JavaScript object as a global, which can easily be evaluated through the list of global variables. SQF can't leak those values because SQF doesn't manage those references. That said you will have a ever growing list of evaluated expressions as global variables, this is a common in REPL environments and they usually come with some sort of command to reset the environment or session(in addition to evaluated expressions many other environm

That not a leak even is the loosest definition of word. There is still a reference to the JavaScript object as a global, which can easily be evaluated through the list of global variables.

"new Something" does not create a global. There are no active references to that object which will lead to this value being garbage collected by default. To hold onto this object (so the reference can be returned to SQF) I would need to grab and hold a persistent V8 handle for this object. The issue is - when do I release this handle? I cannot detect when local/global variables are unset in SQF - which means this object will never get garbage collected, which means a leak.

"new Something" does not create a global. There are no active references to that object which will lead to this value being garbage collected by default. To hold onto this object (so the reference can be returned to SQF) I would need to grab and hold a persistent V8 handle for this object. The issue is - when do I release this handle? I cannot detect when local/global variables are unset in SQF - which means this object will never get garbage collected, which means a leak.

You should reread my first post on this subject, your framework should create the the global reference, and return the name or identifier of that reference as in the second example I posted.

You should reread my first post on this subject, your framework should create the the global reference, and return the name or identifier of that reference as in the second example I posted.

Your global reference is a more uglier method of doing what I described with a persistent V8 handle. It would still leak the object. Pseudo-code illustrating object return by reference:

_something = "something_id_ref = new Something; 'something_id_ref'" call JS_fnc_exec;
_something = nil;

When and who is going to release the global "something_id_ref" in JavaScript land?