SQF feature requests

[Foxyy 16]

sync <code>
 
SQF makes it very easy to achieve asynchrony, while at the same time offering no means of synchronization. Some people seem to think that race conditions aren't a thing in SQF. This is of course wrong, as the scheduler is free to pause execution almost anywhere.

 

A simple example:

if (player hasWeapon _someWeapon) then
{
    player removeWeapon _someWeapon;
    player addWeapon _someOtherWeapon;
};

Here it is entirely possible (though admittedly unlikely) for the scheduler to pause between the condition and the effect, and consequently for the player to drop the weapon, keeping it, while also receiving the new one. I have personally witnessed a similar situation occurring in a real game under very heavy lag.

Using certain hacky solutions it is possible to get around these issues. Relating to the previous example, it is for example possible to create an atomic function which at the same time attempts to remove the weapon, and returns a boolean indicating whether it was. It is also of course possible to create simple mutexes and consequently spinlocks. Neither solution is very pretty however. As such i suggest the following, very simple, script command:

sync { /* some critical section */ }

This command should execute the right hand argument synchronously, without the possibility of interruption by the scheduler. Use of any scheduler-only commands such as sleep and waitUntil should generate errors, just as they do in the unscheduled environment. If used in the unscheduled environment the command would have no special effect of course, behaving instead similarly to call.



[<string>,...] preprocessFileLineNumbers <string>

Many mod developers make clever use of macros for conditional compilation, for instance in order to enable or disable debugging features such as logging, argument validation, call stack tracing, et cetera. This is often done using macro constants defined either in each code file, or in a central header file. Unfortunately it doesn't seem to be possible to include files from the mission in an addon, as it is vice versa. This means in order to enable or disable the aforementioned features, one has to rebuild the mod, or a part of it while defining different macro constants. To ease this, and undoubtedly many other tasks, i suggest the extension of the preprocessFileLineNumbers command with a variant accepting, as the left hand argument, an array of strings representing macro constants to be defined during preprocessing.

 

For example:

["MY_MACRO", "DO_THE_DEBUG"] preprocessFileLineNumbers "myScript.sqf"

Would be equivalent to defining these macros at the top of the myScript.sqf file, like so:

#define MY_MACRO
#define DO_THE_DEBUG

// ...

compileFinal <code>

 
We can all agree that the compileFinal command, as well as the underlying concept, are great additions to SQF. However they seem to necessitate placing each function, no matter how small, into its own code file. I would often prefer to place related functions, such as class member functions, next to each other in the same code file.

 

For instance:

//map.sqf
 
my_fnc_map_add =
{
    // add key value pair to the map
};
 
my_fnc_map_get =
{
    // retrieve value from the map using the specified key
};
 
my_fnc_map_remove =
{
    // remove the specified key from the map
};
 
my_fnc_map_hasKey =
{
    // determine whether the map contains the specified key
};

Now all of these functions end up being non-final, and editable. For a dirty workaround something like this might be used:

my_fnc = compileFinal str {...}

However extending the compileFinal command with a variant accepting code as the right hand argument seems like it would be very simple to implement.



serializeObject <object>   /   deserializeObject [...]
 
Persistence across sessions is nothing new. Implementing it however ranges from arduous to downright impossible (at least without hacking memory via extensions), depending on the desired fidelity. Correctly saving and loading the attachments on a weapon inside a backpack inside a vehicle for instance isn't exactly simple. What i'm suggesting is two new simple commands to streamline this process. The proposed serializeObject command, given a right hand argument of type object, might return an array containing enough information to recreate the object with high precision. Properties such as type, position, orientation, velocity, damage to each part, and the entire inventory tree should be included. The counterpart, deserializeObject, could then be used to bring this array representation back to life. This would simplify immensely the task of persistence seen in many missions and mods, as well as offer a performance boost by moving a significant portion of SQF code over to a native implementation.

 

 

 

Please share your thoughts, or any other suggestions you might have.

If any (or all) of these have been suggested before, i apologize.

[Defunkt 431]

sync <code>

 

Not suggesting it's not true, more just concerned there's a fundamental problem with SQF I don't understand, but how does your example result in the behaviour described if the scheduler resumes at the statement following the last one executed?

[Foxyy 16]

Not suggesting it's not true, more just concerned there's a fundamental problem with SQF I don't understand, but how does your example result in the behaviour described if the scheduler resumes at the statement following the last one executed?

 

You're correct in that the scheduler will only start executing another thread if the first cedes control (sleep, waitUntil). However both unscheduled and native code can obviously execute while the scheduler is paused. In the case of unscheduled code one might use global variables to indicate ownership of an object or a context by a scheduled thread. In the case of native functionality (such as dropping an item in my previous example), one has to resort to some very ugly tricks to achieve correctness, if it is at all possible. Being able to have uninterruptable segments of code in the scheduled environment would simplify both tasks, and i doubt it would be very difficult to implement either.

[NeMeSiS 11]

Being able to have uninterruptable segments of code in the scheduled environment would simplify both tasks, and i doubt it would be very difficult to implement either.

 

I may be missing something here, but why cant you just use 'call' for these segments of code? Just cleanliness/readability or is there something else?

[Foxyy 16]

I may be missing something here, but why cant you just use 'call' for these segments of code? Just cleanliness/readability or is there something else?

A function when called in the scheduled environment executes synchronously only in the context of the scheduler, and then only if the function does not wait. Calling does not make a function uninterruptable by the scheduler.

 

For example:

[] spawn
{
	call
	{
		systemChat str diag_frameNo; // n
		
		//A long running command to force the scheduler to go over the 3ms limit for the frame
		selectBestPlaces [position player, 1000, "forest + trees - meadow - houses - (10 * sea)", 5, 1];
		
		systemChat str diag_frameNo; // n + m, m > 0
	};
};

[NeMeSiS 11]

 

A function when called in the scheduled environment executes synchronously only in the context of the scheduler, and then only if the function does not wait. Calling does not make a function uninterruptable by the scheduler.

 

For example:

[] spawn
{
	call
	{
		systemChat str diag_frameNo; // n
		
		//A long running command to force the scheduler to go over the 3ms limit for the frame
		selectBestPlaces [position player, 1000, "forest + trees - meadow - houses - (10 * sea)", 5, 1];
		
		systemChat str diag_frameNo; // n + m, m > 0
	};
};

Ah yes, you are obviously correct.

[Freghar 73]

Not suggesting it's not true, more just concerned there's a fundamental problem with SQF I don't understand, but how does your example result in the behaviour described if the scheduler resumes at the statement following the last one executed?

The issue is that SQF is currently unable to achieve race-free design without mutexes and at all if the code conflicts with what any existing mutex-less code does.

An example, written in pseudocode,

if (player-has-binoculars) {
    remove-rocket-launcher;
}

on the start of a mission, to remove all rocket launchers from "officers".

The condition might match in a completely different environment than the action is executed in:

• missions starts
• condition matches, player has binoculars
• execution gets suspended
• player drops the rocket launcher
• execution resumes
• code tries to remove rocket launcher, nothing to remove, finishes
• player picks up the rocket launcher

There's currently no way to "fix" that (in a scheduled environment).

[ImperialAlex 72]

Except for sync, all the other suggestions are, imho, 100% awesome. 

 

Regarding sync:

As much as I like extending SQF, we must also consider that most people that use SQF are horribly bad at programming. Professional programmers with years of education, training and experience still have a hard time managing concurrent programming.

If non-hacky atomicity where introduced to the game then I can already see how many un- or misinformed creators would end up wrapping their entire program in one large atomic block. It might also be very hard to implement, given that the scheduler doesn't seem to be very sophisticated right now and sync could introduce arbitrarily long execution times.

 

There's also the fact that I can't really see the lack of atomicity creating serious problems if you write scripts in a failure-tolerant way. The example you gave is valid but it has a very low chance of occurring and doesn't really cause significant problems. Yes, somebody now has access to two guns. Boohoo. That's only really an issue for life/zombie/etc-style gamemods which are bending a platform to a purpose way beyond its original specifications so I'd find it hard to justify major changes to the sqf engine just to solve a very very niche problem.  Maybe there are other examples where non-atomicity doesn't just lead to minor gear inconsistencies but to actual major issues. If you can find one, then I'm sure that'd be a better way to convince people that adding atomic blocks is a good idea :)

 

Even more so, all the examples given here (removing weapons/etc) would be very simply solved by using unscheduled code (right?)

[Foxyy 16]

You are correct imperialalex, it's absolutely not a huge issue and might very well cause, among less experienced programmers, the issues you describe. I personally strive for correctness in all aspects of my code, and the lack of tools to do so in this particular regard irritates me. It's something that, if easily implemented, i would very much appreciate having. That said i probably shouldn't have placed it at the top of my list in the original post, the ordering of which is in no way indicative of the importance of each item. In fact, if i could choose only one of these features to be implemented it would definitely be the object serialization.

[torndeco 128]

Except for sync, all the other suggestions are, imho, 100% awesome. 

 

Regarding sync:

As much as I like extending SQF, we must also consider that most people that use SQF are horribly bad at programming. Professional programmers with years of education, training and experience still have a hard time managing concurrent programming.

If non-hacky atomicity where introduced to the game then I can already see how many un- or misinformed creators would end up wrapping their entire program in one large atomic block. It might also be very hard to implement, given that the scheduler doesn't seem to be very sophisticated right now and sync could introduce arbitrarily long execution times.

 

There's also the fact that I can't really see the lack of atomicity creating serious problems if you write scripts in a failure-tolerant way. The example you gave is valid but it has a very low chance of occurring and doesn't really cause significant problems. Yes, somebody now has access to two guns. Boohoo. That's only really an issue for life/zombie/etc-style gamemods which are bending a platform to a purpose way beyond its original specifications so I'd find it hard to justify major changes to the sqf engine just to solve a very very niche problem.  Maybe there are other examples where non-atomicity doesn't just lead to minor gear inconsistencies but to actual major issues. If you can find one, then I'm sure that'd be a better way to convince people that adding atomic blocks is a good idea :)

 

Even more so, all the examples given here (removing weapons/etc) would be very simply solved by using unscheduled code (right?)

The problem is arma is already screwed up, just most people aren't aware of some of the quirks.

Both of the following can happen in scheduled & un-scheduled enviroment

If you add/remove/add uniform straight away you will run into issue were the unit will end up with different uniforms.

Basically if add/removed same frame engine gets confused.

Hell if you use createVehicle sometimes the returned value will be null for a tiny bit of time, until the unit is created. (Need to basically use a waitUntil check if not null)

-----------------------

The sync aka mutex lock command would be very useful.

Atm the only way to accomplish this behaviour is to use a custom callExtension, you can try to write SQF version but it won't work 100% of time.

[pedeathtrian 99]

(edit) (del)

nvm then. i should stop making useless posts on this forum

[Foxyy 16]

I'm aware of these hacky solutions pedeathrian and they're exactly what i would like to avoid. Just as well you could dispatch to a PFH which is clearly not ideal as it requires polling and waiting. You could even dispatch using object initializers or in MP self-publicVariable if you wanted to go the real dirty route.

 

Any FSM condition based solutions will obviously require suspension and as such most likely require further inter-thread synchronization. Also even if the scheduler were to execute two threads intermittently without explicit waiting, you could construct mutexes using array initializers, which if containing only atomic expressions, are themselves atomic. Unfortunately however, this trick cannot be used for branching.

 

I'm not saying any of this is impossible, everything is perfectly safe from race conditions in the unscheduled environment. What i'm saying is all of these hacks could be made obsolete by a single, arguably very simple command.

[Freghar 73]

For anyone interested, there's remoteExecCall that is actually able to execute code in an unscheduled environment even when run from scheduled. The downside is that it produces network traffic (even if the object is local / this client's ID is specified, tested by packet capture) and doesn't pause the scheduled execution nor does it provide any handle to wait on.

It would still be great to have a way of executing unscheduled code from scheduled execution, waiting for it to complete. This could be used to implement mutexes where necessary (independently spawned scripts needing to modify one object and do so in a specific sequential fashion, ie. hierarchical loadout modification).

[fn_Quiksilver 1636]

The problem is arma is already screwed up, just most people aren't aware of some of the quirks.

Both of the following can happen in scheduled & un-scheduled enviroment

If you add/remove/add uniform straight away you will run into issue were the unit will end up with different uniforms.

Basically if add/removed same frame engine gets confused.

Hell if you use createVehicle sometimes the returned value will be null for a tiny bit of time, until the unit is created. (Need to basically use a waitUntil check if not null)

-----------------------

The sync aka mutex lock command would be very useful.

Atm the only way to accomplish this behaviour is to use a custom callExtension, you can try to write SQF version but it won't work 100% of time.

 

I have only encountered the createvehicle issue in unscheduled environment. For this reason and others I always spawn units/vehicles in a scheduled thread.

[torndeco 128]

I have only encountered the createvehicle issue in unscheduled environment. For this reason and others I always spawn units/vehicles in a scheduled thread.

It can happen in scheduled environment, its just rarer to happen.

I had to add some waituntils to make sure the code worked correctly, was just abit slightly annoying.

Especially when the sqf code didn't error out at all

 

 

For anyone interested, there's remoteExecCall that is actually able to execute code in an unscheduled environment even when run from scheduled. The downside is that it produces network traffic (even if the object is local / this client's ID is specified, tested by packet capture) and doesn't pause the scheduled execution nor does it provide any handle to wait on.

It would still be great to have a way of executing unscheduled code from scheduled execution, waiting for it to complete. This could be used to implement mutexes where necessary (independently spawned scripts needing to modify one object and do so in a specific sequential fashion, ie. hierarchical loadout modification).

 

You can use an fsm to call a sqf function to accomplish the same thing without any network overhead.  

http://killzonekid.com/pub/call.fsm

http://killzonekid.com/arma-scripting-tutorials-code-performance/

---------------

Unfortunately you can have multiple eventhandlers running concurrently at the same time.

There is noway in Arma atm to implement a proper mutex atm, its basically using a atomic bool to function as a lock.

Most of the time you will be ok, but eventually multiple code running at same time will get past the sqf lock.

Exile Mod have ran into this issue quite abit.....

It is kinda funny that with Arma SQF

You can copy an array (but that makes a  reference to the array, unless you deep copy the array).

When in SQF do you need to reference an array ??? When ???

But a mutex is appparently to advanced to implement

---------------

If you need a mutex use a simply callExtension that contains a map of bools to function as mutex lock.

It will be pretty simply & run pretty fast aswell.

The only issue is BattlEye will random block this extension if the backend goes down, so you can only really implement this on a server not on clients.

 

[Foxyy 16]

It is possible to implement a proper mutex in SQF.

dayz_code\util\mutex.hpp

/* Provides a simple mutex implementation for synchronization between the scheduled and unscheduled environments.
See https://en.wikipedia.org/wiki/Mutual_exclusion

Author:
	Foxy
*/

#ifndef _INCLUDE_GUARD_MUTEX
	#define _INCLUDE_GUARD_MUTEX
	
	//Initializes a new unlocked mutex.
	#define Mutex_New() [true]
	
	//Attempts to lock the specified mutex. Returns true if the mutex was locked.
	#define Mutex_TryLock(mtx) ((mtx) call dz_fn_mutex_tryLock)
	#define Mutex_TryLock_Fast(mtx) ([(mtx) select 0, (mtx) set [0, false]] select 0)
	
	//Waits until the mutex becomes available and locks it.
	#define Mutex_WaitLock(mtx) ((mtx) call dz_fn_mutex_waitLock)
	#define Mutex_WaitLock_Fast(mtx) waitUntil {Mutex_TryLock_Fast(mtx)}
	
	//Unlocks the mutex. Use only when you have previously obtained lock yourself.
	#define Mutex_Unlock(mtx) ((mtx) set [0, true])
#endif

Also array references are very useful, they allow you to share and pass objects by reference. This might be used for dependency injection or inter-thread communication for example.

[Freghar 73]

It is possible to implement a proper mutex in SQF.

There are more ways - https://forums.bistudio.com/topic/183993-scripting-introduction-for-new-scripters/?p=3016726 (spinlock essentially).

Since that post, I came to an observation that the atomic execution unit of the interpreter is an expression, not a single command, so it likely could be even more simplified.

[commy2 188]

If you want to force execution in unscheduled environment from scheduled environment, you can always use this method:

https://raw.githubusercontent.com/CBATeam/CBA_A3/master/addons/common/fnc_directCall.sqf

 

It even supports return values.

[killzone_kid 1330]

If you want to force execution in unscheduled environment from scheduled environment, you can always use this method:

https://raw.githubusercontent.com/CBATeam/CBA_A3/master/addons/common/fnc_directCall.sqf

 

It even supports return values.

I strongly advise against this ^^^ method! configClasses as well as configProperties is not designed for this. Their purpose is to run simple filter expression, nothing more. The results from using anything else could be unpredictable, including crashes. If you need to jump to unscheduled use remoteExecCall with local target.

[commy2 188]

Is there any evidence for the game actually crashing though? We are using this in ACE since forever and are executing thousands of lines of code on mission start. The only thing that gets borked sometimes is error messages, but those are uterly usless anyway in this game (sorry).

remoteExecCall is unacceptable for addon makers, because it creates network traffic and because of whitelisting shenanigans, BE filters etc.

[commy2 188]

Here is an alternative way abusing "diag_codePerformance":

CBA_fnc_directCall2 = {
    params [["_CBA_function", {}, [{}]], ["_CBA_arguments", []]];

    if (!canSuspend) exitWith {
        _CBA_arguments call _CBA_function;
    };

    private _CBA_return = [];

    diag_codePerformance [{
        params ["_CBA_return", "_CBA_function", "_CBA_arguments"];

        _CBA_return pushBack (_CBA_arguments call _CBA_function);
    }, [_CBA_return, _CBA_function, _CBA_arguments], 1];

    _CBA_return param [0]
};

For those who don't trust the filter for configClasses/Properties.

 

Downside:

- local variables don't carry over, so all local variables will have to be passed as arguments (not a huge deal really)

 

Test:

0 spawn {
    _fncTest = {systemChat str canSuspend};
    call _fncTest;
    _fncTest call CBA_fnc_directCall2;
    call _fncTest;
};

-> true
-> false
-> true

Note: "diag_codePerformance" was once disabled in MP, but now it seems that it is allowed to use one loop, which is sufficient for what we want.

I have tested this in local hosted MP and it works. Haven't tested in dedicated multiplayer though and never on a machine without interface (dedicated server, headless client), but I see no reason why it wouldn't work there too.

The canSuspend check is necessary, because "diag_codePerformance" cannot be used inside itself.

[killzone_kid 1330]

diag_codePerformance is also not designed for this, just like configClasses or configParams or isNil {}. The only safe way of doing it is to use remoteExecCall with local target. Period.

[commy2 188]

Then we need a command designated to do this just like OP suggested. remoteExec/Call ["call"] needs a local object, because clientOwner reports BS in loaded MP games and even then, it can be blacklisted...

[Foxyy 16]

Since we're all posting totally awesome not at all dirty solutions...

//config

class CfgVehicles
{
	class Logic;
	
	class MyShittyObject : Logic
	{
		scope = protected;
		
		class EventHandlers
		{
			init = "my_shitty_result = my_shitty_args call my_shitty_function";
		};
	};
};



//SQF

my_shitty_synccall =
{
	[
		missionNamespace setVariable ["my_shitty_args", _this select 0],
		missionNamespace setVariable ["my_shitty_function", _this select 1],
		missionNamespace setVariable ["my_shitty_object", "MyShittyObject" createVehicleLocal [0,0,0]],
		deleteVehicle (missionNamespace getVariable "my_shitty_object"),
		my_shitty_result
	]
	select 4
};

[commy2 188]

 

Since we're all posting totally awesome not at all dirty solutions...

 

I don't think method works. Init is executed right after you use createVehicle, so it's impossible to assign object namespace variables before that. "setVariable" also needs the object reference on the left side, which you seem to have left out.

 

A solution in A2 was to create a dummy object (e.g. helipad) and add a killed event handler to it and then use "setDamage 1".