Server profileNamespace for saving all data

[Tisor 17]

Hi all.

I know that has been already discussed time ago. I've made a lot of research on the forum and spent 2 days reading, but all the posts ends in the same: this is bugged, this doesn't work, but when I see the ticket, it is meant to be fix already a long time ago.

So now, I'd like to ask the mission gurus over here if they can help me and all the people wondering how, what would be the way to make this works. It's already being used, I think, by Epoch mod servers and so on, so shouldn't be imposible.

I'm having trouble understanding how and what to run server side and what on the client side. I want to make clear that what I want to save the stats to the server profilenamespace and not to the user's one (don't one anyone to hack it :mad: )

So, the "algorithm" I got in mind would be something like this:

mission init -> Calls a server_init.sqf file saved on an addon (@myserver) which is only loaded in the server. (Just to avoid the things to be looked at easily)

In that server_init:

If (is server) then {

load the server data such as vehicle positions, custom buildings, etc... This only must be done on the first run of the server, not everytime a client join.

}

If (is a client) then {

load his own stuff associated with it's own uid stored on the server profilenamespace as an array (uid->data).

}

Then, use a onplayerdisconnected EH to save the client data again to the server profile on the "master array", but, yeah, should this run on the server or in the client?

And, of course, every x minutes, update all the data to the server profile, just in case server crash...

I know this is a bit tricky and maybe I'm on the wrong way on the way I'm thinking, but yeah, I think we all together can get a working code we all can use in our missions (like life, wasteland or any other projects you have in mind).

Hope we can help each other.

Regards

[Tajin 348]

- You don't have to put your server-side scripts into an addon, just put em in the ArmA dir or a subdirectory on the server. That works too and you can easily edit them without having to repack an addon.

- onPlayerDisconnected should run on the server.

- for bigger amounts of data, its not recommended to save them in the profile anyway. Use a database extension like extDB or inidbi.

Aside from that, the approach is pretty solid.

- All scripts concerning the gameflow should be kept strictly server-side.

- Include a few functions that provide data to the users, on a per-request basis.

- All scripts that are about visual gameplay effects, or GUI are in general best run local on each client.

Of course that is only a rough guideline and still largely depends on what each script does.

[Tisor 17]

The reason why I'm trying this approach is because setting up extdb or inidb is so hard to understand and difficult on the server side.

Using profile namespace, you dont need that much to setup more than a -mod=@server...

So, I'm still trying to get to my approach, but i've not been able atm.

Will post back if I got something but hope someone could throw a bit more of help over here so I could finnally understand how can a client ask for a value stored on a server variable... That's what is becoming hard to me to understand :(

Thanks for your points Tajin.

[jshock 512]

Could you just publicVariable the server stored variable? This may be incorrect in some capacity, but to be honest I'm in the middle of learning this sort of thing as well, but hopefully DreadedEntity (or any of the other well qualified people) will see this, he has a pretty good grasp of it all :p.

[Tajin 348]

inidbi is actually very simple to setup. All you need to do is activate the mod.

Anyway, as far as requesting data from the server is concerned, here is a simplified example to request or save a loadout-array from the server (with inidbi).

On the server:

call compile preProcessFile "\inidbi\init.sqf";
campaignFile = "Campaign_0";

fnc_getLoadout = {
private ["_player","_uid","_cid"];
_player = _this select 0;
_uid = getPlayerUID _player;
_cid = owner _player;

loadout = [_uid, campaignFile, "loadout", "ARRAY"] call iniDB_read;
_cid publicVariableClient "loadout";
}

fnc_saveLoadout = {
private ["_player","_loadout","_uid"];
_player = _this select 0;
_loadout = _this select 1;
_uid = getPlayerUID _player;

[_uid, campaignFile, "loadout", _loadout] call iniDB_write;
}

On the client (to get the loadout):

loadout = false;
[ [player], "fnc_getLoadout", false] spawn BIS_fnc_MP;
waitUntil {typeName loadout == "ARRAY"};
// do something with loadout

On the client (to save the loadout):

[ [player,loadout], "fnc_saveLoadout", false] spawn BIS_fnc_MP;

Alternately, instead of using public variable you could also write functions for the client that process the data sent back by the server. That would actually be a bit cleaner and safer.

Edited November 6, 2014 by Tajin

[iceman77 18]

Thankyou for posting an example Tajin. :)

[Tajin 348]

Here is another example with functions instead of publicVariable (again with inidbi):

Server:

call compile preProcessFile "\inidbi\init.sqf";
campaignFile = "Campaign_0";

fnc_getStringData = {
private ["_player","_uid","_callback","_data"];
_player = _this select 0;
_entry = _this select 1;
_callback = _this select 2;
_uid = getPlayerUID _player;

_data = [_uid, campaignFile, _entry, "STRING"] call iniDB_read;
[ _data, _callback, _player] spawn BIS_fnc_MP; 
};

fnc_saveData = {
private ["_player","_entry","_data","_uid"];
_player = _this select 0;
_entry = _this select 1;
_data = _this select 2;
_uid = getPlayerUID _player;

[_uid, campaignFile, _entry, _data] call iniDB_write;
};

Client:

fnc_myCallbackFunction = {
player sideChat format["Data recieved: %1",_this];
};

// save data on the server
[ [player,"myDBEntryName","somedata"], "fnc_saveData", false] spawn BIS_fnc_MP;

// get data from the server
[ [player,"myDBEntryName","fnc_myCallbackFunction"], "fnc_getStringData", false] spawn BIS_fnc_MP;

Note that in both examples, the data gets sent only to the client who requested it.

-> link to inidbi <-

Edited November 6, 2014 by Tajin

[dreadedentity 278]

It's a good idea to reduce network traffic wherever possible, while working on a similar system I've found that having clients send data to the server was unnecessary, clients need only make requests and receive data.

Run this server-only

"GLOBAL_requestData" addPublicVariableEventHandler
{
[(_this select 1)] execVM "selectDatabaseEntry.sqf";
GLOBAL_sendData = _result; //this part would be inside "selectDatabaseEntry.sqf"
(owner(_this select 1)) publicVariableClient "GLOBAL_sendData"; //also this
};

[] spawn
{
while {true} do;
{
	{
		if (isPlayer _x) then
		{
			[_x] execVM "savePlayerGear.sqf";
		};
	}forEach playableUnits;
}
sleep (60 * 5); //will run every 5 minutes
};

Run this on client

"GLOBAL_sendData" addPublicVariableEventHandler
{
[(_this select 1)] execVM "loadPlayerGear.sqf";
};

Please note that it is 100% possible to split large data into "chunks", to further reduce network traffic.

Edited November 6, 2014 by DreadedEntity

[Tajin 348]

True but that really depends on what you want to save. ;)

Btw. In your example it would probably be safer to use a different variable for each client. Simply add/remove the eventhandler dynamically when a player joins/leaves and include his clientId in the varName.

[Tisor 17]

inidbi looks very easy to use... gonna give it a try tomorrow... thank you Tajin for your help.

[Viba 1]

It's a good idea to reduce network traffic wherever possible, while working on a similar system I've found that having clients send data to the server was unnecessary, clients need only make requests and receive data.

Run this server-only

"GLOBAL_requestData" addPublicVariableEventHandler
{
[(_this select 1)] execVM "selectDatabaseEntry.sqf";
GLOBAL_sendData = _result; //this part would be inside "selectDatabaseEntry.sqf"
(owner(_this select 1)) publicVariableClient "GLOBAL_sendData"; //also this
};

[] spawn
{
while {true} do;
{
	{
		if (isPlayer _x) then
		{
			[_x] execVM "savePlayerGear.sqf";
		};
	}forEach playableUnits;
}
sleep (60 * 5); //will run every 5 minutes
};

Run this on client

"GLOBAL_sendData" addPublicVariableEventHandler
{
[(_this select 1)] execVM "loadPlayerGear.sqf";
};[/spoiler]

Please note that it is 100% possible to split large data into "chunks", to further reduce network traffic.

To even further improve this:

- Have the client run the saving loop each 5 minutes

- Also have the client to process and gather all the required saving information accompanied with UID or another way of identification into an array or string for later use (Array overhead http://feedback.arma3.com/view.php?id=15391)

- Only have a pubvarEH on the server for actual saving function

-> Clients send the already processed saving information in the array/string to the server that then saves it immediately

-> This way any "custom" information saved on the clients via setvariable can be local minimizing network traffic even further when the server doesn't have to have the capability to access it

This way the server has less to do and clients that have no problem performing such tasks help out instead. I see no point in having the server constantly process each client in a separate loop.

Another fun thing to do would be to save the previously sent information locally on the client, cross check during the next saving if any information is the same -> don't send the identical information to the server, only the new info.

Also about the saving, if you even have a slight thought about expanding other functions to utilize saving I strongly recommend you directly jump on the extDB & MySQL bandwagon. If not then inidbi or profileNamespace is probably good for you.

Edited November 6, 2014 by Viba

[Tisor 17]

To even further improve this:

- Have the client run the saving loop each 5 minutes

- Also have the client to process and gather all the required saving information accompanied with UID or another way of identification into an array or string for later use (Array overhead http://feedback.arma3.com/view.php?id=15391)

- Only have a pubvarEH on the server for actual saving function

-> Clients send the already processed saving information in the array/string to the server that then saves it immediately

-> This way any "custom" information saved on the clients via setvariable can be local minimizing network traffic even further when the server doesn't have to have the capability to access it

Can anyone please explain whats the utility of the publicVariableEH? Or at least how it works in this case. Your are talking about it but I dont get the point.

I know it runs something everytime that variable changes, but why we need it for saving?

And Viba, you mean that the client get the arrays ready before they ask the server to save them, right? So it doesn't have to make the calculation at the same time? Is that what you mean?

This way the server has less to do and clients that have no problem performing such tasks help out instead. I see no point in having the server constantly process each client in a separate loop.

You mean every client have the info ready and the server just collect everyclient info every x minutes? right?

Edited November 7, 2014 by Tisor

[Viba 1]

https://community.bistudio.com/wiki/addPublicVariableEventHandler

myVariable = "my string";

Now myVariable is a global variable on the "node" (client or server) where this part of a script has been executed.

With publicvariable, publicvariableclient or publicvariableserver you can broadcast this global variable to everyone, a specific client or only the server.

A publicvariable eventhandler is triggered for the specified variable when the client/server that has the pubvar eventhandler receives a broadcast when one of the three commands above has been executed on another node.

So say we do this on the server:

"myVariable" addPublicVariableEventHandler
{
   diag_log myVariable;
};  

Now the server has an EH specified for myVariable. If a client publicvariables myVariable with some information the server then immediately diag_logs that information.

Client:

myVariable = "my text";
publicvariable "myVariable";

-> Open up the servers .rpt log and you should find "my text" from there.

This is used to request and send information to the server on demand, otherwise you would need to have somekind of loop checking for changes to a variable.

About the saving loop, I mean every client should have the loop running locally for them. Upon saving time the clients themselves gather all saving information and just passes it onwards to the server that then saves it.

[Tajin 348]

Upon saving time the clients themselves gather all saving information and just passes it onwards to the server that then saves it.

The downside beeing, that it creates more network traffic.

I would normally only use that method for things that are inherently clientside, depends on the script though.

[Tisor 17]

https://community.bistudio.com/wiki/addPublicVariableEventHandler

myVariable = "my string";

Now myVariable is a global variable on the "node" (client or server) where this part of a script has been executed.

With publicvariable, publicvariableclient or publicvariableserver you can broadcast this global variable to everyone, a specific client or only the server.

A publicvariable eventhandler is triggered for the specified variable when the client/server that has the pubvar eventhandler receives a broadcast when one of the three commands above has been executed on another node.

So say we do this on the server:

"myVariable" addPublicVariableEventHandler
{
   diag_log myVariable;
};  

Now the server has an EH specified for myVariable. If a client publicvariables myVariable with some information the server then immediately diag_logs that information.

Client:

myVariable = "my text";
publicvariable "myVariable";

-> Open up the servers .rpt log and you should find "my text" from there.

This is used to request and send information to the server on demand, otherwise you would need to have somekind of loop checking for changes to a variable.

About the saving loop, I mean every client should have the loop running locally for them. Upon saving time the clients themselves gather all saving information and just passes it onwards to the server that then saves it.

Perfect explanation Viba. Really much appreciated, now I got the point on that.

Very clear.

So, what's better for the performance/network traffic?

The client push the data to the server or the server pull the data from the client?

I think we are getting a really useful post for everyone and could be a nice resource for all of us :)

[dreadedentity 278]

Perfect explanation Viba. Really much appreciated, now I got the point on that.

Very clear.

So, what's better for the performance/network traffic?

The client push the data to the server or the server pull the data from the client?

I think we are getting a really useful post for everyone and could be a nice resource for all of us :)

Those are equally terrible for network traffic. If you're trying to get specific information from an object that exists, you should use commands that can accept global arguments. This is when you have to start learning about locality. Take getPos, for example. Say I was doing some testing on my dedicated server and I needed the location of my player, so I could publicVariable that from the client, but it's already constantly being broadcast, so we'll use getPos from the server with my controlled unit (non-local) as an argument and get it that way. Now I've gotten my location, and without adding any extra traffic.

I like to call these kinds of commands "grabbers", because they can get whatever they want, from whoever they want, whenever they want :p

[Tisor 17]

Those are equally terrible for network traffic. If you're trying to get specific information from an object that exists, you should use commands that can accept global arguments. This is when you have to start learning about locality. Take getPos, for example. Say I was doing some testing on my dedicated server and I needed the location of my player, so I could publicVariable that from the client, but it's already constantly being broadcast, so we'll use getPos from the server with my controlled unit (non-local) as an argument and get it that way. Now I've gotten my location, and without adding any extra traffic.

I like to call these kinds of commands "grabbers", because they can get whatever they want, from whoever they want, whenever they want :p

So looking at this... Would be better to sacrify server resources processing those queries (weapons player, getvariable, blablabla)? Wont be better to actually send the data to the server as an array/string already processed?

I think is a matter of resources vs network traffic?

[dreadedentity 278]

It depends on what you're trying to do. But both resource use and network traffic should be minimized as much as possible, while retaining the abilities of the original code. That's pretty much the motto of optimization

[Viba 1]

Hmm this tradeoff ofcourse depends on the situation. Either you are using the servers resources and less network bandwidth for the saving, or vice versa. If you have only a few simple and fast checks that need to be done to gather the information then process it on the server. I'd rather send a preprocessed string/array from the client that's a few characters longer to the server if the checks are a bit more requiring. Simply put I don't like the idea of having 60-100 clients poking the server each in x min intervalls, "save me and process all this stuff for me before it". And if you process the information on the client you can opt to have anything "custom" local to the client, for saving you can then utilize local setvariables and global variables, no need to broadcast them through the network (setvariable with true flag for global).

Optimizing is interesting and it can often be hard to say what solution is actually the best, it's good to hear other options and opinions regarding it. Testing something like this would require some effort to get accurate results as the effects would probably not even be that noticeable :)

Edit:

I just had to check, as a string for one typical save in my mission it would be around 400 characters. I'm not that familiar with how ArmA handles packets or even generally about network packets. But say there's some overhead and thus the packet size would be 500 bytes/save. So with saving at 5min intervalls and 100 clients we'd get a bandwidth cost of the following: 1/3 saves per second * 500 bytes/save = 0.167KB/s of network traffic, if the information would be processed and sent from the client to the server. With more information about setvariable broadcasts it could be interesting to compare this to, say about 5-10 global setvariables on each client repeatedly broadcasting through the network to everyone.

Maybe puts it more into perspective with calculations like that, someone correct me if I'm horribly wrong with these figures :)

Edited November 8, 2014 by Viba

[dreadedentity 278]

Don't use setVariable for broadcasts, it creates a game logic that synchronizes the data for all clients, even JIP clients. If you've ever joined a server that got a red chain every time someone joined, that's specifically why.

[Viba 1]

I believe the downloading of the mission file has a bit bigger impact on most of the servers regarding the red-link problem during joining, so maybe not "specifically" :)

But yes, publicvariables, global setvariables and so on need to be synced upon clients joining the server.

Edited November 8, 2014 by Viba

[vbawol 47]

Hi all.

I know that has been already discussed time ago. I've made a lot of research on the forum and spent 2 days reading, but all the posts ends in the same: this is bugged, this doesn't work, but when I see the ticket, it is meant to be fix already a long time ago.

So now, I'd like to ask the mission gurus over here if they can help me and all the people wondering how, what would be the way to make this works. It's already being used, I think, by Epoch mod servers and so on, so shouldn't be imposible.

I'm having trouble understanding how and what to run server side and what on the client side. I want to make clear that what I want to save the stats to the server profilenamespace and not to the user's one (don't one anyone to hack it :mad: )

So, the "algorithm" I got in mind would be something like this:

I know this is a bit tricky and maybe I'm on the wrong way on the way I'm thinking, but yeah, I think we all together can get a working code we all can use in our missions (like life, wasteland or any other projects you have in mind).

Hope we can help each other.

Regards

Epoch Mod does not use profileNamespace only those "hacked/leaked" server files do that... We use our own custom solution.