Removing whitespace script release

[das attorney 857]

Preamble:

 

From the wiki, we can use splitstring and joinstring to remove white space from code.  So if we have a function like this one (from the wiki):

 

fnc_deleteTestObj = {
    _this addMPEventHandler ["MPKilled", {
        _this = _this select 0;
        {
            deleteVehicle _x;
        } forEach (_this getVariable ["effects", []]);
        if (isServer) then {
            deleteVehicle _this;
        };
    }];
    _this setDamage 1; 
};

 

We can do:

 

copyToClipboard(([fnc_deleteTestObj] joinString "") splitString toString [9,13,10] joinString "")

 

and it returns:

 

{  _this addMPEventHandler ["MPKilled", {   _this = _this select 0;   {    deleteVehicle _x;   } forEach (_this getVariable ["effects", []]);   if (isServer) then {    deleteVehicle _this;   };  }];  _this setDamage 1;  }

 

Which is quite nice.  But I wanted a way to remove as much white space as I can (for sending code/variables over the network etc), plus I just fancied playing about with recursion.

 

So with this function, we can use our "fnc_deleteTestObj" function again and plug it in:

 

copyToClipboard(([fnc_deleteTestObj] joinString "") call fnc_whitespace)

 

This returns:

 

{_this addMPEventHandler["MPKilled",{_this=_this select 0;{deleteVehicle _x;}forEach(_this getVariable["effects",[]]);if(isServer)then{deleteVehicle _this;};}];_this setDamage 1;}

 

I'm using joinstring in these examples as opposed to format as it doesn't have the 8k limit.  Apparently formatText isn't limited and I'm not sure about str.

 

But you could do this I suppose - just be aware it might have a limit:

 

(str fnc_deleteTestObj) call fnc_whitespace

 

Uses

 

As mentioned, you could shorten variables prior to sending them over the network (then compileFinal at other end).  I'm aware that programs like these exist already, but I wanted to do it in sqf (like within the engine).  It should be ok but will have a problem if you do silly things like putting spaces into variables ex: player setVariable ["there are spaces in this",true]; which is a bit of a no-no imo and bad practice.

 

You could probably obfuscate your code with this if you want but there are already good online obfuscators available anyway, and also good online de-obfuscators, like this one:  http://unminify.com/

 

I think Dwarden said he used to do this to his mission files to make them smaller so they wouldn't suck up as much bandwidth, so theres a use for it there I suppose.

 

Obv, I haven't tested this much but it seems ok so far - ymmv.  Be aware if you use it on very large functions, then it is recursive so it might freeze your game for a few seconds.  If you rename it, make sure you rename the call to it in the code as well or you will break it.

 

Function:

 

Here it is for any interested:

 

fnc_whitespace = {
    private _noWhitespace = toArray(_this);
    private _anyWS = false;
    {
        if (0 isEqualType _x) then {
            if ([9,13,10] find _x > -1) then {
                _noWhitespace set [_forEachIndex,objNull];
                _anyWS = true;
            } else {
                if ([34,39,94,42,40,41,123,125,91,93,59,58,62,47,43,45,61,46] find _x > -1) then {
                    if (_forEachIndex - 1 > -1) then {
                        if ([32,9,13,10] find (_noWhitespace select (_forEachIndex - 1)) > -1) then {
                            _noWhitespace set [_forEachIndex - 1,objNull];
                            _anyWS = true;
                        }
                    };
                    if (_forEachIndex + 1 < count _noWhitespace) then {
                        if ([32,9,13,10] find (_noWhitespace select (_forEachIndex + 1)) > -1) then {
                            _noWhitespace set [_forEachIndex + 1,objNull];
                            _anyWS = true;
                        }
                    };
                }
            }
        };
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    if (_anyWS) then {
        _noWhitespace = (toString _noWhitespace) call fnc_whitespace;
        _noWhitespace = toArray _noWhitespace;
    };
    toString(_noWhitespace)
};

 

And here it is compacted (ho ho ho):

 

fnc_whitespace={private _noWhitespace=toArray(_this);private _anyWS=false;{if(0 isEqualType _x)then{if([9,13,10]find _x>-1)then{_noWhitespace set[_forEachIndex,objNull];_anyWS=true;}else{if([34,39,94,42,40,41,123,125,91,93,59,58,62,47,43,45,61,46]find _x>-1)then{if(_forEachIndex-1>-1)then{if([32,9,13,10]find(_noWhitespace select(_forEachIndex-1))>-1)then{_noWhitespace set[_forEachIndex-1,objNull];_anyWS=true;}};if(_forEachIndex+1 < count _noWhitespace)then{if([32,9,13,10]find(_noWhitespace select(_forEachIndex+1))>-1)then{_noWhitespace set[_forEachIndex+1,objNull];_anyWS=true;}};}}};}forEach _noWhitespace;_noWhitespace=_noWhitespace-[objNull];if(_anyWS)then{_noWhitespace=(toString _noWhitespace)call fnc_whitespace;_noWhitespace=toArray _noWhitespace;};toString(_noWhitespace)}

 

Credit to @dreadedentity for his initial code HERE

 

Enjoy!

[sarogahtyp 1104]

Thx for this release which could be really handy.

Maybe this is a stupid question but:
Could one go a step further and write a script which uses a lightweight packing algorithm and then get a much smaller result?
Or are there disadvantages which idk bout?

sent from mobile using Tapatalk

[MrCopyright 107]

So the only difference is the size of the variable being transferred across the network? How much of a performance impact does this have?

[sarogahtyp 1104]

3 minutes ago, MrCopyright said:

So the only difference is the size of the variable being transferred across the network? How much of a performance impact does this have?

depends on how much such transfers u r doing ...

 

EDIT:

there are guys which look for ways to send all scripts from server to client... I think they have use for such things.

[MrCopyright 107]

I currently send over 30+ functions from the server to the client when they connect. I haven't noticed any performance hit as this process takes less than a few seconds but this can only improve network traffic. Thank you @das attorney.

[das attorney 857]

Thanks for the comments  ;)  Here's an updated version that removes pointless ";" before closing brackets.

 

IE: instead of ;};};}; it would look like }}};

 

So before it would return something like this:

 

{_this addMPEventHandler["MPKilled",{_this=_this select 0;{deleteVehicle _x;}forEach(_this getVariable["effects",[]]);if(isServer)then{deleteVehicle _this;};}];_this setDamage 1;}

 

And now it returns this:

 

{_this addMPEventHandler["MPKilled",{_this=_this select 0;{deleteVehicle _x}forEach(_this getVariable["effects",[]]);if(isServer)then{deleteVehicle _this}}];_this setDamage 1}

 

Look at the chars after deleteVehicle _this  (near the end).  Obv it doesn't make much difference in this small function, but can remove a fair bit of ; ; ; in larger functions.

 

 

fnc_whitespace = {
    private _noWhitespace = toArray _this;
    private _anyWS = false;
    {
        if (0 isEqualType _x) then {
            if ([9,13,10] find _x > -1) then {
                _noWhitespace set [_forEachIndex,objNull];
                _anyWS = true;
            } else {
                if ([34,39,94,42,40,41,123,125,91,93,59,58,62,47,43,45,61,46] find _x > -1) then {
                    if (_forEachIndex - 1 > -1) then {
                        if (32 isEqualTo (_noWhitespace select (_forEachIndex - 1))) then {
                            _noWhitespace set [_forEachIndex - 1,objNull];
                            _anyWS = true;
                        }
                    };
                    if (_forEachIndex + 1 < count _noWhitespace) then {
                        if (32 isEqualTo (_noWhitespace select (_forEachIndex + 1))) then {
                            _noWhitespace set [_forEachIndex + 1,objNull];
                            _anyWS = true;
                        }
                    };
                    if (125 isEqualTo _x) then {
                        if (59 isEqualTo (_noWhitespace select (_forEachIndex - 1))) then {
                            _noWhitespace set [_forEachIndex - 1,objNull];
                            _anyWS = true;
                        }
                    }
                }
            }
        };
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    if (_anyWS) then {
        _noWhitespace = (toString _noWhitespace) call fnc_whitespace;
        _noWhitespace = toArray _noWhitespace;
    };
    toString(_noWhitespace)
};

 

 

EDIT:  Working on a much quicker version (simpler too).  There's a couple of probs with it, (but also some improvements hopefully) and I'll post up when I can  :)

[Grumpy Old Man 3540]

Holy moly, I already like this!

Do you think it makes sense to diag_log in debug mode how much stuff got removed?

 

Cheers

[das attorney 857]

That's a nice idea, but because it's recursive it's a little tricky to add in (although the next version isn't recursive, so I can add it very easily).  :)

 

For now, you could put

 

diag_log count toArray str myFunction

 

Then run the script, then count the output

 

diag_log count toArray str _return.

 

[das attorney 857]

Here's a new, and much faster version.  Just be aware that this (and all of the earlier ones) mess up formatting within strings.  I'm working on a way to escape strings, and will post back up when I find something reliable.

 

So if you did this:

 

{"hello, my name is Mr Stkrdknmibalz.  I live in a beautiful house, with my lovely wife and 2 kids"}

 

Then you get:

 

{"hello,my name is Mr Stkrdknmibalz.I live in a beautiful house,with my lovely wife and 2 kids"}

Anyway, here's the new script (with a couple of diags thrown in to log start and end length):

 

fnc_whitespace = {
    private _noWhitespace = toArray _this;
    diag_log format ["char count start: %1", count _noWhitespace];
    {
        if (0 isEqualType _x) then {
            if ([9,13,10] find _x > -1) then {
                _noWhitespace set [_forEachIndex,objNull];
            } else {
                if ([34,39,94,42,40,41,123,125,91,93,59,58,62,47,43,45,61,46,44,33] find _x > -1) then {
                    scopeName "scope";
                    for "_i" from 1 to 100 do {
                        if (_forEachIndex - _i > -1) then {
                            if ([32,9,13,10] find (_noWhitespace select (_forEachIndex - _i)) > -1) then {
                                _noWhitespace set [_forEachIndex - _i,objNull]
                            } else {
                                breakTo "scope"
                            }
                        }
                    };
                    for "_i" from 1 to 100 do {
                        if (_forEachIndex + _i < count _noWhitespace) then {
                            if ([32,9,13,10] find (_noWhitespace select (_forEachIndex + _i)) > -1) then {
                                _noWhitespace set [_forEachIndex + _i,objNull]
                            } else {
                                breakTo "scope"
                            }
                        }
                    }
                }
            }
        };
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    {
        if (125 isEqualTo _x) then {
            if (59 isEqualTo (_noWhitespace select (_forEachIndex - 1))) then {
                _noWhitespace set [_forEachIndex - 1,objNull]
            }
        }
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    diag_log format ["char count end: %1", count _noWhitespace];
    toString _noWhitespace
};

 

[das attorney 857]

Ok - got a version that respects "STRING"

 

This is the code:

fnc_whitespace = {
    private _noWhitespace = toArray _this;
    diag_log format ["char count start: %1", count _noWhitespace];
    private _stringCharIndexes = [];
    {
        scopeName "pluto";
        if (34 isEqualTo _x and {not (_forEachIndex in _stringCharIndexes)}) then {
            _stringCharIndexes pushBack _forEachIndex;
            for "_i" from (_forEachIndex + 1) to 999999 do {
                if (_i > count _noWhiteSpace -1) exitWith {};
                if not (34 isEqualTo (_noWhiteSpace select _i)) then {
                    _stringCharIndexes pushBack _i;
                } else {
                    _stringCharIndexes pushBack _i;
                    breakTo "pluto"
                }
            }
        }
    } forEach _noWhiteSpace;
    {
        if (0 isEqualType _x) then {
            if not (_forEachIndex in _stringCharIndexes) then {
                if ([9,13,10] find _x > -1) then {
                    _noWhitespace set [_forEachIndex,objNull];
                } else {
                    if ([34,39,94,42,40,41,123,125,91,93,59,58,62,47,43,45,61,46,44,33] find _x > -1) then {
                        scopeName "mars";
                        for "_i" from 1 to 100 do {
                            if (_forEachIndex - _i > -1) then {
                                if ([32,9,13,10] find (_noWhitespace select (_forEachIndex - _i)) > -1) then {
                                    _noWhitespace set [_forEachIndex - _i,objNull]
                                } else {
                                    breakTo "mars"
                                }
                            }
                        };
                        for "_i" from 1 to 100 do {
                            if (_forEachIndex + _i < count _noWhitespace) then {
                                if ([32,9,13,10] find (_noWhitespace select (_forEachIndex + _i)) > -1) then {
                                    _noWhitespace set [_forEachIndex + _i,objNull]
                                } else {
                                    breakTo "mars"
                                }
                            }
                        }
                    }
                }
            }
        };
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    {
        if (125 isEqualTo _x) then {
            if (59 isEqualTo (_noWhitespace select (_forEachIndex - 1))) then {
                _noWhitespace set [_forEachIndex - 1,objNull]
            }
        }
    } forEach _noWhitespace;
    _noWhitespace = _noWhitespace - [objNull];
    diag_log format ["char count end: %1", count _noWhitespace];
    toString _noWhitespace
};

This is the test function I ran it through:

 

fnc_deleteTestObj = {
    _testLine = "Hello, my name is Mr Stkrdknmibalz. ""%1""  I teach the Canadian ABC to Americans.";
    _this addMPEventHandler ["MPKilled", {
        _this = _this select 0;
        {
            deleteVehicle _x;
        } forEach (_this getVariable ["effects", []]);
        if (isServer) then {
            deleteVehicle _this;
        };
    }];
    _this setDamage 1;
};

And this is the result:

 

{_testLine="Hello, my name is Mr Stkrdknmibalz. ""%1""  I teach the Canadian ABC to Americans.";_this addMPEventHandler["MPKilled",{_this=_this select 0;{deleteVehicle _x}forEach(_this getVariable["effects",[]]);if(isServer)then{deleteVehicle _this}}];_this setDamage 1}

 

 

I tried it on bis_fnc_dynamicGroups and it reduced the size from 38538 characters to 29001 (plus all formatting was OK).

 

:)

 

 

Edit:  next up is excluding strings encapsulated with '

Might have to do some reformatting from " to '

[Harzach 2505]

Very clever stuff, DA!

[das attorney 857]

56 minutes ago, Harzach said:

Very clever stuff, DA!

 

Thank you but nah, I'm not that clever.   :)               (I just wanted to put a normal smiley up, but this forum made me put that nonsense up)  :)

 

I would have got some stuff done tonight if it wasn't for that pesky Bad Benson.  Bad, Bad Benson!   xD