AAR (After Action Review - replay of your battle in Arma2)

[DZR_Mikhail 14]

Ok. I've found out that it's not jlib causing blocking. It's something in my cycle prior to jlib. I uploaded the sources for you to investigate my SQF piece of c...

Please, have a look.

https://gist.github.com/1369526

Everything is shiny until script reaches Line #52 in RECORD.SQF and then game is lagging monstrously.

[.kju 3240]

a) use tags: http://www.ofpec.com/tags/

b) you fail to init 'global_buffer = [];'

c) comment lines starting at the end to find out which area/command exactly causes the issue

d) it might be the array/too much data to handle

e) why do you have a separate thread for the timer?

[DZR_Mikhail 14]

a) wow, thanks for the hint. I'll revise this.

b) sorry, forgot to add init.sqf to the gist. Now it's added at the previous link. global buffer is working ok and initialized.

c) most of the end-line comments are added right before I uploaded the files. I'll fix this.

e) The separate thread with timer I guess will ensure I have global uninterrupted timestamp of the time started right after I started recording. There's also the game time, since mission start.

Anyways, a lot of my ideas and implementations are of very old design and even if new they are a bit amateur and in some points may lack logics or even common sense. I do whatever I can and sometimes different people suggest things and I use it, but very very few people, like you, look at the whole picture and understand the full idea.

So your help is very much appreciated and I'm happy to get any advice you can give. Thank you, PvPscene :)

---------- Post added at 01:47 PM ---------- Previous post was at 01:38 PM ----------

Ok. Tag registered.

[.kju 3240]

ref e)

my problems with it is that if you are unlucky you could have same timer var for a second recording loop

ref c)

this is not about the comments, but to comment code to find out where the bottleneck exactly is

like disable this line and see if it is better:

global_buffer set [count global_buffer, _write_array];

do this until you find the problematic area or areas

you can also try to use diag_tickTime logging to get a better idea of the running of each statement or code block

http://community.bistudio.com/wiki/diag_tickTime

[DZR_Mikhail 14]

ref e) I guess here comes the miscommunication. English is not my native :) I sometimes fail to completely understnd things :) Can you paraphrase? You mean I could do the same timer by using the recording loop and get rid of timer.sqf at all?

I think it's possible, because timer was from the beginning, but recording loop was introduced few days ago.

ref c) Ouch! That's what you meant! Ok. I thought you meant my comments are causing lag :) Rgr. I'll continue debugging by commenting.

[.kju 3240]

ref e)

yep. imo doing it in one loop is safe®/best

[DZR_Mikhail 14]

I tested almost everything. It appears that the main lag is coused by the two foreach loops where unit and vehicle data is gathered. I inserted the sleep 0.001 and it got more stable, but of course less data if captured for the frame.

The separate timer thread must be returned, because it has different pause from the one in the buffered write loop. I'll separate them back.

If I will be forced to use these pauses in foreach loops then time for capture will grow progressively :(

Things could get waaaaaaay easier if I we had an eventhandler for movement. I'll investigate the animation EH and I will be just letting everything idle and the whole data sizes and logics will get simpler I guess. There will be no loop to gather data at all. Every event would be just simply dumped directly to the buffer.

I didn't know arma is soooo not multithreaded. As far as I understand there's no multithreading at all. All the actions are just consecutive. It's awful. I'm a bit at a loss...

---------- Post added at 10:16 AM ---------- Previous post was at 10:07 AM ----------

Hey, may be I shall capture only those units who are animated? I mean, unit's can't move without animation right? Then it's kinda "Move" eventhandler. What do you think?

---------- Post added at 11:04 AM ---------- Previous post was at 10:16 AM ----------

Well, it was a nice try to optimize full unit capture. So time runs out and I'm switching to simple record and map playback. Let's leave moving soldiers for the future.

[DZR_Mikhail 14]

It seems that the lag is caused outside arma... I had no time for final check, but tomorrow I'll know for sure.

I disabled whatever I could and lag still started at record. So there can be only one cause - my shitty external app, written in Delphi and compiled on XP, but now I have Windows 7.

[mahuja 12]

"I didn't know arma is soooo not multithreaded. As far as I understand there's no multithreading at all."

Except in the absense of specifications on what may yield to another (pseudo-)thread, it's incorrect to write code that isn't MT safe. Even if it appears to work. (At best it would be "relying on undocumented behavior".)

It appears that the game engine is basically halting (at the very least, thread barrier style) until the script engine is "finished" - which will basically mean "reached a yield point and didn't continue (even in another thread)". I made a sqf containing a single repeated line like x=x+1; a few million times or so, and after the 15min compile was finished, calling the resulting function caused the game to freeze for ~0.05 seconds or so. Not much, but very noticable. If it would have been called every frame or so, you'd have a real framerate problem.

I believe there's a yield point in entering loops - else loops would freeze the game without explicit sleeps. I would suspect there's a yield point in calling a function. (If x = { _this call x; }; 0 call x; freezes the game, it doesn't have a yield point.) I would have thought there would be a yield point between each call to the foreach body, but your post ("main lag is coused by the two foreach loops") indicates there is not.

By yield point, I may mean potential yield point. It's unspecified.

----

"All the actions are just consecutive. It's awful. I'm a bit at a loss..."

On the other hand, the way it works now is probably why the average mission hasn't collapsed to a crawling compost heap. If the script engine itself ran the pseudothreads in true parallel mt, (and we could ignore the synchronization for doing something to objects), race conditions would probably break every second session or more on average. That's the state of most sqf code out there.

SQF coders* aren't experienced MT programmers by trade. Which is probably one reason why it's so half-baked in that respect.

[DZR_Mikhail 14]

This weekend I had a chance to rewrite BIS_fn_UnitCapture and BIS_fn_UnitPlay function and extended them to AAR needs. So far it's a very very promising solution. These functions had some very nice frame handling optimizations which my scripts lacked for sure. And of course it's a proven playback algorithm.

When I integrate the most vital features in these functions, such as engine capturing, health and damage then I'll make a pbo for these functions to be available ingame as MHL_adv_UnitCapture and MHL_adv_UnitPlay. Yes, only two of them needed.

I tuned it and extended. Tested it on my AAR demo mission and I like the results.

I assign the function to units I want to record, pass it's variable and a unique index to the function.

Hit ESC when you want to stop recording. Hit F2 to copy all data to clipboard, or hit F1 to play it all back before copy.

You can see the demo on the video below.

3Cj_hNxcmEw

[kremator 1064]

Very nice indeed!

[DZR_Mikhail 14]

Fighting with engine event right now. Engines now start correctly. I just need to turn them off reliably... BIS had broken engine commands for helicopters when released TOH.

Then vehicle firing is next. It's not working for vehicles like original function since I changed it. Trying to find out what causes it.

[.kju 3240]

One question regarding the pipe.

Why do you want to write the data realtime to the outside source?

You could save the data inside arma and dump it at the end/on request.

Atm the only benefit I can see is potential loss of data if the recording player/instance crashes meanwhile.

[DZR_Mikhail 14]

It was realtime. Every line, every bit was transferred to pipe rightaway.

Now its semi realtime, or, so to say, it's buffered. Arma updates it's huge array of recorded data. Parallel process reads this array line by line and dumps "outside" then erases the read record to save memory and the external app writes all it gets to a file.

If I don't clean up data in arma memory, then, probably, very likely, after some hour of recording the main array can get 600 mb... I don't think arma can handle such amount. And I don't think this monster can be easily processed.

And the second arguemtn is the one you named. This will be too bad if you get all replay lost just because of a stupid or undocumented bug, addon error or some other crash.

[DZR_Mikhail 14]

Ok. Good progress.

Vehicles now are firing the weapon used at recording as in original function.

Vehicle engine "turn off\off" event captured.

Still have some other issues:

There's some chance engine event won't be captured if it happened too close to the start or end of recording. But it's not affecting anything but aesthetics :)

When unit is in the vehicle his firing is cought in two variants - 1) for his firing as a human 2) his vehicle firing. In gameplay these actions hadnled separately, but when recorded it's not that good. For example, as a Mi24 pilot you fire the 1-st weapon, cannon. All is ok. But when you switch to rockets and fire them, on the replay your chopper will fire rockets and cannon. I just need to implement the check for unit being in a vehicle and not capture it's firing, the same way function stops capturing it's movement when in vehicle.

Had no time to record demo :)

By the way, now, in the end of recordinggame is paused and you have choice to press F2 and exit with data being copied to clipboard or press F1 and you'll start playback of what you just have recorded for debug purposes. Then in the end of the playback all data is dumped to clipboard and you're ready to record again.

No need to restart game or reinitiate scipts.

I also had an idea of including one more option for the function. This option activated will make function not ask or interrupt user input or pause game when record finishes. It will lag for some time while putting the monster array to the clipboard, but then you just continue playing. You're able to alt-tab and save your data.

When I sort out the problem with my unit double firing in vehicle and implement weapon selection capture for other units I'll release the function.

[DZR_Mikhail 14]

Sickboy helped a lot with some difficult obstacles in my scripts. Thanks!

Double firing solved. Weapon selection solved. Now the only thing that needs to be fixed is the vehicle slot of units. It's implementation was bad and outdated and now it's broken and unit is always a driver. Playback became a bit jittery but pretty accurate. Dead units are recreated, destroyed vehicles are not tested, but should also be recreated. Overall code is polished a bit to be not so laggy on capture and playback start and stop due to multiplied hints.

So I will implement new way of detecting slots in vehicle and I also need to indent and comment all my code extensively for you to check it effectively. Then I will upload everything at dev-heaven repository. But I warn you some pre-alpha test so be patient :) Release is aimed to get your responses about everything and share ideas about how it can be improved.

After release I plan to add cursorTarget to capturing and doTarget for units in playback. This will make them at least try to hold wepon vertically close to source position. For player unit this will be like in early AAR, a red circle at the target.

There's also some untested behaviour for the situation when recording unit is killed inside vehicle. Sometimes camera just not attached and you cant scroll menu to go spectating :) But it's left for future.

[kremator 1064]

Sounding good mate !

[DZR_Mikhail 14]

Ok guyz. I haven't got time to pack a good archieve for a release, but I've created a project home, set up a repository and uploaded my development mission with current function versions. Feel free to risk and try it :)

[lightspeed_aust 681]

This is gr8 to see and something I would really like functioning for upcoming coop tournie

[rcdxph 10]

well， what is the progress?

[J-Guid 10]

@rcdxph - progress in progress ;) New Year is coming soon ;)

[Dawks 10]

Found the GWave very hard to read / use, so I stripped out the text from source and am trying to reformat it as a normal HTML doc ... so crufty, it'll take some work, but this is worth it!

AARs rock.

[DZR_Mikhail 14]

Tremblay, thanks for this. But the wave was used merely for development details discussion. It's not very handy or useful for users who want to try or read about my AAR. All videos are shared in this thread, all manuals and latest files are at project home on dev-heaven. Just in case you didn't know it :)

[lightspeed_aust 681]

how is this tool progressing?

is it suitable to use in current state to record replays of mp coop very limited use of vehicles?

or causing lag and stutter?

thanx

Lighty

[DZR_Mikhail 14]

Still no progress, sorry. I very much hope it all improves a lot in Arma3 with java and other engine features.