AI vs AI contest (How to evaluate AI effectiveness?)

In the old quest to better use AI, better modify it, better improve it... one needs to understand it of course! Before any solution the requirement is an accurate description of the problem.

tl;dr: we need tools to objectively evaluate AI performance given a set of circumstances, lets trial and error attempts at solutions and measure results.

In any case these efforts can only be judged to an extent: the specific context under which one comes up with such an assorted solution, still the modder might try to imagine the multiple situations but the most boring part will be testing them.

My proposal to face this: let's automate the process, let's remove the boring out of it, let's create a shared workflow which allows us to test our solutions in a sufficiently wide set of typical and relevant scenarios.

I made a demo video of what is possible, i have split it in two parts: (Sorry about my hesitating discourse, i have as much trouble picking suitable english words as AI picking threats, i'm rusty :P)

Sensors - what information can we extract from the engine right now, both provided directly through the scripting commands and with customized sensors.

FSMs (actuators) or the logic we can embed AI with to process available sensor information. I share some of my learning regarding fsms in this part.

To make an automated AI vs AI contest could help us collect the relevant information for us modders not only in the sense of which settings are better against which, but also with even settings on both sides allowing to evaluate the balance of some initial geograhic or other conditions, (ie created a mission with one objective... but at which distance from each side should we place the spawn point to keep the chalenge reasonable? from 200m out red loses 8 out of 10 times, from 100m out red loses 6 out of 10 == there you go!, rerun the tests with swaped sides... i am bored just thinking of it, just let the mission bake the results to you, after an hour or two the statistics should be prety solid)

Really, i have "serendipityoulsy" collected a lot of useful stuff doing some runs of this mission, initialy only interested in figuring how my covering script was hinting at good or bad behaviour from default AI, stuff that could have used for better effect in previous mods. I have been keeping a lot of notes from these multiple tests made while creating this, i am interested in compiling this in some sort of tutorial, i shall see if it all fits together nicely. Some suggestions/feature requests are online too towards BI, of which i'll shall make proper tickets, but:

• AI should not share nearTargets information in and immediate and 100% accurate fashion
  
• Possible BUG: nearTargets threat is also immediatly updated even when out of FOV (if and only if) it already exists in the nearTargets array (this could explain some exclamations)
  
• BI could let us define individual behaviour/combat modes independently of group (lets us escape temporarily from the inflexibity of lower engine impositions) - player leading AI groups have this functionality provided by engine
  
• AI perceived position of a target could be updated to a earlier but possible position as opposed to a position where no threat can be verifiably seen
  Temporary fix of the above, could be to let a script remove/"forget" a target in units db, as a feature could also have other productive uses
  

That's some awesome stuff. :) Such an amazingly interesting video.

This is awesome, Very informative videos!!

Any thoughts on sharing the test mission and scripts?

Nice tests gammadust! I was under the impression that custom FSMs weren't really useful, as the internal danger.fsm took over as soon as combat started. Can we now use custom FSMs for specific individual groups?

...

• AI should not share nearTargets information in and immediate and 100% accurate fashion
  

...

As far as I can see, since OFP the AI knowledge base is done on a group level, not individual. This means that if one AI spots an enemy, all the AI in the group know where that target is. In Arma2 OA, this lead to situations such as being shot by a unit without any line of site (through dense forest), but this seems harder to reproduce in Arma 3. It might have changed, I'll have to do some testing..

AI perceived position of a target could be updated to a earlier but possible position as opposed to a position where no threat can be verifiably seen

Someone please, please make a ticket of this. It hamstrings the AI in CQB more than most people know.

Very good, I was hoping somebody would do this. Much, much more useful than my text stuff.

Edit: Ideally you would run this in high speed on a headless client to avoid graphics rendering, find some clear yardsticks by which to measure the AI performance (spotting distances, did they fire inside or out of cover, where they killed in cover or out and which stance from where by what weapon/vehicle, etc, etc) and then put that up in a table to see what the averages are.

Individual tests produce literally zero significant results. I once did a test row where the AI did something fancy every time for ten or so tries, and after that never behaved that way again (under the same patch, after patches behaviour is usually affected again, voiding everything found before at least to a degree). After 100 tries in a controlled setting you can begin to estimate where the AI has drawbacks and where it is performing well. Player anecdotes are often extremely unreliable because everybody has different expectations, skills and playstyles, as well as often discrepant settings. I hope this will go somewhere as development progresses!

Looking forward to seeing more come from this

Excellent idea (subscribed)

Thanks for all the encouragement guys, i am confident this can be of value to all of us, i really want to see this through. (hopefully i will be able not to get ahead of myself in the process, it is so tempting the possibilities down the road, i'll try to keep it a modest effort)

Any thoughts on sharing the test mission and scripts?

It is the whole point. Though the mission is relatively tied to the coverage mod i was working on. For the whole thing to work the mission (accompaigning scripts mostly) should be provided free from any strings required by a mod. I'm pressing for the idea to enable any mod/ai settings be tested.

Nice tests gammadust! I was under the impression that custom FSMs weren't really useful, as the internal danger.fsm took over as soon as combat started. Can we now use custom FSMs for specific individual groups?

As far as i tested what really makes AI unresponsive is not danger fsm (though it can lead to that too - CanFire cause stopping unit) but it is mostly native FSM "directives", when the ai enters the cycle Provide Cover > Hide or Out > ... it gets uneasy to leave from there and also such "orders" retain control and have priority over anything i could throw at it from scripting (i tested simple doMove's and lower level moveTo's against it, to no avail) i even custom doFSMed some ai moves, in which case i found i was unable to interrupt the control but could otoh retain it myself for the duration of it's execution - at the expense of access to otherwise useful engine behaviour (nativeFSM).

dangerFSM runs in bursts, and only rarely loops (since it empties its queue - but it does loop when another cause triggers in the middle of the current execution), so for the most part one is fighting against nativeFSM (and it's obscurity).

dangerFSM custom log example:

["2917(39.006) | R1/L0 - START *(#1 - I[[0]])"
"2917(39.006) | R1/L0 - 0 EnemyDetected (P3) Until: 43.7611 (4.75515)"
"2919(39.034) | R1/L0 - REST"
"2921(39.059) | R1/L0 - END"
"3015(40.192) | R2/L0 - START (#1 - I[5])"
"3015(40.192) | R2/L0 - 7 Scream (P2) Until: 46.0652 (5.87318)"
"3017(40.218) | R2/L0 - REST"
"3019(40.243) | R2/L0 - END"
"3253(43.064) | R3/L0 - START *(#1 - I[[7]])"
"3253(43.064) | R3/L0 - 7 Scream (P2) Until: 48.1536 (5.08957)"
"3255(43.092) | R3/L0 - REST"
"3257(43.118) | R3/L0 - END"
"3279(43.387) | R4/L0 - START *(#1 - I[[5]])"
"3279(43.387) | R4/L0 - 5 DeadBodyGroup (P1) Until: 47.5814 (4.19444)"
"3281(43.415) | R4/L0 - REST"
"3283(43.441) | R4/L0 - END"
"3331(44.02) | R5/L0 - START (#1 - I[5])"
"3331(44.02) | R5/L0 - 5 DeadBodyGroup (P1) Until: 48.3517 (4.33167)"
"3333(44.045) | R5/L0 - REST"
"3335(44.069) | R5/L0 - END"
"3451(45.461) | R6/L0 - START *(#1 - I[[7]])"
"3451(45.461) | R6/L0 - 7 Scream (P2) Until: 51.4308 (5.96983)"
"3453(45.487) | R6/L0 - REST"
"3455(45.514) | R6/L0 - END"
"3503(46.093) | R7/L0 - START *(#1 - I[[5]])"
"3503(46.093) | R7/L0 - 5 DeadBodyGroup (P1) Until: 51.5176 (5.42464)"
"3505(46.119) | R7/L0 - REST"
"3507(46.144) | R7/L0 - END"
"3747(49.02) | R8/L0 - START *(#1 - I[[6]])"
"3747(49.02) | R8/L0 - 6 DeadBody (P1) Until: 54.2398 (5.21982)"
"3749(49.044) | R8/L0 - REST"
"3751(49.068) | R8/L0 - END"
"7589(52.168) | R9/L0 - START (#1 - I[5])"
"7589(52.168) | R9/L0 - 7 Scream (P2) Until: 57.4495 (5.28149)"
"7591(52.195) | R9/L0 - REST"
"7593(52.22) | R9/L0 - END"
"7621(52.557) | R10/L0 - START *(#1 - I[[5]])"
"7621(52.557) | R10/L0 - 5 DeadBodyGroup (P1) Until: 58.3213 (5.76434)"
"7623(52.583) | R10/L0 - REST"
"7625(52.608) | R10/L0 - END"
"7747(54.022) | R11/L0 - START *(#1 - I[[5]])"
"7747(54.022) | R11/L0 - 5 DeadBodyGroup (P1) Until: 59.2022 (5.18025)"
"7749(54.046) | R11/L0 - REST"
"7751(54.07) | R11/L0 - END"
"7931(56.149) | R12/L0 - START *(#1 - I[[8]])"
"7931(56.149) | R12/L0 - 8 CanFire (P4) Until: 60.9482 (4.79921)"
"7939(56.239) | R12/L0 - REST"
"7941(56.261) | R12/L0 - END"
"7981(56.711) | R13/L0 - START (#1 - I[6])"
"7981(56.711) | R13/L0 - 6 DeadBody (P1) Until: 60.7485 (4.03748)"
"7983(56.734) | R13/L0 - REST"
"7985(56.756) | R13/L0 - END"
"8011(57.059) | R14/L0 - START (#1 - I[6])"
"8011(57.059) | R14/L0 - 6 DeadBody (P1) Until: 62.7845 (5.72548)"
"8013(57.083) | R14/L0 - REST"
"8015(57.106) | R14/L0 - END"
"10032(57.914) | R15/L0 - START *(#1 - I[[6]])"
"10032(57.914) | R15/L0 - 6 DeadBody (P1) Until: 63.1856 (5.27159)"
"10034(57.937) | R15/L0 - REST"
"10036(57.96) | R15/L0 - END"
"10046(58.078) | R16/L0 - START (#3 - I[4,7,5])"
"10046(58.078) | R16/L0 - 4 Explosion (P4) Until: 62.6137 (4.53567)"
"10048(58.102) | R16/L0 - REST"
"10050(58.125) | R16/L0 - END"
"10060(58.242) | R17/L0 - START *(#1 - I[[6]])"
"10060(58.242) | R17/L0 - 6 DeadBody (P1) Until: 62.7712 (4.52921)"
"10062(58.266) | R17/L0 - REST"
"10064(58.29) | R17/L0 - END"
"10090(58.602) | R18/L0 - START (#3 - I[3,3,3])"
"10090(58.602) | R18/L0 - 3 EnemyNear (P1) Until: 63.3846 (4.78256)"
"10092(58.626) | R18/L0 - REST"
"10094(58.65) | R18/L0 - END"
"10186(59.713) | R19/L0 - START *(#1 - I[[8]])"
"10186(59.713) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (4.03107)"
"10334(61.484) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (2.26007)"
"10342(61.585) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (2.15907)"
"10350(61.687) | R19/L0 - REST"
"10352(61.711) | R19/L1 - START (#3 - I[5,6,6])"
"10354(61.735) | R19/L1 - 8 CanFire (P4) Until: 63.7441 (2.00907)"
"10362(61.831) | R19/L1 - REST"
"10364(61.855) | R19/L1 - END"
"10420(62.583) | R20/L0 - START *(#1 - I[[6]])"
"10420(62.583) | R20/L0 - 6 DeadBody (P1) Until: 66.8024 (4.21944)"
"10422(62.611) | R20/L0 - REST"
"10424(62.639) | R20/L0 - END"
"10572(64.985) | R21/L0 - START *(#1 - I[[8]])"
"10572(64.985) | R21/L0 - 8 CanFire (P4) Until: 70.279 (5.29395)"
"10622(65.785) | R21/L0 - REST"
"10624(65.817) | R21/L0 - END"
"10876(69.849) | R22/L0 - START *(#1 - I[[4]])"
"10876(69.849) | R22/L0 - 4 Explosion (P4) Until: 75.0437 (5.19466)"]

"10186(59.713) | R19/L0 - START *(#1 - I[[8]])"
"10186(59.713) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (4.03107)"
"10334(61.484) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (2.26007)"
"10342(61.585) | R19/L0 - 8 CanFire (P4) Until: 63.7441 (2.15907)"
"10350(61.687) | R19/L0 - REST"
"10352(61.711) | R19/L1 - START (#3 - I[5,6,6])"
"10354(61.735) | R19/L1 - 8 CanFire (P4) Until: 63.7441 (2.00907)"
"10362(61.831) | R19/L1 - REST"
"10364(61.855) | R19/L1 - END"

Can we set danger for specific groups? afaik only through config can we set danger.fsm and customize to which units/groups it will be assigned, running doFSM or execFSM, will run parallel to danger (i believe), besides we won't get it triggered with the useful variables that are set with it (_dangercause, _dangeruntil ...) since we'll be triggering it ourselves (haven't explored the usage of set/getFSMVariable for that matter - and we don't have a "handle" for the running default).

In short it is possible, but it not in a dynamic way, afaik.

As far as I can see, since OFP the AI knowledge base is done on a group level, not individual. This means that if one AI spots an enemy, all the AI in the group know where that target is. In Arma2 OA, this lead to situations such as being shot by a unit without any line of site (through dense forest), but this seems harder to reproduce in Arma 3. It might have changed, I'll have to do some testing..

I confirm this, i just don't know to which extent of conditions that goes, is there a distance limit, does the information degrade progressively or simply ceases to be shared... some of the runs of the mission i made could tell us that but i was just paying attention to other stuff. But this is definitely something to explore.

Edit: Ideally you would run this in high speed on a headless client to avoid graphics rendering, find some clear yardsticks by which to measure the AI performance (spotting distances, did they fire inside or out of cover, where they killed in cover or out and which stance from where by what weapon/vehicle, etc, etc) and then put that up in a table to see what the averages are.

(...)

This is a great suggestion, and the most relevant one imo being not just the load decrease on the test machine... the coverage script was pressing the cpu a lot when i had it (unecessarily) running every frame for every units on map for my own visual candy. Observer effect rings a bell here.

Someone please, please make a ticket of this. It hamstrings the AI in CQB more than most people know.

I've just been trying to track any existing ticket of this issue, i found:

AI knows the enemy position at all times, no matter what

AI has to high awareness and too high accuracy at long ranges

AI can see and shoot right through objects that completely obscure them

AI Still see's player through dense vegetation

AI OpFor can spot you too easily

Dev feedback on those tickets hint at a usage of the nearTargets extrapolated position in sophisticated manner, maybe invalidating the worries:

The AI is planning path in bigger "circles" in which is looking for the player. Which is correct - AI tries harder to resolve the threat. On lower difficulties they went to the last known position[*] of me(player) and when they didn¨t find anything, leader called them back.

This seems to me as correct behaviour. I'm not sure what happened on the video but the troopmon seems a bit unreliable or maybe a possibility that Ai planned bigger circle and accidentaly saw you. But that seems unlikely since they started shooting before they could actually see you.

and

I don't think programmers made any additional changes in newer builds. I've been testing this issue today and couldn't find anything wrong. Take note that AIs aren't always going straight to the last position - they're trying to flank you. For example one group can go from one side of the hill and second from the other. So they can discover you, if you're in their way.

[*] is this the same position as in nearTargets? I do think it is, but ghost hints at more subtle usage of the information ("The AI is planning path in bigger "circles" in which is looking for the player"), which I totally confirm because i've seen this as represented by the {expectedDestination} spheres of the subordinates in a group move from place to place at a very fast rate. This "circle" is actually "spiralling" out from the suspected position until it finds the target or quits searching.

The consequences of the use of the position or the corresponding behaviour is something i would keep my eyes open in next runs of the mission (yes keeping in mind the case of CQB)

Those are separate issues, Gammadust.

I was referring to the problem whereby the AI will pivot 90 degrees to target a player's perceived/predicted position when he runs behind cover, aiming and shooting at the empty air even when their eyes should see that the enemy is still hidden behind the obstacle, having halted.

Very interesting stuff!

I've been trying to monitor AI behavior too, but this is very nice graphical display!

Quite useful to examine and I'm really curious to see the impact of e.g. tpwcas

I've also been looking at AI target knowledge/ acquiring capabilities about a month ago, to get a clear view if AI are able to see trough fog, trees, bushes and grass, yes or no?

I've used a script, which if a player is detected by AI, will deploy red smoke at the place the AI believes the player is located.

In case there is no line of sight, the AI uses a perceived player position, in combination with a position accuracy value (radius in meters from this perceived position).

So the red smoke will indicate the (perceived) position, and the bigger the accuracy radius, the bigger the smoke circle.

https://dl.dropboxusercontent.com/u/96469595/reveal/arma3%202013-07-03%2020-33-14-13.jpg

A) - Are AI able to see trough trees, bushes and grass?

The answer is a clear NO!

A) Details

I also checked if moonlight and fog has an effect on AI spotting targets capabilities?

The answer is a clear YES!

B) Light conditions

@ollem

I was absolutely sure i was not alone in getting deeper into the AI shenanigans... it is overwhelming the amount of conditions stuff may or may not happen. Sometimes to our general pleasure other times to our frustration. Maybe this thread will be able to shine some light into these more obscure conditionings. If we can settle all of this information such as the one you collected, i think the quality of AI mods will jump. I am very keen for this to happen.

Edit:

Those are separate issues, Gammadust.

I was referring to the problem whereby the AI will pivot 90 degrees to target a player's perceived/predicted position when he runs behind cover, aiming and shooting at the empty air even when their eyes should see that the enemy is still hidden behind the obstacle, having halted.

No doubt those were different issues, or could it be a maniphestation of the same base engine particularity. Sometimes trying to reproduce a issue 100% of the times is not easy, or better yet: coming up with a scenario that can.

I've just created the ticket 0013229: Ability to individualize setBehaviour type commands to group members

Vote for it if you agree with it or improve it at will please.

EDIT:

Also please attend these other tickets since they can become quite some showstoppers:

0012669: BIS_fnc_spawnGroup creates AI with skill 1 by default

and the base issue originating the above:

0012695: createUnit (array) creates AI with skill 1 by default

I believe i was able to reproduce another wierd detection behaviour, which before going on a ticket spree, would like your input, maybe i am looking at this wrong. This is still not what maturin mentions but might also be related.

Edit: @InstaGoat

They kind of already do this. This is especially obvious at the edge of their detection range: they extrapolate their targets position based on the last movement, after they moved out of sight (for example, moved behind cover). They don´t anticipate change in direction of movement, however, and as soon as you reappear, they instantly see you again: they never "guess wrong", so to speak. They have perfect eyesight once they´re aware of you.

Making the AI more human is a difficult feat to do. I hope BI will show some improvements at E3: so far we´ve barely seen anything but promises.

Is this a faithful reproduction of what you meant here about a year ago?

I don't think we can call that a problem unless we see units 'second-time-acquiring' enemies from behind, or out of the normal human range of peripheral vission.

gammadust, the AI FOV in the test mission looks like 90 degrees.

I moved the destination waypoint of the second OPFOR unit (west, detected via hearing) a bit further north, the BLUFOR detecting AI will turn further towards the north, so that the first OPFOR unit (south, detected by sight) is further from the centre of vision. Then, the first AI is not located at all during it's first run past. This suggest the AI FOV is greater than the 90 degree FOV indicated by the red lines.

From my own experience in ArmA 2, I don't think AI FOV always matters.

We all know that the AI changes detection routines based on their awareness state, going from probability-based systems where a unit's camouflage matters, and switching to pure LoS checks that ignore grass when in combat.

I have seen units detect enemies from around their 3 or 9 o'clock. I have also seen them suffer from tunnel vision that ignores enemies at 2 o'clock.

FOV is sometimes ignored, it seems.

Gamma, the work you've done is fantastic. Actually seeing how the AI operates makes diagnosing problems exponentially easier. I wish more people could see the work you've done. Have you considered posting this elsewhere to raise other peoples awareness? I was thinking the AI thread here.

Edit: Oops, I see you've been posting there already, its odd that I've only come across your work in this thread by accident. I'm sure others in the AI thread have also missed your work and would appreciate seeing it.

I don't think we can call that a problem unless we see units 'second-time-acquiring' enemies from behind, or out of the normal human range of peripheral vission.

From my own experience in ArmA 2, I don't think AI FOV always matters.

We all know that the AI changes detection routines based on their awareness state, going from probability-based systems where a unit's camouflage matters, and switching to pure LoS checks that ignore grass when in combat.

I have seen units detect enemies from around their 3 or 9 o'clock. I have also seen them suffer from tunnel vision that ignores enemies at 2 o'clock.

FOV is sometimes ignored, it seems.

^^ i tend to agree with this latter part, in the repro above the blufor unit prioritizes too much the 1st enemy detection, once it goes out of sight and detects the 2nd opfor AI (west side) it should dismiss the 1st detection (even if temporarily only) and turn it's full attention to seen enemy. It was striken by "tunnel vision" as you put it. (I just used that 2nd unit exacly under the assumption that i would be able to make the blufor look north in order to make it obvious the detection of the initial enemy on it's 6 o'clock - not too good of an effort if it wasn't for this additional awkward behaviour :/)

"FOV is sometimes ignored, it seems." - I am feeling a need to describe the conditions under which this happens in more detail. And if and when a "pure LoS" enters into play.

gammadust, the AI FOV in the test mission looks like 90 degrees.

I moved the destination waypoint of the second OPFOR unit (west, detected via hearing) a bit further north, the BLUFOR detecting AI will turn further towards the north, so that the first OPFOR unit (south, detected by sight) is further from the centre of vision. Then, the first AI is not located at all during it's first run past. This suggest the AI FOV is greater than the 90 degree FOV indicated by the red lines.

I should be uploading a video regarding the FOV for the first encounter at least. But i can advance that i already got several detections above the 45 degrees to the side. Also i am keeping the units with regular skill settings and assuming (according to tests) that both spotDistance and spotTime don't influence the FOV. I whish to have a fairly representative number to include in the base sensor for the mission.

Gamma, the work you've done is fantastic. Actually seeing how the AI operates makes diagnosing problems exponentially easier. I wish more people could see the work you've done. Have you considered posting this elsewhere to raise other peoples awareness? I was thinking the AI thread here.

Edit: Oops, I see you've been posting there already, its odd that I've only come across your work in this thread by accident. I'm sure others in the AI thread have also missed your work and would appreciate seeing it.

Thank you for your kind words. Right now the mission is in an initial state of development, there will be some basic information that i whish to get as accurate as possible, hence the time i spent in the tracker lately, trying to see how already reported issues might influence the objective of this effort. I had to catch up with some stuff indeed.

I am not making a lot of pomp out of this right now since it has most interest to modders and scripters, more than the general player. I am trying to drive this more with the objective of facilitating an understanding of AI depths and it's modding than a blunt tool to facilitate finding bugs, though it could be use for that effect too.

"FOV is sometimes ignored, it seems." - I am feeling a need to describe the conditions under which this happens in more detail. And if and when a "pure LoS" enters into play.

More power to you, but that could be very difficult. Maybe an inquisitive PM to Suma or Dr Hladik would be worth a shot.

Well here is the video with the detection fov tests. It is still inconclusive enough, but I will adjust the mission with a more tested value when able.

More power to you, but that could be very difficult. Maybe an inquisitive PM to Suma or Dr Hladik would be worth a shot.

I shall try if the mission becames too dependent on this.

You could use disableai "move" to prevent head movement, I suppose.

Are you going to test this with varying levels of alarm state, as well?

^^ i intend too, but at the same time do not whish to delay too much the progress with the main contest mission, this is a value that i can easily update later on.

Awesome work Gamma, gonna follow this one for sure!

/KC