How can we find out how good ArmA AI is?

[walker 0]

Hi all

There are frequent threads in the forums asking how good the ArmA AI is as well as, others saying the ArmA AI is stupid or the how brilliant they are.

Sadly all answers so far are subjective.

What we need to do is objectively establish how bright the AI is against humans.

To do this we need an experiment!

First of all we need to decide what we can reasonably expect to experiment on. to kick it off I suggest this procedure then the community can refine it. We can do all this with the mission editor.

We are looking for some metrics to define how well the AI does against:

1) A single human player

2) A single human controlling a Five and Four entity AI fire team

3) A human Five and Four player fire team

4) A human thirteen player squad fire teams of 5,4,4

5) A human platoon three squads

6) A mechanised human platoon, 3 x IFVs and 3 x 6 player infantry squads

7) A tank platoon of 4 MBTs

We repeat with both BLUEFOR and REDFOR averaging the two results in order exclude weapon factors.

We will need a dialogue to copy and paste the metrics out as a comma delineated file or dump it to the error log. Then we just copy and paste it into a spread sheet or database.

To get the metrics arbitrary I suggest we initiate the missions at 1 to 1.

We then run the missions a sequence of say 3 to 5 times via base respawn and return the percentage of AI killed. We then iterate the sequences by increasing the numbers of AI until say less than 30% of the AI are killed in a sequence before the humans are defeated.

The more people who can run the experiments the more statistically significant the figures will be; as other factors such as the power of persons PC can have an effect on how well the AI does.

Anyway this is the point of a forum, it would be nice if others contributed as to what the experiment should involve and how we should conduct it.

Kind Regards walker

[kylania 568]

AI Skill Level, Difficulty Based Skill Modifiers, CombatMode, Waypoint Types, Weapon Types, Target Types, Time of Day are all some things to take into consideration as well.

[Decerto 10]

It depends on terrain as well. AI squads will sometimes try and bound across roads causing half to lie down for 'cover fire' while the other half run across even if there is no enemy in site.

[DaSquade 0]

Don't forget terrain (urban/woodland/desert), stance & cover...

Also computer spec + settings will make a difference, if not mistaken AI calculation is done by the CPU (one of the X amount of cores)....

All thingy items that can influence the end result.

And maybe most of all....repeatability. Meaning, especially in case of human player versus AI....will AI make the same dissicions each time you play the same test mission + how will AI react on different manouvers.

Not my cup of tea to help in this, but i'm sure AI by default is pretty 'stupid' or at least predictable. I think that is a bit the beauty about it as it all lays in the hands of missionmakers to add additional scripts (grouplink to name something) or elements of suprise.

I think most people are most pick about AI inteligence with simple things and expect AI to be smart enough, like simply ordering an unit to go to location X under danger, board that vehicle and drive to, heal that guy, pick up weapons etc...without having to babysit and follow each move they make.

[STALKERGB 6]

Also computer spec + settings will make a difference, if not mistaken AI calculation is done by the CPU (one of the X amount of cores)....

YEah the AI is far more "clever" when a system is running smoothly, if you run the game in slow-mo "SetAccTime 0.1" the AI seem more clever too, I guess the game has longer to think between each action.

EDIT: Thinking about it, I have a mission set up so that the camera follows my character while he does the "c7a_longwalk" animation, behind him I have the USMC and Russians fighting and almost every time something different happens, different stuff gets shot down, different men killed, different tank positions. Infact when I ran it once, the USMC pushed forward with their M1A1's right at the start, hadn't seen that before.

Unfortunately all of this is anecdotal, so can't really be quantified in any way.

Edited August 31, 2010 by STALKERGB

[Txheat 10]

Actually i got a great Idea find any othor game thats AI unscripted could beat ARMA2s AI unscripted. That would be a great guage, I don't think too many people will complain after that.

[-)rStrangelove 0]

It's difficult trying to judge whether an AI is 'good' (how good is good?) or not.

Imo an AI is good when its adapting to different situations and manages what it's supposed to archieve. And it must be 'good enough' and not perfect. Why?

Because we expect our enemies to make mistakes at times. A perfect enemy is unrealistic. Humans get tired / exhausted and make mistakes when under stress (covering fire etc). The AI is supposed to simulate a human enemy or in general to provide the player with a good show - so the AI has to make mistakes.

So i'd just watch the AI what / how it does with the typical ArmA situations:

POSITIONS

1. Attacker(A) and Defender(D) (same vehicle type) both in flat terrain

2. A high up/on a rooftop, D in flat terrain

3. A inside a building, D in the open

4. A and D in different buildings

5. A and D in the same building

WEAPON TYPES

All combinations of A/D having Infantry / SF soldiers / Groundvehicle / Tank / ARI/Mortar / Heli / Plane. Who engages who and at what distances.

SPOTTING / DISTANCES

The old knowsabout feature again. Maybe settings changed again, needs to be remeasured. Has an effect on the first 'POSITIONS' tests too.