Darwinian Evolution of AI skills

[TRexian 0]

Darwinian Evolution Experiment for AI

It sounds like this is a new mission or something, and it is, if you like AI battles as spectator sports. It started out as an experiment within JTD to use ArmA/ArmA2 to mimic evolutionary dynamics for AI. The premise was to start a group of AI with a low set of skills (within a narrow random range), have an evolutionary iteration where they fight, then when there is a "winner," use the skill values of those units to establish the skill values (within a narrow random range) for the next iteration.

(I have mentioned this experiment in various other posts around the 'net, so this may be familiar to some of you.)

Premise

In typical Darwinian fashion, the strong will survive and the weak will perish.

Experiment description

We settled on the current rubric, which I consider to be basically a beta stage.

- 100 AK-wielding insurgents v. 100 AK-wielding CDF soldiers in an open area at the Utes airstrip

- each has a MOVE waypoint at the other's spawn point, roughly 200m apart

- each unit has a starting value of ~.2 in each skill (more on this later)

- each battle is an iteration, and the battle is over when either side has 10 or fewer units remaining

- each assessment is 30 iterations (more on that later)

- the remaining units are filtered, such that only the ones that have points (that is, engaged in the battle) are measured

- each skill of each filtered unit is measured, then averaged among all the measured units

- that average skill is then used as the base, with a narrow random modifier (+/- .1), for each unit in the next iteration

- each spawned unit is a skill-set twin; that is, for each blue unit, there is an identically skilled red unit

- the groups switch sides after each iteration, to remove any terrain advantage

- the various skills are logged to the rpt

Measuring the results

At the end of each iteration, there is a section of the rpt that looks like this:

Iteration: 5
red: 2
blue: 30
Shooters: 32
Accuracy: 0.244765
Shake: 0.263234
AimSpeed: 0.26531
Endurance: 0.185303
SpotDist: 0.231191
SpotTime: 0.198323
Courage: 0.206779
ReloadSpeed: 0.216462
Command: 0.22059
General: 0.219461
Time: 9.18514

(BTW, note to BI - can you PLEASE address the rpt spamming that results from not having voices assigned?!? Especially when they are immediately mapped to "Dan," so it really isn't necessary to log that in the rpt?)

That portion of the rpt is then manually copied and pasted into an OpenOffice Calc spreadsheet and graphed.

The result looks something like this:

Larger version:

http://i26.photobucket.com/albums/c114/TRexian/JTD/testing/Assessment10_1_data1.jpg

First Full Assessment

The first full assessment was accomplished using the latest beta. I cannot tell if it was the new beta, or just better scripting on my part that allowed me to overcome an issue with the 144 group limit, but the important part is that it was overcome. (However, more on that later.)

Here are the graphing results from the first full iteration. Interestingly, they are consistent with some earlier tests I did that were limited to 13 iterations.

Larger:

http://i26.photobucket.com/albums/c114/TRexian/JTD/testing/Assessment10_1_chart1.jpg

Larger:

http://i26.photobucket.com/albums/c114/TRexian/JTD/testing/Assessment10_1_chart2.jpg

Larger:

http://i26.photobucket.com/albums/c114/TRexian/JTD/testing/Assessment10_1_chart3.jpg

Larger:

http://i26.photobucket.com/albums/c114/TRexian/JTD/testing/Assessment10_1_chart4.jpg

Mission pack:

This zip contains the mission I used, plus the ods template, and the filled in spreadsheet for this assessment, if anyone is interested.

*** download removed, as I found a goofy error that doesn't really change the data but is a bit embarrassing, plus there will likely be a new version that is more configurable, relatively soon ***

N.B.

1) There remains an issue with spawning units after approximately 15 iterations. The spawning slows down considerably. The units appear to be assigned appropriately to valid groups, but the whole thing slows down. This can introduce some measurement issues, in that there can be an artificial bias towards the numbers of the last few groups, as they are more likely to be alive at the end. It also explains the larger jumps in the duration of the iterations toward the end. However, there is still evidence that higher skill values result in longer battles.

2) I think this testing scenario is biased against endurance, courage and command, which might somewhat explain the falling numbers. The forces start out just a couple hundred meters apart, in relatively open view of each other. So, endurance is not really a requirement. It takes the measurements of the last surviving units. Units with low courage are more likely to survive. And, there isn't alot of reward for a well-commanded unit.

3) IMHO, this testing supports an inference of the relative merit of the various skills for AI. Higher aimingShake values is something that the AI v AI environment rewards.

4) While this may represent a basic test of Darwinian evolution, I'm not sure that one can automatically extrapolate that certain skills are better for gameplay. That is up to AI addon-makers. My hope is that those guys can use the data (or share the data they are using), to get closer to the goal of "better" AI.

5) In this first full assessment, Blue inexplicably had an advantage. In prior testing, the final battle-results were more balanced.

Plans

I intend to further design this mission to allow the user to set the starting values of the skills, and to "lock" in a value for each skill. Statistically, this will provide a better picture for how the skills inter-relate.

I basically thought some other folks would be interested in this, so figured I'd post it!

Edited June 1, 2010 by TRexian

[kremator 1065]

VERY interesting. Thanks for sharing !

[Fox '09 14]

good read, thanks for the effort :)

[jeza 5416]

That sent me back to school got a headache now :P , nice work JTD follow this with interest on the DL now. Cheers

[Jelliz 10]

[eggbeast 3684]

awesome thinking here mate. i did my degree in evolutionary biology and particularly in GA's in spatially separated population systems. it never occured to me to use GA's in arma.

downloaded and will look through all this with relish

cheers

---------- Post added at 06:45 PM ---------- Previous post was at 06:42 PM ----------

um did you haveto cut and paste from the RPT? or did you write a script to dump the data to csv?

[TRexian 0]

hehe

I cut and paste manually. :D

Part of the problem is that the rpt is (semi-literally) spammed with BI errors. For my purposes, it is mostly something like:

No speaker given for Pavel Nohavica
No speaker given, patched to Dan
No speaker given for Pavel Nohavica
No speaker given, patched to Dan
No speaker given for Filip Medvedev
No speaker given, patched to Dan
No speaker given for Otakar Jurcik
No speaker given, patched to Dan
No speaker given for Vyacheslav Litvinov
No speaker given, patched to Dan
No speaker given for Vladimir Rybski
No speaker given, patched to Dan
No speaker given for Filip Kamenev
No speaker given, patched to Dan
No speaker given for Miroslav Homolka
No speaker given, patched to Dan
No speaker given for Konstantin Pushkin
No speaker given, patched to Dan
No speaker given for Boris Skopek
No speaker given, patched to Dan
No speaker given for Pyotr Alekseev
No speaker given, patched to Dan
No speaker given for Jaromir Horvat
No speaker given, patched to Dan
No speaker given for Valery Alekseev
No speaker given, patched to Dan
No speaker given for Jiri Bednar
No speaker given, patched to Dan
No speaker given for Fedor Lermontov
No speaker given, patched to Dan
No speaker given for Matous Prutko
No speaker given, patched to Dan
No speaker given for Mikhail Beregovoi
No speaker given, patched to Dan

I have thought about writing a script that will make it csv, but it would take ALOT of manual effort to match the effort to make an app with the right filters, etc. :) Plus, I've got it down to about a 10 second process of copying from rpt, pasting into OOC, and filtering it there so it pastes correctly.

I'm open to ideas on making it more efficient, though!

And, how freaking awesome to actually get the attention of an evolutionary biologist! :lol: I'm not sure if this really reflects anything Darwinian, but it sounded good. My background is in Political Science, and statistics related to it, but that goes back a LONG time. It seems to me, though, that there's some sort of survival-of-the-fittest at work, otherwise all the numbers would basically congregate around the initial setting.

Edit:

Also, I can upload the "raw" rpt and a "filtered" one (I manually cut out the crap, just to have a clean version). I can also post the init, or just the "mechanical" parts of it, if anyone is interested.

Edited June 1, 2010 by TRexian

[[asa]oden 0]

You've outdone yourself here TRex :D

I unleash an insane amount of applauds for you.

(and hell yeah, pls BIS look into the voice rpt spammage)

[TRexian 0]

hehe

Thanks, AO, but this is really just taking ArmA2 and converting it into a statistics simulator, for math geeks. ;)

Hell, no one really knows if this will even help anyone improve anything! :D Save the applause for people who might be able to use this kind of thing!

[comradechaos 10]

Not to sound uneducated, but what exactly does it apply to Arma? I feel this could be an extreme breakthrough though, but I just don't get it.

[TRexian 0]

hehe

That, my friend, is a fair question. :)

Basically, this can document what AI skills are "more important." My idea was that it might allow AI addon makers, and even mission-designers, to better understand the changes they make.

For instance, in these early tests, it appears that the AI skill "aimingShake" is probably the MOST important skill for them to be effective. That was surprising to me, but it does kinda make some sense. AI that shake less (who have a higher value for that skill) acquire their target more easily, and hit it more often.

It is too soon to draw too many conclusions, but that is an initial observation.

One of the next steps I am planning is to be able to isolate any skill. For instance, if I lock aimingShake at .5, so that all AI have that skill level, what impact will that have on accuracy, or spotTime?

I am also planning on being able to start the skills at different levels. So, if I start accuracy at .8, but aimingShake at .2, will the accuracy fall, or remain steady as the aimingShake increases?

This is probably not going to be "revolutionary" or even "evolutionary" (pardon the pun) ;) itself, but may be a tool to help community devs understand the system better, so as to produce better AI mods. (I'm hoping BI already understands the system.) ;) Right now, this is basically more informational than anything else.

[dmarkwick 261]

This basically came out of a conversation we (myself & TRexian) had about what mixes of attributes might make the best, or most effective AI opponents. We can draw obvious conclusions about accuracy & endurance etc, but we wondered what other mixes would surprise us, maybe even some counter-intuitive ones. Maybe troops with less courage & endurance might not move toward enemy as much, and thus survive for longer, and out-shoot more tired enemy. Stuff like that.

So TRexian devised this experiment where AI troops can engage in battles with different mixes of attributes, and the survivors of those battles get rewarded by having their attributes form the base for the next round's AI attributes. It requires a long time to run, and TRexian ran into some problems with the ArmA2 engine clogging up once a certain number of AI had lived, died & respawned. Only recently has the newest beta release overcome this problem, and so TRexian releases his code :)

Hopefully the experiment will eventually reveal what sort of mix is best suited for battlefield survival. I think other more complex scenarios would be needed to find out other important traits, like kill ratios and goal accomplishments etc.

[_William 0]

This is probably not going to be "revolutionary" or even "evolutionary" (pardon the pun) ;) itself, but may be a tool to help community devs understand the system better, so as to produce better AI mods. (I'm hoping BI already understands the system.) ;) Right now, this is basically more informational than anything else.

Great step and interesting stuff. One thing that worries me a bit is that you measure increased performance against an (co)-evolving opponent rather than the default AI. Probably gets you a good enough result, but keep this in mind when drawing conclusions.

Personally, for game play, I don't think skill settings will do as much for better Arma AI as good basic behaviors. One of the things lacking is good 'facing behavior' notably for armored vehicles; even with known threats present, vehicles tend to stop and expose their weakers sides or backs to the threats rather than making sure their front is are facing the threats.

Another missing basic capability is fighting from an the appropriate distance for the weapons at hand: snipers, Javelin soldiers and gunships all tend to close with the threat rather than stay at a distance and use their long-range attack capability.

Hopefully the experiment will eventually reveal what sort of mix is best suited for battlefield survival. I think other more complex scenarios would be needed to find out other important traits, like kill ratios and goal accomplishments etc.

I'm not sure it is of help, but I'm able to generate large battles in a consistent way with my mission generator (see sig). Let me know if there's something I can do.

[TRexian 0]

First off, indeed, it was really DM's idea. :D One of those, "Hey, wouldn't it be cool if...." and then he gets me to actually do it. ;)

Great step and interesting stuff. One thing that worries me a bit is that you measure increased performance against an (co)-evolving opponent rather than the default AI. Probably gets you a good enough result, but keep this in mind when drawing conclusions.

Indeed, that was a conscious choice, elevating both sides. I may create a way to change that - have side-specific settings - but I felt like that would also introduce a complacency of sorts. After a certain point, the Darwinian AI would just beat the crap out of the default, and there'd be little "upward" pressure.

Now, I *did* consider making one side super-AI and the other Darwinian, and only taking the scores of the Darwinian survivors. But, I figured that would reward low courage scores.

Minor update, also. I'm having some good luck with the user setting/locking the various parameters. It seems to be working on a small scale. I need to have it run several iterations, though, before I'm completely comfortable. Probably have the new version in a few days.

I appreciate the feedback and peer-review type criticism though! The point about being careful not to draw gameplay conclusions is well taken, too!

With regard to the armor/vehicle situation, I completely understand, but that is also beyond what I'm prepared to work on at this point. Although, that could be the testing scenario for the command skill.

[TRexian 0]

Ok, new upload of the mission:

http://www.mediafire.com/file/qgeqeinmkzy/JTD_DarwinsFitnessPack02.zip

This is more complicated than the original, so there may be some bugs. :) But, so far it looks like it is both consistent with the original, and has additional capabilities. So much so that it now has a Readme. :)

JTD_DarwinsFitness mission readme.

Zip contents:

- JTD_DarwinsFitness.Utes.pbo - default mission

- init.sqf - an init script which can be dropped into any mission to produce the same mission

- assessmentTemplate.ods - OpenOffice Calc spreadsheet that can serve as a template for importing and assessment of data.

- this readme

This mission is an attempt to simulate Darwinian evolution in ArmA2. It writes various skill settings to the rpt, which can then be imported into a spreadsheet program for analysis. (This zip contains an OpenOffice Calc spreadsheet to serve as a template.)

The contained mission is on the airfield at Utes. The contained init can be used in any mission that contains the following:

- markers named "BlueStart" and "RedStart" that will serve as the spawn points and objectives of the AI

- a player that is not one of the combatants

The mission maker can set global variables in the init of the player unit to control various settings. The default starting skill values are .2, but they can be changed via:

JTD_skAcc

JTD_skShake

JTD_skAimSpeed

JTD_skEndurance

JTD_skSpotDist

JTD_skSpotTime

JTD_skCourage

JTD_skReloadSpeed

JTD_skCommand

JTD_skGeneral

For example, placing this in the init line of the player unit:

JTD_skAcc = .5

Will set the starting accuracy skill value at .5.

The following non-skill variables can also be set:

JTD_fleeVal = the allowFleeing value

JTD_unitW = classname for West unit*

JTD_unitE = classname for East unit*

JTD_mutRate = "mutation" rate; variable that describes range for random change to settings between iterations

* All units of each side will be the "same" unit class. Each individual unit will have different skills, though.

To lock a skill variable at a certain level, set the following variables to true to lock that skill. This is best used if the skill is first set to a certain level by the above commands.

JTD_skAccLock

JTD_skShakeLock

JTD_skAimSpeedLock

JTD_skEnduranceLock

JTD_skSpotDistLock

JTD_skSpotTimeLock

JTD_skCourageLock

JTD_skReloadSpeedLock

JTD_skCommandLock

JTD_skGeneralLock

For instance, to lock the accuracy value at .8:

JTD_skAcc = .8; JTD_skAccLock = true

And it will be set .8, and that will remain constant with each iteration.

[Uziyahu--IDF 0]

If "aimingshake" is the thing that makes it hard to aim when scoped-in and suppressed, then yeah, this is the thing that makes it most difficult for me to survive an engagement.

(I understand that you are talking about AI Vs. AI.)

[paragraphic l 2]

This tests the individual AI-soldiers skill right? No groups, leaders, support, specialists?

I understand this is a first step in researching the perfect balance for an AI soldier, but do you think it will be possible to create preset fire-teams with defined skills per member and check which has te biggest succes-rate in 5 different scenarios for example?

But I love the idea behind this and the way you've set it up, hope this will evolve into a community experiment

[eggbeast 3684]

i'm wondering how formation and combatmode (and to go further - changes to combatmode during the engagement) affect team survival time and total enemy elimination time...

these are probably the key to a good AI mod.

haven't had a chance to look through the basic GA yet, but I will I promise.

There are some basics to get right in GA work. Like inheritance process. It's usually sensible to add in some noise rather than exact step mutation, so so your 0.1 variation is a good move. It might be better to have a range of variation though that is randomised individually, say 0.08-0.12 just to avoid any eccentricities created by making uniform changes in the population.

Also the start time and start conditions often affect a model disproportionately (e.g. the coreolis affect where you can see water going opposite ways round in a plughole is only visible in extremely still tanks of water, general bath users will create so much noise in the water that the affect of the earths spin will be erased). It's worth looking closely at anything that affects starting conditions. e.g. you may want to spawn everyone and hold them still until after all your initialisation scripts have been exec'd. you might want to let 10 out at a time from each team, to synchronise team response... etc etc

in a larger model, giving groups the ability to organise themselves through mutation of rules (e.g. allowing certain formations, or orders to be available as mutations) and maybe making 4 mutually opposing teams might create an interesting set of dynamics...

I'll have a good look at your parameters and feed back.

good work!

[TRexian 0]

Some excellent advice eggbeast! :) Getting ready to go on vacation, with limited internet access (and no A2), literally in 2 hours. :d0h: But, as to the start time/conditions, each iteration re-sets to the mission start time. So, whatever time the mission maker sets (to allow for testing night missions, also) is when each iteration starts.

The part about spawning them in groups is something I considered, but couldn't really sort out how to do it without knowing where on the map the start positions were. One fundamental goal I had was to allow the init to work on any map, at any location. I'm very open to ideas on how to achieve it, though!

The rule mutation would be fascinating, and I kinda have the ability to do that sketched in, but without custom FSMs, we're pretty much limited to changing combatmodes.

I look forward to returning to this in a couple weeks! :)

[KeyCat 131]

Interesting experiment and a good read, will follow your work closely and have a nice vacation!

/KC

[eggbeast 3684]

random thought of the day: i also think that ammo count will be a major factor (as it is in real life) to survival in an extended ironsights firefight.

it might be worth checking the ammo capacity of your units (as it may vary), and the ammo you give them. Increasing the ammo count may change the outcome significantly i don't know.

also do they have medics?

[dmarkwick 261]

All the units are equipped identically in this first experiment to keep things simple. No medics, all units are riflemen. Expansion of this experiment will possibly take the form of using different unit mixes.

[-)rStrangelove 0]

What i'd love to see would not be AI skill values being used, but make AI groups use random scripted tactics.

Both AI groups would use the same set of tactics, but randomly. Tactics for both teams would be traced and a winner would be recorded. It could lead to precise information whether tactics really work and/or make a difference in ArmA2 AI wars.

Tactics could be:

1. Bait

Group A builds (spawns) entrenched positions and sends a small group of troublemakers ahead. Those get the attention of Group B and then retreat. Entrenched Group A opens fire on advancing Group B.

2. Split & Flank

Group A splits up into 3 large subgroups and tries to surround Group B.

3. Indirect radius damage

A Group uses hidden spotters and mortars instead of advancing towards enemy.

4. Trapped retreat

Group sets up a minefield and tries to draw the enemy into it.

5. ???

Just an idea.

[eggbeast 3684]

nice thoughts. i'd humbly suggest that so much can go wrong in an arma group script that the results of this might well be meaningless.

[dmarkwick 261]

nice thoughts. i'd humbly suggest that so much can go wrong in an arma group script that the results of this might well be meaningless.

'Tis true :) this is partially why TRexian is trying to keep it as simple as possible by only using identical units in identical groups, just to try & get a basic "feel" as to what general traits, or combinations of traits, prevail. It might provide a basic template for general battlefield survival, but I doubt it will provide much in the way of meaningful detailed information on more complex matters. As you say, so many more factors begin to influence outcomes that the main purpose of the experiment becomes lost.