InstaGoat's AI Test Range

[instagoat 133]

This is the thread where I will, as the Alpha goes up the development path, test the AI in a more or less scientific manner (ie, in limited, non-cluttered, controlled settings.) to see what it does and what it doesn´t do.

The goal is to provide a platform to demonstrate the AI, to evaluate their performance and provide feedback to the development team for improvement via the devtracker. I will be limited to uploading pictures, most of the time, as my current internet connection does not permit me to upload video footage. I will upload videos whenever I can, however, and when I do tests I will make protocols of them. I will also try and figure out statistical points of interest, such as time to aquire target, ratios of friendly-fire to hitting an actual enemy, employment of grenades, spotting distances, etc, etc. I have not yet figured out what is useful and what isn´t, as I´ve not paid attention to all this pretty much since the beginning of Arma 2 when I made a couple of such testing ranges and ran the Arma 2 AI through them. The videos can be found here:

These were done in an -early- Arma 2.

I will also try and provide suggestions for test protocol and arenas for other players. I am not a coder, so if I find the time to actually do this properly, help by a coder to provide tools to actually measure AI performances in selected key areas would be great.

I have one arena in A3 now, set up as a rectangle of 50 x 100 meters (approx.) as shown here:

It uses Hesco (Big) and standard Hesco barriers. My goal with this is basically to just have a starting point: the AI move towards their guard points in aware mode, make contact, and then dynamically engage each other. In Five tests I have run so far, opfor has won four times (equal AI settings in difficulty and config) due to the machinegunner's mean accuracy, and the inabillity of the bluefor AI to hit anything when standing up firing their MX's full auto. I have not seen a single thrown grenade used, to my surprise.

I will provide a packet of a handful of arenas once I have built them. Again, I cannot script, so all I can do is just observe and build very basic things. To know what is happening with the AI proper, I would need the help of someone who understands how to access these data as they happen. There was a tool for Arma 2 that had such a feature, I believe, it was called Troopmon (was it?). So if anyone wants to help with polishing the AI for coop and SP, and is interested in an ordered, semi-scientific approach, I am usually available on the arma 3 irc on gamesurge, and here on the forums.

Right now, I will do my own thing and post my findings and Ideas as I get them.

UPDATE 1 (Initial Release version, 23.03.2013): http://forums.bistudio.com/showthread.php?151246-InstaGoat-s-AI-Test-Range&p=2352860&viewfull=1#post2352860

Edited March 23, 2013 by InstaGoat

[Gorbachev 1]

This should be very interesting. I hope to see the AI evolve between now and release through this thread.

[gliptal 25]

Interesting, will surely follow!

Yay!

[instagoat 133]

I have concluded the first round of testing. These were simple setups, using a maximum of 8 troops on each side, or up to 2 troops and 1 static, armed vehicle. Super AI on, skill sliders in difficulty were set rather low, skill sliders in editor either 0 or 0.5.

The setups included mazes of different layouts to test what the AI can navigate and what it can´t, and how they do it. I didn´t keep any of these as these were just to figure out how to build reasonable setups later. I have made notes, however. All in all, I ran about 25 tests, with 2 - 6 runs with each layout.

Preliminary findings are as follows:

- AI does not use handgrenades even against enemies they are aware of, who are behind cover or an object blocking LOS.

- AI on Blufor side lost in 5 out of 6 sessions, even if given a small numerical advantage of 8 vs 6 or 4 vs 3. This seems to be linked to the recoil behaviour of the MX rifle. I have observed at least two occasions where a single AI soldier expended -all- his MX ammo at a distance of 20 meters, each shot going over the standing enemies head (who was for some reason non-responsive in each case). The MX ammo was expended, causing the AI soldier to pull out his pistol and drop the enemy with one or two shots. Conversely, Opfor AI tends to be extremely accurate with the first shot and miss all subsequent shots, unless prone. Particularily deadly are the autoriflemen, who appeared to account for just under half of all kills, with the rest shared among the other three members of their fireteams.

These opfor autoriflemen in most of the cases killed with their very first bullet, with the remainder flying high.

- AI moves much more coherently and covers better than in Arma 2, but still often looses cohesion especially when casualties are taken, and fails to anticipate enemy movement. This becomes especially gross a disadvantage when competing against human players, who when moving with extreme decisiveness and speed can run circles around AI, as long as viewblocks and soundblocks are present. In open terrain, AI usually wins due to their much keener long range spotting senses as well as their impeccable first shot accuracy.

- The killzone detection, while still present, is still just as wonky as it was in Arma 2. In all tests, AI usually noticed a killzone (or appeared to, not having any Idea how they work under the hood, only observed movement could give hints as to their "intentions"), hesitated, and then charged right into it again. At least four occasions were observed where AI suffered from the "glueing themselves to a wall" behaviour often observed in Arma 2, where they stand up on the wrong side of cover, only to be mowed down by a single prone enemy. In all of these occasions, multiple AI glued themselves to the wall in close proximity, but in all but one case failed to return fire or make movement out of the enemy line of sight.

- AI still reliably walks through walls if camera is not paying attention to them, or if they are more than ~50 meters away from the camera. If camera is close and/or pointed directly at them, this does not seem to occur as often. In all of the tests, this occured at least four times, always when using the 8 vs 8 setup.

- AT AI will not use their launchers against close enemy vehicles, employ grenades or try to position themselves outside of the vehicles arc of fires when trying to destroy it. In a 2 blufor AT vs 1 Ifrit HMG setup, where there were barriers arranged at various distances and of various heights around the vehicle, providing one clear LOS for AT missile fire to "make the right choice easy", in not one single occasion did the blufor At soldiers use their launchers, or employ grenades against the enemy vehicle. When using their rifles, they also failed to fire at any obvious weak spots, such as tyres or the engine bay, but always tried shooting at the driver, failing to penetrate the bulletproof cabin and glass.

Blufor also always shot at the vehicle only after walking out into open terrain, far outside of any cover, and did not seek cover even when the enemy reacted by turning his turret towards him.

I will provide a link for every update I post in the OP on the first page, as well as other links of interest. Coming week, if I find the time after work and in preparation for moving, I will post some arenas for the community to work in for AI tests, as well as some of my specific test protocols along with mission files.

Cheers

Insta

[CptCubano 1]

Wow I really appreciate your work! Very interesting

[whisky 1]

awesome work Insta!

[Fuse 1]

Interesting and valuable. Keep up the good work!

[iconoclastdx 5]

I'm looking forward to your findings. BTW. What are "killzones" in terms of AI behavior?

[CAI 1]

Great work, I hope you manage to get some videos up :) Doesn't surprise me about the MX rifle, I made my opinion on the recoil thread.

[Levinin 1]

Very interesting. I made some small experiments and created a scenario where a rifle squad attacks and another defends and repeated for each side. The attacker always wins no matter which side attacks or defends, although I was not scientific enough to see which units made the kills. To me it seems that defending units don't understand their terrain enough to defend positions.

I then thought, maybe defender didn't have enough notice of the attack and were ambushed so I put a defending unit further out to force a small action before main attack but result was the same. Attacking AI always wins regardless of opfor or bluefor.

[Scoggs 1]

Very good feedback. Interested how Ai will change over the course of the alpha.

[instagoat 133]

I'm looking forward to your findings. BTW. What are "killzones" in terms of AI behavior?

Killzones as I use them are defined as fixed areas that will almost inevitably result in a casualty if being crossed. For example, a killzone could be an alley that´s being faced down by a tank at one end. Crossing it while the tank is looking down the alley will result in casualties, making the alley a killzone. Another killzone for example would be a street corner that when rounded results in a casualty. The AI loves to run blindly around corners, so pretty much any corner with an enemy behind it that is static will result in this, with multiple AI being shot as they round the corner, even if somebody right in front of them gets shot.

Very interesting. I made some small experiments and created a scenario where a rifle squad attacks and another defends and repeated for each side. The attacker always wins no matter which side attacks or defends, although I was not scientific enough to see which units made the kills. To me it seems that defending units don't understand their terrain enough to defend positions.

I then thought, maybe defender didn't have enough notice of the attack and were ambushed so I put a defending unit further out to force a small action before main attack but result was the same. Attacking AI always wins regardless of opfor or bluefor.

This is heavily dependent on the terrain, and who spots who first. For example, I made an assault on the far southern LZ with a squad of Blufor and a truck of theirs coming down the road from the buildings in the north, and they lost every time. I added a second squad assaulting from the eastern hillside, halfway up the coast, and then the attackers won every time, because the lower squad could basically walk right into the fortifications without being spotted.

I will also do small attack/defend setups later and see how the AI handles those in a controlled environment.

[froggyluv 2136]

This was done before the big Arma 2 beta merge, no? Would be interested if any change. Also are those barriers your using a proper height for the AI crouch position? One thing I've noticed (and like) is that they are far more likely to crouch than in Arma 2 which was almost non-existent unless using AI mods.

[everyone 10]

I'm glad someone is taking a measured look at this. I spent twenty minutes yesterday trying to force an AI to shoot an anti-air rocket at an incoming helicopter, and it was generally a miserable failure. I've had similar poor experiences trying to get AT soldiers to engage hostile Ifrits. Come to think of it, relying on AI to take on anything at all generally turns out poorly...

[instagoat 133]

Very busy with real life, so I could only get a handful of tests in.

I´ve found a bunch of small AI glitches, such as AI (happens very rarely and I couldn´t ever reliably reproduce it) getting stuck with the pistol out.

However, I have noticed a handful of other things.

First shot accuracy of the AI with any type of weapon is -stupefying- at any range. In fact, at medium (100 - 400 m) ranges they seem to be better than at very close or very far ranges. The hit-detection also seems to be wonky at times, where I had bullets pass through my camera as I dodged, but remained unharmed. At other times, tracers passed me only closeby and my character fell over.

However, the most annoying thing about the first shot accuracy is the grenade launcher people. I have done tests on the airfield, as well as repeated runs of the infantry showcase, and if I wasn´t killed by sniping machinegunners, I was killed by single-shotting grenade launchers from 400 meters away. The accuracy of the AI in other cases is -far- below acceptable, for example in CQB they will fire full auto, and with most guns they will miss more than they will hit.

Another severe problem is the CPU load the AI takes. I run the game at 40 - 50 fps with medium settings on a 2008 era rig, with no AI on the map.

Today I have tried running a scenario with 140~ Units total, that including about six vehicles on each sides (Blufor spawned all over the map with the modules, Redfor commanded by rydigiers HAC).

As soon as the AI started to move, fps dropped from 30 to about 15 - 20. As soon as the AI got into combat, the fps dropped from 15~ to about 2 - 5 fps. The AI also turned blind, and started to walk through objects, stand inside each other, and exhibit all the glitches known from previous Arma games when CPU load was severely high.

They stopped spotting each other and only rarely fired at each other, often at positions where an enemy had been spotted minutes ago. I have screenshots of a bunch of opfor running around a lone blufor. Nobody seemed to be spotting or attempting to target the enemy. Nobody fired a shot.

Picture here:

Red dots denote positions of nearby opfor soldiers that could´ve shot him (there were a lot more around the compound, I estimate around 30, and two trucks).

Blue dots are alive Blufor. There were probably more, but I cancelled the mission at this point as the fps were regularily down in the 1s and 2s by then.

Note how there is a redfor standing right next to the soldier at the portable building. The blufor was pointing his gun right past his head the entire time, while redfor was staring at my direction. The guy in the grass to the left of the hut in front of the bush was crawling past, also not bothering to do anything. I watched this for about 30 seconds, when my machinegunner (with one shot) killed the blufor rifleman.

Even my own team took 30 seconds from me spotting the enemy (standing right next to me all the time) to aquiring and shooting him.

I have a dualcore, I did not observe the cpu loads, but I´d be curious wether or not both cores were used, and how. Generally I think this may need looking at, especially if there are going to be bigger battles. I remember Arma 2's manhattan mission, where you had situations similar to this, albeit not as severe. This is a problem.

[lightspeed_aust 681]

Great idea mate. What would be really interesting is to use Robalos ASR AI mod and see how they respond to see whether the issues can/ would be resolved.

[instagoat 133]

I am focusing on this without mods. I am tired of using AI alterations, which usually make missions that are not designed with them in mind slightly wonky at the least, and completely broken at worst. I will update once or twice a week now that I have more time as things slow down irl (which I hope they will do, at least.).

Doing this kind of testing is kinda dry and boring, you have to do things over and over rather often to see patterns. The time I found the last couple of weeks was spent actually playing, not testing. :I

[lightspeed_aust 681]

I understand that. I suggested as a mean of testing whether the AI issues could b resolved by better or more effective scripting. I might try and mimic what you've done and see whether results differ.