I've read the whole thread and agree with a lot of the recommendations being made (especially about recognising and using cover). I have a feeling that there are reasons for the current situation that won't allow these kind of changes however. (Also I'm bemused by the people who are arguing against any improvement in the game, especially since they seem to be the biggest fans.)
I'd like to add one of my own - that the AI recognise and react in some manner to the death of someone in the squad. This alone could fix some of the worst behaviour if implemented right (stacking up in stupid areas, travelling across open areas when under fire), and could be used to provide a rudimentary 'learning' of the layout of the battle.
Here are some suggestions off the top of my head:
The area the squad member is shot in is given some kind of significance. Ideally it would be some kind of 'danger' rating, so areas where a few people get shot would be avoided unless necessary to return fire or other mission-critical action. Obviously these would need to 'fade' quickly to avoid building up too much over a firefight. This could be a timer, but preferably it would be based on the shooter having moved a certain distance, or been killed, or newer deaths taking priority. This 'reset' could also apply to re-evaluating cover positions, to save processing. Moving through these zones would also trigger a burst of suppressive fire from squad-members in the direction the original killing shot came from.
A kill in close proximity should set off an immediate scramble for cover (with of course some return fire if certain conditions exist - shooter position spotted, shooter is close, no cover, only cover available is in a 'danger' zone). The squad could then hopefully reassemble with movement based on suppressive fire (lol, right). It should also result in the prioritising of the shooter as a target (if visible), or that direction for suppressive fire.
I'm aware the specific examples may not be possible, but even if thats true I'd ask you don't discount the basic idea - I think a lot of the problems with the perceived lack of self-preservation come from the apparent obliviousness to danger. (Danger being most clearly illustrated in AA by the guy-next-to-you getting shot in the head).
It's also worth noting a lesson Bungie learnt - the AI in Halo was only seen as good as the feedback to the player. When they added in more complex routines that didn't have shouts explaining what the characters were doing (e.g. "Run away!"), the testers ranked the AI worse. Now I realise BIS are aiming for a simulation more than a simple appearance of intelligence, but I think some kind of feedback would help both in immersiveness *and* tactics, as awareness of the AIs actions (retreat in disarray vs controlled falling back vs rushing another position) affects your decision-making and general overview of the battle. As for what this feedback could be, I honestly dont' know - audio, animation, formation/movement - there are a lot of options, but the lesson seems to be the clearer the better. One aspect of AA thats a problem here is that soldiers should be well-trained, so it would be hard to justify showing them running, screaming and flailing their hands wildly when retreating
I have a bunch more ideas about systems for use of cover, but that seems to be pretty well covered and I should be working