I've found this issue is somewhat reliably recurring when hosting my own server and playing on it with friends. I've always had the theory that it's something to do with leftover AI spawns as I progress through. The issue hasn't really shown up when playing on a server not hosted locally.