Jumpy game, GPU usage problem?

So after I play for a while (~1hr) I often get a very jumpy game with smooth FPS for a few seconds (25ish) and then 3-5FPS in between. It seemed like a memory problem because after I flush the game it fixes itself for a few seconds, but then goes back. I looked at GPU usage to see if this was the problem, and this is what I got:

The high parts are where it's smooth, and the low are the crappy parts. Not sure what the problem is here...

Working with a GTX460, i7 950.

This was in multiplayer, in SP it hovers around 60% usage normally.