I'm having this problem at the moment. I'm running a GTX 980ti i7-4790k and 16GB of ram and after 15min of playing with crappy fps it drops to an unplayable 2-3 and textures stop loading. None of the different memallocs help. I'm at a point where I've tried everything and am in desperate need of some advice.
For a while I thought it might just be the server me and my friends were playing on because it was running Exile, but my buddy who has the exact same rig that I do (minus the 980, he's using my hand-me-down 960 2GB) is getting 30+ fps consistently.
EDIT: Going to try the fix mentioned above tomorrow.