I am 100% sure this error is a Exile-problem. Just check Exile forum. I have the same issues, but Vanilla runs perfect. All exile-servers, fps drops to 2-3, then I get the memory error or my game crashes to black screen.
Another issue I have, still only on exile servers, viewing distance seems like its conflicting with server settings. I can play for a while, then terrain in the distance starts popping on and off. This causes major fps-drops ofc. I usually run all settings on ultra/high with 8xaa and ultra fxaa. Whenever this issue occurs, fps drops to around 20, when im usually getting around 60.