Object Quality seems to have the greatest influence with regards to this 0 to 5 FPS issue. It can vastly change the total object count within the player's field of view (including objects hiding behind other objects which aren't even being rendered, objects behind terrain appear to get culled). With regards to the heavy stuttering the total complexity (textures and polygons) doesn't seem to be an issue, just the object count.
Lowering the setting shortens the LOD distances and culls smaller objects at much shorter distances. You will experience more LOD-pop and pop-in/-out closer than the Object View Distance but Objects very close to the player still have high levels of detail.