ArmA 2 I/O analysis results

[qwertz 10]

THIS THREAD IS CONTINUING THE "RAMDISK" THREAD - PLEASE CONTINUE THE RAMDISK DISCUSSION HERE!

Original thread: http://forums.bistudio.com/showthread.php?t=88388

_________________________________________________________________

Update 19 Oct 2009

I have pimped my analysis tool a little so that everyone can download and use it to easily analyze his/her own ArmA 2 disk performance.

I am not a programmer so this is one large Excel file. It includes all instructions, and the process is rather simple (read the instructions carefully though).

If you use it, It would be great if you guys would share the results here in the forum so we can all learn more.

Here's the download link to the ArmA_2_IO_Analysis_v0.22.zip file (now version 0.22, should work with everyone's data now):

ArmA_2_IO_Analysis_v0.22.zip

Please let me know what you think, i.e. if there are any problems with the tool.

___________________________________________________________________

OK guys,

here's the results of a few hours of analysis (I will just post them and comment later/tomorrow in detail) - very interesting findings:

Arma does not use file caching - period. This means all the tools that are advertising to optimize the windows cache are useless for our purpose. On the other hand, it means that Arma is really 100% streaming the content. Because as a 32 bit app it can only address 2GB max (and I have yet to see it use more than 1GB), with >9GB of game assets, it means that there must be a LOT of disk activity.

Lets have a closer look:

This is 3 x the same data, sorted (1) by PBO name (2) by MB/s per PBO and (3) by total number of read I/Os per PBO. You can use the two tables in the middle and on the right to see which PBO's you should prioritize when you have limited RAM available for a RAMdisk - just look at the "FS (MB) cum.(ulated)" column. Also, note that my 6 minutes of gameplay left some PBOs completely untouched - this alone saves 1.7GB, but might be caused by the rather short game play time for the dataset.

This dataset is based upon 6 minutes of gameplay, with ProcMon capturing all file activity in the ArmA 2 folder (for this exercise, the ArmA 2 folder was on my 2 x Velociraptor RAID0 disk array, not on the RAMdrive). As you can see, ProcMon recorded over 10,000 read events, totalling ~1GB of data reads in 360 seconds :eek:. This is mostly "random read" activity - wait a moment - hard disks are not very good at random read - this might get interesting.

Here is another look at the data, now on the 360 seconds timeline - and this is the key to understanding why we have stutter with HDDs but not with RAMdisks:

On average, ArmA 2 creates a load of 30 IOPS and reads 2.9 MB/s. This might sound harmless, but as the graph shows, the distribution of I/Os is not at all even - it shows heavy spiking !

As a reference, this is what a fast HDD (Velociraptor) is capable of:

As you can see, in random read the fastes consumer HDD is capable of 50-141 IOPS, and 0.07-50.00 MB/s (mind you, your average 7200 rpm HDD is way slower). What is more important is that as you can see above, the average I/O size across all reads recoreded is 97 KB. The mighty Velociraptor is somewhere between 50 and 127 IOPS for this I/O size. So, the spikes on the IOPS graph are definitely saturating the HDD, causing stutter.

But it gets worse. The 97k average I/O size is not evently spread out on the timeline, either. As you can see on the graph above, it oscillates between a few KB to up to 1000 KB, and many times it dips into the 200k and bigger numbers.

At these I/O chunk sizes, even a single Velociraptor can not deliver the 50+ Iops and 10+ MB/s that ArmA 2 is churning through !!

This is what causes the stutter - ArmA 2 simply oversaturates the average hard disk subsystem. And this is why a RAMdisk helps so much in increasing the smoothness of the game!

This also means that this data is not reflective of what the real I/O need for ArmA 2 is - without the HDDs bottlenecking, we would see higher numbers here. I will do the same test running off the RAMdrive when I have time to verify/falsify this thesis.

Also have in mind that I am still working with averages here and one-second "buckets" - there might be more extreme oscillation going on that is just not visible at this resolution.

First conclusion: put ArmA 2 on the fastest disk subsystem you have. This test was run on a Veloci RAID0, and I had a lot of stutter (well, coming from my RAMdisk, that is). Even on my Vertex SSD RAID0, there is still stutter. Only the RAMdisk removed the bottleneck completely - go figure.

I'm tired now, will share some more observations with you tomorrow.

Best,

qwertz

Edited January 21, 2012 by qwertz

[Tankbuster 1747]

Resize those images, fella. You'll break the internet posting more than 100k pics!

[Peonza 10]

Cool find. Now it explains why some people playing with 8800 cards owns ones with 4870's in arma mark

[Greg 0]

Very interesting findings Qwertz. This kind of profiling would be useful to BIS if they chose to pay attention to it.

[4 IN 1 0]

I personally dont think there is much for BI can do on this matter

[.kju 3245]

Sincere thanks for the in depth analysis Qwertz!

One aspect nice to verify is that if lower graphic settings

mean less data to stream / read from the HDD.

PS: BI can make special contracts with SSD or RAM drive companies. :P

[Inkompetent 0]

Very nice analysis indeed!

It was already known that RAM/SSD drives vastly improve ArmA2 performance, but it's lovely to get see some actual numbers on it :)

[Alex72 1]

Thanks a lo Qwertz!

RAMDISK it is! Since ARMA2 is the only thing i play really.

[bono_lv 10]

Thank you for taking your time to do this analysis. Really informative.

On other side, I must agree with 4in1. I don't think there is anything that BIS can improve with that.

Also... It would be nice if someone of community could find explanation why buildings in Arma 2 are such a fps killer.

[PogMoThoin 10]

This explains why the Sahrani caa1 mod runs much smoother than Chernarus, no constant loading of the plants2_tree.pbo or structures.pbo as the map is less dense in plants and buildings.

Now I've only 4GB ram (wow, I just said it, 4GB is loads), there must be a way we can add the most used .pbo's to a 1GB ramdisk or force the game to load them to ram. Hope BI are watching, this is an idea for the next patch

Nice work qwertz

[liquidpinky 11]

Nice work qwertz, from that data I should be able to optimize better what I have put on my RamDisk. Just wish I was in your position and could put it all on it.

[Barso 10]

This is fantastic and very informative.

How much RAM do I need to load the stock game with no mods into a RAMDISK and which RAMDISK is the one to buy?

It would be great for a step by step instruction for the non-technically minded like myself to get the game to run as optimal as we can on a PC with windows7 64 bit and 12 GB RAM.

Thanks.

[S-M 10]

The best post i have seen on this forum, its almost a shame the chap is 100% sure to be chastised and thrown into a rape jail, for posting pictures that are to large.

Just wait until hitler see`s it ;)

The law is the law and nobody can apparently escape it :D

[Bulldogs 10]

This is very interesting. Looks like I'm going to have to do some heavy testing myself. Although, if this thesis is correct, then I wonder why I don't get any studdering using a standard hard disk. I'll have to look into it and see if I can figure it out.

[Pappy60 10]

Certainly one of the most detailed studies of the issues, absolutely phenomenal job !

However I am not so sure it addresses EVERYthing it certainly shows us some dramatic evidence of why we are seeing slow painting and stuttering...but that said, I followed LiquidPinkys instructions for a ramdrive (very nice job by the way) and setup a ramdrive with the files he detailed and did get it to work as a junction under XP but still had stuttering and slow painting , albeit less of both but still there.....I think Qwertz is right that if I get the entire game into ram it will probably solve my issues...but thats an $800 upgrade for me(I currently only have 8gb) ...quite painful and of course at that point I will need to upgrade my quad core 2.5 ghz cpu because it won't be able to handle the streams coming out of ram ...

Don't get me wrong, the work being done in this post is incredibly good and necessary but I am afraid the evidence that results may tell us all that it is going to take very deep pockets to play this game as shown in the videos......

I certainly would like to see a posting from BIS as to what the machine specs were when they created their videos ?

[S-M 10]

I certainly would like to see a posting from BIS as to what the machine specs were when they created their videos ?

One would assume that they used the recommended specs listed on the back of the box maybe ;) :D

Or maybe not :p

[Greg 0]

...I certainly would like to see a posting from BIS as to what the machine specs were when they created their videos ?

I don't know about their videos but Suma (developer) posted his spec here:

Intel Core 2 Quad 2.4 GHz

nVidia 8800 GT

8 GB RAM

Vista x64

http://forums.bistudio.com/showthread.php?t=75747

That would have been a good spec machine during much of ARMA2 development.

---------- Post added at 12:22 PM ---------- Previous post was at 12:09 PM ----------

I personally dont think there is much for BI can do on this matter

BI is the developer, they control exactly how data gets from hard disk into system and video memory. Armed with the knowledge of how their application performs they have all kinds of options available to them.

Here's just a few things they could try:

1) Use the large memory flag to access up to 3gb of memory on 32bit system.

2) Use available memory to hold game data longer or preload more data.

3) Use HW threads on common multicore CPUs to perform background loading and preparing of data.

4) Profile the game before release and/or collect stats during runtime (since the game is moddable) and use these stats to preload or cache portions of the game for best performance.

5) Tweak their streaming code to make better use of asynchronous data transfers and hardware threads.

6) Slap on a internal cache to their file system which simply uses available memory to cache more data, transparent to the rest of the game.

7) Look at their on disk and in memory data structures for ways to reduce or compress the foot print.

8) Look at their runtime data retention logic (resource management) to see why data is repeatedly or regularly loaded.

9) Tweak the LOD system to reduce jerky behavior in anyway.

They know their own game, they've been developing it for the last 10 years. I'm just suggesting things I've done in games I've developed.

[S-M 10]

I don't know about their videos but Suma (developer) posted his spec here:

Intel Core 2 Quad 2.4 GHz

nVidia 8800 GT

8 GB RAM

Vista x64

http://forums.bistudio.com/showthread.php?t=75747

That would have been a good spec machine during much of ARMA2 development.

He also says that the game runs great for him on vista!

I wish mine did on xp :(

[dogz 10]

Very good work qwertz... keep it up!!

sadly those that had to have this explained to them are not very PC smart... and you will always find it difficult to keep your pc running well...

It has been posted/explained a number of times how much hard disk activity affects this game and surely we can all see the HD LED on our supreme gaming rigs????

[lee_h._oswald 0]

Fantastic analysis, thank you for this!

People should really test their HDD speed, easy tool for this is HDTach.

Screen shows the performance of my Velociraptor SATA Raid0.

MfG Lee

[t3K-Mac 0]

i am not able to use sym links on winXP using hardlinkshellextender. hardlinks just work inside one ntfs volume. i cannot create a link from my ram-disk to my harddisk (both ntfs). anyone got that working on xp?

[polecat.uk 11]

Guys i've been reading this tread with intrest, not that i'm having the problems like some ppl are, i dont get stutter unless i'm in a jet going really low and fast and this with a res of 5040x1050 i run the game on a gtx285, XP 32bit, 4gig ram and 3 22" monitors at around 30 fps, i get the odd slowdown mostly happens when i'm zoomed in using a scope and then try to move, its just now and then i get this though.

If i try to play on 1 monitor i feel my movement feels very stiff but when i'm using all 3 it feels more natural and a much nicer way to play, now i can't run everything full tilt but my setting are high very high, views at 3500 and the game looks stunning plays smooth and even though my fps fall using 3 monitors my movement feels better than having more fps on one monitor.

Now saying that :rolleyes: i do suffer from things not loading to fast, it takes a good 10sec to load all the textures when i join a game or respown and ingame it take along time to tell if a tank or apc is dead, i have to wait aleast 20sec to let the texture for a dead tank load, i was thinking of a ssd but not sure this would fix my problem what do you guys think?

[Dwarden 1125]

i will repeat this ... if you have multiple drives or raids it's always wise to spread OS, swap, games to standalone drives/raids if you got even more spare spread the programs to one also :)

SSD are very nice option but they expensive for most of users

and RAMdrives aren't option for everyone due e.g. hardware limitations (chipset, slots or price etc)

still it's nice to see such posts around (i wrote huge post like year ago with nearly the same content about certain files only needed to be put into ramdrive :)

[Tankbuster 1747]

I don't know about their videos but Suma (developer) posted his spec here:

Intel Core 2 Quad 2.4 GHz

nVidia 8800 GT

8 GB RAM

Vista x64

He also says that the game runs great for him on vista!

You'd expect him to say that, though wouldn't you?

[Muahaha 10]

Hi qwertz ,

Thanks for the detailed analysis, any idea how much of % of improvements you have ever since using Ramdisk/SSD?