ArmA 2 I/O analysis results

[tcp 10]

I didn't come up with that number, but a number of sites that showed up on google reported numbers around there.

What's ROF?

ArmA2 currently only uses about half of its allocated 2047MB (not 3GB).

So what is the source of the bottleneck really?

What does 64-bit offer?

"...those who use 64-bit systems can enjoy the perks - slightly sped-up processes on double-executed 32 bit code, a full 4GB of RAM, protected execution, more stable and accurate code due to the increased data size and enhanced overall system efficiency due to relative pointers..." - http://www.bit-tech.net/bits/2007/10/16/64-bit_more_than_just_the_ram/5

Those are supposed to be the advantages for existing 32-bit programs, but obviously running ArmA2 under 64-bit windows has not been reported to offer any advantages.

Although 64-bit processing has been proven in multi-application servers, I doubt it will have much of a performance impact on games for some time.

Also, how long do you think you'll be waiting for 64-bit support in ArmA2?

[qwertz 10]

I didn't come up with that number, but a number of sites that showed up on google reported numbers around there.

What's ROF?

ArmA2 currently only uses about half of its allocated 2047MB (not 3GB).

So what is the source of the bottleneck really?

What does 64-bit offer?

"...those who use 64-bit systems can enjoy the perks - slightly sped-up processes on double-executed 32 bit code, a full 4GB of RAM, protected execution, more stable and accurate code due to the increased data size and enhanced overall system efficiency due to relative pointers..." - http://www.bit-tech.net/bits/2007/10/16/64-bit_more_than_just_the_ram/5

Those are supposed to be the advantages for existing 32-bit programs, but obviously running ArmA2 under 64-bit windows has not been reported to offer any advantages

Although 64-bit processing has been proven in multi-application servers, I doubt it will have much of a performance impact on games for some time.

Also, how long do you think you'll be waiting for 64-bit support in ArmA2?

Whilst we don't know for sure and are really only guessing, I think that that there is some evidence that points to a scenario that would scream for 64bit addressing. (note: 64bit PROCESS on 64bit OS)

[GUESS MODE ON]

As hypothized earlier (and thanks to oktane, somewhat verified in his code review), Arma2.exe does not use system level Windows file caching (which is a place in RAM that buffers reads and writes to the filesystem). The likely reason is that the Windows cache would only cover for a very small fraction of the assets needed by this game, effectively rendering the cache useless (or even counterproductive in some way). The effect is, that Arma2.exe is now forced to stream directly from disk, which in turn (and based on filesystem I/O performance) causes stutter.

Of course, any application can create and use its own proprietary buffer implementation (i.e. reserving a large part of available ram and prefetching stuff there). However, as a 32bit application (and without PAE/Largeadressaware), Arma2.exe can only adress 2GB of ram in total for everything including the game code. Therefore, any proprietary prefetching would again only be available for a small fraction of the game assets, with the same problems as described above.

Now, with 64 bit, on a 8 GB machine, the engine could take 6GB as a cache, would prefetch all assets it needs for a certain escenario dynamically, and would keep like 1GB for the OS and another 1GB for program code. This would likely remove the bottleneck that we are discussing here.

[/GUESS MODE OFF]

Edited October 23, 2009 by qwertz

[tcp 10]

Right, BI could add 64-bit support now and possibly solve a lot of our problems, but will they do it?

What is still unexplained is why ArmA2 doesn't use up the memory is allocated (half of its 2047MB) and why Bohemia turned off file caching to begin with.

It's strange because you never see ArmA2 use up its application memory space, but people still get out of memory errors. Also, if you create a bloated mission that is prone to crashing, I've found that setting -maxmem will prevent the crash.

It's a mystery, but I wonder if the memory issue is something to do with a quirk in Windows. For a game that has been in development so long, why can it run well with what it has and are the developers even on the same page as us?

[Alex72 1]

Now, with 64 bit, on a 8 GB machine, the engine could take 6GB as a cache, would prefetch all assets it needs for a certain escenario dynamically, and would keep like 1GB for the OS and another 1GB for program code. This would likely remove the bottleneck that we are discussing here.

Remember Operation Arrowhead and the words "Core Engine Changes".

Hey i can dream. :)

[ModeZt 0]

64-bit executable you say? That's cool, but what about all the customers who already have the game and don't want to buy the new OS and send to trash existing 32-bit versions of XP, Vista or Seven?

Do BIS have to abandon the 32-bit executable? Leave it in current state and start development of a 64-bit only game?

[qwertz 10]

That's cool, but what about all the customers who already have the game and don't want to buy the new OS and send to trash existing 32-bit versions of XP, Vista or Seven?

The same that happened to the happy customers that love Win 3.11 so much that they never upgraded to Win 95: They are bitten by the dog named "anachronism". :D

[.kju 3240]

The other solution is to let ppl upload their CSV and someone else does the processing. :)

In regards to 64b version: Most likely it is not simple to "just port it to 64b".

Especially if the engine uses some nasty hacks of 32b addressing.

Every software engineer knows that old code is scary.

In the end we can only assume that the benefit does not outweigh the effort

to do it - for now. First the game should use the 2 GB to the max - if that is

useful at all.

[lee_h._oswald 0]

And the best soloution for the devs would be:

- PROPER designed/modeled islands/buildings/trees etc.

- Proper use of "Level of details", not like BIS trees/buildings/vehicles.

- Lower res textures and bring them to life with features like bump mapping and stuff.

- Smaller islands with less trees, no need for simulating a "Schwarzwald"

- Look at topics like this and DO something!

MfG Lee

Edited October 24, 2009 by Lee_H._Oswald

[bono_lv 10]

And the best soloution for the devs would be:

- PROPER designed/modeled islands/buildings/trees etc.

- Proper use of "Level of details", not like BIS trees/buildings/vehicles.

- Lower res textures and bring them to life with features like bump mapping and stuff.

- Smaller islands with less trees, no need for simulating a "Schwarzwald"

- Look at topics like this and DO something!

MfG Lee

I disagree with smaller islands and less trees. Just do something with those god damn buildings. I have 50-75 fps in any place where is no buildings. Even in dense forest.

But I agree with all other mentioned things.

[Leon86 13]

I disagree with smaller islands and less trees. Just do something with those god damn buildings. I have 50-75 fps in any place where is no buildings. Even in dense forest.

But I agree with all other mentioned things.

Then you probably have a fast gpu. For me the trees are really heavy on the gpu while the city's are heavy on the cpu. I have a e6750 (on 3.4) and an 8800gtx. that should be within optimal requirements. I can only imagine how performance must be on the minimum requirements.

The biggest problem with the trees (not in forests but just bunches of them) is that no matter how low your terrain and object detail are they ALWAYS go to the highest LOD if you get close.

[cas 41]

If you want to use a ram-disk, does STEAM have any advantages because it installs all files on the hard disk? Or do the DVD versions also install all relevant files on the HDD?

[evilfury 0]

Of course dvd version install all files too disk like steam does.

[bono_lv 10]

Then you probably have a fast gpu. For me the trees are really heavy on the gpu while the city's are heavy on the cpu. I have a e6750 (on 3.4) and an 8800gtx. that should be within optimal requirements. I can only imagine how performance must be on the minimum requirements.

The biggest problem with the trees (not in forests but just bunches of them) is that no matter how low your terrain and object detail are they ALWAYS go to the highest LOD if you get close.

Yes, I have HD4870x2 with Q9550 at 3.4GHz. I really hope BIS gona do something with those buildings. I believe there is something wrong with those building models or game engine itself.

Arma 2 engine should take advantage of fast gpu's too, imho. It also should take advantage of 64bit OS, because Arma is really large scale game.

[SWAT_BigBear 0]

Do BIS have to abandon the 32-bit executable? Leave it in current state and start development of a 64-bit only game?

If Crytek can offer both exe's with their game, anyone can.

[tcp 10]

He may be more concerned that optimization will no longer be done for the 32-bit process. Yes, code changes will be made simultaneously on both. However, will 64-bit be the only way to run the game smoothly?

[oktane 0]

If Crytek can offer both exe's with their game, anyone can.

Heh, did it occur to you that it may be an issue of resources? BIS doesn't have a lot, Crytek/EA does. Stop treating BIS like they are some huge developer with 100 programmers and millions of dollars to spend on development (yes, AAA games cost MILLONS to develop, Crysis cost 22M), you can't draw parallels. Not to mention, if BIS was a huge developer, we wouldn't be having this conversation because the game would be a turd and there wouldn't be a chance in hell any 'suggestions' would be even read by the development team. Big studios and publishers have one purpose, to sell games to the largest market and make money to make their shareholders happy. Does this sound remotely like BIS's MO?

Not to mention their code base extends back before 2001. Do you think they want their game to run bad and have streaming problems, etc? If it was a simple fix like you make it out to be, they would have done it already.

---------- Post added at 09:52 PM ---------- Previous post was at 09:50 PM ----------

Yes, I have HD4870x2 with Q9550 at 3.4GHz. I really hope BIS gona do something with those buildings. I believe there is something wrong with those building models or game engine itself.

check out Kju's test addons that remove those LOD's and see if it helps ya.

---------- Post added at 09:54 PM ---------- Previous post was at 09:52 PM ----------

In the end we can only assume that the benefit does not outweigh the effort

to do it - for now. First the game should use the 2 GB to the max - if that is

useful at all.

Yep. Maybe Suma can shed some light on why it doesn't seem to use all 2GB. (or more with largeaddressextensions) If there was a hard coded limit, I bet keg could find it in the asm dump. :D

Edited October 24, 2009 by oktane

[Vipera 10]

There are some RAM disk programs for your tests.

RAMDisk "Enterprise" for Windows 2000 / XP / Server 2003 / Vista / Server 2008 / Windows 7

http://members.fortunecity.com/ramdisk/RAMDisk/ramdiskent.htm

Free Ramdisk for Windows Vista, XP, 2000 and 2003 Server

http://www.mydigitallife.info/2007/05/27/free-ramdisk-for-windows-vista-xp-2000-and-2003-server/

RamDisk 10 for Desktop Systems

http://www.superspeed.com/desktop/ramdisk.php

---------- Post added at 04:05 PM ---------- Previous post was at 03:31 PM ----------

Maybe it will help to increase I/O disk performance.

The following tells you tweak Windows Vista to more cache memory to use for NTFS files. Especially as many folders can be opened and closed this tool performance-enhancing work. It is this tool, it's important that you have enough internal memory. I would tweak with less than 2 gigabytes of internal memory does not apply.

For the Microsoft TechNet page where this topic is described click the link

this can be done with command prompt. We will be as follows:

1. Start -> Run -> Type "cmd" (enter)

2. The command prompt window opens and we fill it in: fsutil behavior set memoryusage 2 (enter)

to give us permission to Windows Vista(Win7) more (cache) memory to use for NTFS files.

4. You can check whether it actually went well with the following code to execute: fsutil behavior query memoryusage (enter)

5. The default setting back in place gives you the following code: fsutil behavior set memoryusage 1 (enter)

memoryusage Value

Configures the internal cache levels of NTFS paged pool and NTFS non-paged pool memory. Set to 1 or 2. When set to 1 (the default), NTFS uses the default amount of paged pool memory. When set to 2, NTFS increases the size of its lookaside lists and memory thresholds. A lookaside list is a pool of fixed-size kernel memory buffers that the kernel and device drivers create as private memory caches for file system operations, such as reading a file.

Using memoryusageValue

Increasing physical memory does not always increase the amount of paged pool memory available to NTFS. Setting memoryusage to 2 raises the limit of paged pool memory. This might improve performance if your system is opening and closing many files in the same file set and is not already using large amounts of system memory for other applications or for cache memory. If your computer is already using large amounts of system memory for other applications or for cache memory, increasing the limit of NTFS paged and non-paged pool memory reduces the available pool memory for other processes. This might reduce overall system performance.

This parameter updates the HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsMemoryUsage registry key.

Edited October 25, 2009 by Vipera

[frag85 10]

you would think a feature dealing with memory allocation on the OS level would be available and optimized for something like gaming to handle the data unloaded by a program by placing it into unused RAM, so if called upon again it would be loaded from the cached ram to the program ram, and not the hard drive.

this would be essentially like using ramdisk type software with your frequently read files loaded. the way i picture it is using something along the lines of softlinks to the cached ram. when called upon, if not in the cahced ram, THEN it goes to the HDD. i would imagine the overhead associated with these links would still allow for greatly increased performance, same as the case with ramdisk.

yes BIS has to work on how data is loaded from the hard drive, but if Windows had a feature like state above, it would greatly help A1/A2 and other programs (especially 32bit) that frequently read off the hard drive.

as far as large memory addressing, is it really that difficult for a developer to add? i remember the fix for the Project Reality mod with BF2 was fairly simple (if anyone knows the problem with CTD's with the highest texture levels and the game running out of memory while loading large/complex maps).

EDIT:

4. You can check whether it actually went well with the following code to execute: fsutil behavior query memoryusage (enter)

5. The default setting back in place gives you the following code: fsutil behavior setmemoryusage 1 (enter)

I'm using Win7 64-bit, and have not done any tweaking, and my default value was 0. MemoryUsage = 0

edit2: apparently this was just 'not set' before, as 0 is not an option. running armamark, it seemed to be smoother with 1 though, and used slightly more system ram (~50mb).

Edited October 25, 2009 by frag85

[bono_lv 10]

check out Kju's test addons that remove those LOD's and see if it helps ya.

Are those addons allowed on multiplayer? Asking because MP is only thing I play.

[liquidpinky 11]

Are those addons allowed on multiplayer? Asking because MP is only thing I play.

RamDisk doesn't effect multiplayer at all if you have Vista, I have also seen non Vista users using it like an addon and they were not kicked from the servers I was on at the time.

I primarily play Warfare and it works great for it.

[oktane 0]

Are those addons allowed on multiplayer? Asking because MP is only thing I play.

Every server is different, that's not possible to answer. Each is hosted by a different group or person and has it set up their way. Besides that, the addon is for testing at this time afaik, I don't think I remember seeing server keys.

PS: Here is the thread for kju's test addons http://forums.bistudio.com/showthread.php?t=88899 They may or may not be signed with the PROPER key, which admins should allow if they are smart. :D

Edited October 25, 2009 by oktane

[evilfury 0]

So ramdrive is only usable for arma 2 if you have 4GB+?

[Leon86 13]

RamDisk doesn't effect multiplayer at all if you have Vista, I have also seen non Vista users using it like an addon and they were not kicked from the servers I was on at the time.

I primarily play Warfare and it works great for it.

He was talking about the proper-mods I think, not the ramdisk thingy (and you are indeed right, you are using the default files and sign key's with ramdisk so the server doesn't see it as a mod)

So ramdrive is only usable for arma 2 if you have 4GB+?

you can have a ramdisk with any amount of ram. Arma 2 may use up to 2 GB of virtual memory (that is part actual ram and part data on pagefile). If you make your ramdisk too big you dont have a lot of memory left for applications and windows will start using part of the page file (on your harddisk) instead of (the much faster) ram. Arma2 will rarely use much more than 1gb (let's say 1.5 to be safe) windows uses about 1GB on my machine. That will leave about 1.5 GB for your ramdisk, enough for structures.pbo and buildings.pbo, you'll see a massive improvement with just those.

Edited October 25, 2009 by Leon86

[SWAT_BigBear 0]

much uncalled for rant...

Back up......I was only pointing out that "it is possible" to have both 32 & 64 bit apps within a game, to the guy I quoted.

My post had absolutely nodda damn thing to discredit BIS!

Suma had all ready pointed out the 64bit issues a long time ago.

http://forums.bistudio.com/showpost.php?p=1267018&postcount=28

Edited October 25, 2009 by SWAT_BigBear

[oktane 0]

Back up......I was only pointing out that "it is possible" to have both 32 & 64 bit apps within a game, to the guy I quoted.

My post had absolutely nodda damn thing to discredit BIS!

Suma had all ready pointed out the 64bit issues a long time ago.

http://forums.bistudio.com/showpost.php?p=1267018&postcount=28

Sorry, I will lay off the crack pipe.

Doesn't a 64 bit app end up using more ram due to the double width of the addresses? So it's almost as if once you make the jump to a 64 bit exe, then you're committed and your minimum ram amount goes up? Your data would remain the same, but all the 'header traffic' (requesting data, storing data) would be longer.

I'm amused that in the Far Cry/Amd64 case, the improvements were software restricted to 64 bit cpus and used as a marketing vehicle, lol.

I have been running my ramdisk for the past few days and its been cool, but last night I've been getting some odd 'freezes', I think because of my ramdisk, it makes windows a bit upset. I have 8gb (if I add a 4th stick) and I had a 4gb ramdrive.. But when windows boots, it says 'oh you have 8gb, i'll optimize for that', then suddenly the ramdisk comes on and sucks up half of it. I tried all kinds of ways to disable windows usage of the upper 4gb (using bcdedit) and then telling the ramdisk software (ramdisk plus) to use the 'unaddressable/unused' ram, but it doesn't work in windows7 x64, or my motherboard is not compatible with creating separate NUMA nodes because of interleaving?

If someone had a compatible system, I'd guess that the best thing to do is tell windows to self limit itself with BCDedit, then use the ramdisk software to address and use the now unaddressed ram. Best of both worlds.