Identical file's in ArmA PBO's

[Locke@Germany 0]

Hi,

after playing a little bit around with the ArmA pbo's i've noticed that many files seemed to be identical. I've extracted all pbo's

in the Addon folder and started comparing the files with doublekiller for their size and CRC.

The result is that almost 4000 file's with about 314 MB apear to be totally useles.  

A few examples:

Air.pbo\AIM9xSidewinder.p3d

identical with

Air.pbo\Sidewinder_proxy.p3d

Air.pbo\Data\Anim\Mi17_Cargo02_V2.rtm

identical with

 

Air.pbo\Data\Anim\Mi17_Cargo02_V6.rtm

The file Sara.pbo\Data\Layers\m_000_000_lco.paa has almost 1200 copy's in the same folder, from m_000_001_lco.paa to

m_042_042_lco.paa, resulting in 1171 file's with a total size of 195 MB that ALL HAVE THE SAME CONTENT.

I would be happy if BIS would remove the identical file's in future Updates, as i don't think it helps the performance of a game that already is very hardware demanding.

Locke.

[Maddmatt 1]

I highly doubt that is causing performance problems.

[ebud 18]

There are what seems to be a huge amount of unused textures as well. Lots of Xbox leftovers that I can't seem to see being used anywhere.

[Maddmatt 1]

There are what seems to be a huge amount of unused textures as well. Lots of Xbox leftovers that I can't seem to see being used anywhere.

They wouldn't have left the original islands by any chance? Guess not. Maybe someone could find a use for the files.

[Jack-UK 0]

The CWR team are working on a port for the original islands =) I dont think they would be included in ArmA... but they will be porting over CWC + its campaign

Though it is interesting why there are so many unused/duplicate textures...

[dm 9]

Though it is interesting why there are so many unused/duplicate textures...

Part and parcel of having more than 1 person working on the art. That or having a logically poor file structure/naming convention

[inveigle 0]

This explains why the Chech version is over 4 gigs and the German only a little over 2.

[moricky 211]

There ar not all identical textures. The ones with s_ before numbers are parts of satellite map. There is 1849 of them which is 43x43 (s_000_000_Ico.paa to s_042_042_Ico.paa) and it's same with the ones beggining with m_ - I think they are connected somehow.

[suma 8]

I agree this might be somewhat inefficient, but other than using some disk space, real performance impacts should be almost zero and it certainly is not a serious problem.

[charliereddog 9]

Can't say fairer than that Suma.

[ThePredator 0]

But removing the weapons.pbo sound folder (the one NOT in the Data folder) greatly decreases compilation time.

[Locke@Germany 0]

I agree this might be somewhat inefficient, but other than using some disk space, real performance impacts should be almost zero and it certainly is not a serious problem.

the performance impact should be zero ?

As i don't have the tools to test i can't say if all these duplicate textures and *.rvmat file's are even used.

But if they are used wouldn't this result in a lot of wasted VRAM too and not only wasted Diskspace ?

For example: a 3d-Model uses 3 textures A,B and C. Another model uses the textures A,D & E. Let's assume texture D is identical to texture B, just a different name.

If these two models are used ingame the VRAM has to load and contain texture A,B,C,D & E instead of A,B,C & E.

Wouldn't this result in longer loading times and higher memory consumption ?

Locke

[Maddmatt 1]

I agree this might be somewhat inefficient, but other than using some disk space, real performance impacts should be almost zero and it certainly is not a serious problem.

the performance impact should be zero ?

As i don't have the tools to test i can't say if all these duplicate textures and *.rvmat file's are even used.

But if they are used wouldn't this result in a lot of wasted VRAM too and not only wasted Diskspace ?

For example: a 3d-Model uses 3 textures A,B and C. Another model uses the textures A,D & E. Let's assume texture D is identical to texture B, just a different name.

If these two models are used ingame the VRAM has to load and contain texture A,B,C,D & E instead of A,B,C & E.

Wouldn't this result in longer loading times and higher memory consumption ?

Locke

Now you are just making up problems that probably don't exist. I don't think this is anything to worry about. Why would two completely different objects share the same textures? That wont happen often.

[suma 8]

The performance impact is very slight, as resources are loaded in RAM/VRAM only as needed. Unless you see both of the objects using the same texture in a close bitmap (which is almost impossible), their textures are not fully loaded. As the duplicates present only a small fraction of the total resources, there can be some wasted space even in VRAM, but nothing dramatic (a small fraction of total VRAM).

[baddo 0]

Hi all

It would be ideal if there were a resource manager in the game which would allow only one copy of resource data to be loaded into memory. From what Suma says we can say there isn't such a strict system in place.

One reason I can think of why the system is like it is can be that the resource manager relies only on checking the full path of the resource file, and it is not checking if the particular file is already loaded. When I say full path I mean the path from the game's root directory to the resource file. Then there will be multiple copies of the same resources in memory of course: if you put texture.paa to x.pbo and to y.pbo and load both of those pbo's then texture.paa would be loaded two times because their path is different even though the file has exactly the same name and data. I assume from what Suma said that this is how the resource system in ArmA (and in OFP) works. There could be an unjustifiedly big performance hit during loading of the resources if you wanted to always check if resource data you are about to load is already loaded among the thousands of resource files. ArmA streams resources from hard disk drive when needed and it is easy to say that you don't want to put too many checks in there to slow the streaming.

This doesn't still tell much about why there are multiple copies of the same files (equal filename and content or just content) distributed with the game. I happen to have experience in handling very large 3D model assemblies. I'm talking about thousands of 3D model entities combined under one product file. Multiple entities in such an assembly can reference to the same 3D model file. In an ideal data hierarchy we would only have one single file for one single 3D shape. "Why it is not always strictly required in practice, then?" you ask. In practice it is often not a good thing to aim for, let me explain why.

After working with large model assemblies for some time it became obvious for me that there can arise serious problems if you try to enforce a strict rule that you must at any one time have only one 3D model file of a certain 3D shape in the data hierarchy. I will tell you what the problem is with such a strict rule. If assemblies A and B contain exactly the same 3D models (the models can be positioned and orientated in the assemblies differently) it sounds ideal to share the same model files between the two assemblies to conserve hard disk drive space, and RAM when loading both of the assemblies in the same time. But what if we need to go and modify a certain 3D model in assembly B, how can we know that the modified model still fits into assembly A? We must open and check assembly A of course! And if the modified model is used in many other assemblies too, we need to check them all. That isn't what you want to do when you only modify a file just a little bit, I am sure of that. Practically this means that you can not go and just modify the model file without knowing all the places where the model is used, and without checking if the model still fits all of those places after the modifications are done. It is much much easier, safer and faster to just make a new file and modify that. Then you can tell for sure that you didn't break any old assemblies with your modifications. And if you use an existing 3D model (which you need not modify) created by someone else, how can you be sure that your model assembly will stay correct over time? You can't! So the problem with sharing data between many entities is that it can create tremendeous dependency problems which can be a nightmare to fix after the damage has been done. Even if you modified a file "just a little bit" it doesn't matter if it breaks the whole thing. In fact, in the industry my experience is from, it is not at all exceptional to create new copies of all the existing model files when creating a new assembly, even if only a few of the files need to be modified. That way maximum safety and flexibility is achieved. Standard model libraries are a way to improve the situation. In standard model libraries no one is allowed to modify a model after it has been accepted into the model library. But I think data files which are never allowed to be modified after initial release doesn't fit well into games, especially if we expect the game to be patched into better. It is very possible that there are duplicate data files in the ArmA distribution just to avoid dependency problems which would create unjustified maintenance workload.

I'm sure you can imagine how all this fits into the data hierarchy used in OFP and ArmA. The same problems are there if you require that only one single file of one single shape/texture/whatever data (which is used in many different places) is in the distribution. It would be desirable though if the BIS developers aimed towards the ideal data hierarchy, even if it means more work for them. In my example the files are not distributed to end customers but with BIS products it is so. There might be other main reasons for BIS to do their system like they have done it, but avoiding a data hierarchy with a lot of dependencies between different data tree branches is for sure a good bet.

Best Regards,

Baddo.

Edit:

...identical data in the same directory? Then the theory of avoiding dependencies between different data tree branches is not that likely to be the reason! What is up with that, BIS...?

[baddo 0]

After reading about game development I have come to a conclusion that this issue of duplicated files is because ArmA is based on the OFP:Elite game which is an XBox game. On XBox the game is intended to be run from the CD. The seek times have been reduced by duplicating content on the optical disc. That has stayed in the way how data is arranged over evolution into ArmA. This is my guess.