PBO Memory usage

[ebud 18]

Since this doesn't fit neatly into one of the editing boards I thought I'd ask here.

Say I make a mission or an addon that needs soemthing very little from a HUGE pbo, how is the rest of the pbo "stored"? Does OFP unpack the entire pbo into RAM just to get one little object? Does the rest of the unused pbo still hog memory resources? Or does it not really matter at all?

For example. Say I for some reason made a car and it used the headlight tetxures from some other addon. The car pbo is very tiny but the pbo it references is say... 200megs compressed. Now each time that car is used and references that pbo... what happens "under the hood"?

I can't seem to remember if this info has ever been posted. It probably has, but I can't seem to find it.

[-)rStrangelove 0]

How are the soldiers organized ?

Officers and soldiers aren't in separate pbo's, are they ? Thing is: memory usage has a great effect on spawning units - if spawning an addon you don't already have in your mission (not in memory), opf loads it into memory after the 'createunit' command, resulting in the unfamous spawn lag.

If you already have normal soldiers in your mission, spawning more won't cause any lags at all.

2nd thing is: even if you have soldiers in your mission, spawning an officer creates a spawn lag as well, although i don't think an officer uses that much of special textures.

So i think opf only looks for specific things in PBOs and throws away the rest. Just my opinion though.

Another thing i noticed lately is that opf wipes unused addons away from its memory within seconds - i created and deleted A10 planes, and everytime i got a spawn lag because the former A10 was gone.

When i placed an A10 over the ocean as a 'ref' object, the lags went away.

Yup.  

[Jackal326 1182]

OK, so we've established the spawn lag issue, but thats not really what Ebud is asking. He's asking more along the lines of how PBOs are loaded i.e. do they simply load the resources they need from them for a particular addon, or do they load the entire PBO?

[NeMeSiS 11]

OK, so we've established the spawn lag issue, but thats not really what Ebud is asking. He's asking more along the lines of how PBOs are loaded i.e. do they simply load the resources they need from them for a particular addon, or do they load the entire PBO?

Well, multiple units can be stored in a single PBO, but if you preload 1 unit and spawn another, you will get spawnlag if yuo spawn the not-preloaded unit, so it only loads what it needs from the PBO.

Unless i understand you wrong...

[sanctuary 19]

I think Ebud is asking if OFP open/extract/whatever exact english term it is, the whole PBO content in the memory when it needs even a single unit from it, or is OFP capable of just getting only the needed files from the PBO, always for that single unit, in the memory.

About this good memory usage question , i would like to add another :

After reading in the wiki

Quote[/b] ]The config.bin file is a binarized version of the config.cpp file. For more detailed information see Config.cpp.

Sometimes binary form configs are referred to as 'encrypted'. This is misleading, as the encoding used for config.bin is not an encryption, and it was never intended to be, The purpose of the binary form is to provide a representation which can be read by the game engine as fast as possible, avoiding any text parsing.

Does that mean that a pbo that feature a binarized config.bin for the addon, instead of the usual config.cpp will be better performance and memory wise ?

[vektorboson 8]

Hi,

I've programmed my own PBO-loading routines, and I guess that BIS has done it in a similar manner:

I'm getting a file descriptor to the PBO and I'm reading the header in. From the header you get the list and the length of the included files.

From this you can calculate the offsets into the files, and map it to the filenames.

Now every time you want to get a file from the PBO, you just check the filename and jump to the offset in the file and read on (or decompress).

In my case, the PBO does not use any memory except the filename-offset-table.

As BIS aren't amateurs they have it done in a similar or even better way.

[ebud 18]

Thanks for those quick replys.

Let me put it as simply as possible.

I have a pbo with 20 vehicles. I place one vehicle on the map and it uses, say 10% of the pbo as far as model size and textures. What happens to the other 90%?

If I had 20 single pbos for each of those 20 vehicles would it make a difference?

I want to know if one huge pbo is less efficient that several pbos, especially since there would never be a time when everything from the pbo would be used in the same mission.

[ebud 18]

Hi,

I've programmed my own PBO-loading routines, and I guess that BIS has done it in a similar manner: ......

ok, thats over my head, but thanks

Does that mean that only the relevant files from within a pbo are used when called for?

So it would make no difference if I for example had 5 500 meg pbos, and made an addon that used a couple of textures from each pbo? It would run as smooth as if I just took those few textures out from those pbos?

[nephilim 0]

well im no coding expert but i think ofp will have to extract the textures from each pbo

and a 500 mb pbo wil prolly take some more time than a 10 mb one...

think best purpose is to extract the used textures from the original large pbos and put them into the new one..

also this helps with redistribution..

cuz downloading a ass huge addon for just a single texture is :S..

however i think ofp preloads all textures while mission loading

so ingame there shouldnt be any slowdown or lagg whatever

but only the mission loading time might take a while?

not entirely sure.. rough guess *cough* ... hiding..

[-)rStrangelove 0]

OK, so we've established the spawn lag issue, but thats not really what Ebud is asking. He's asking more along the lines of how PBOs are loaded i.e. do they simply load the resources they need from them for a particular addon, or do they load the entire PBO?

Well, by bringing up the spawn lag issue i wanted to say something like:

"If Opf would really load all resources from a huge PBO into memory because a single addon is used from that file, why do i get spawnlags when i spawn an officer besides a normal soldier ?"

I think that sums it up. I'm not an expert when it comes to PBO tricks & tips, but i'd say that chances are high many many little PBOs offer more performance than 3 huge PBOs, because OpF searches for single files in the huge files.

Errrr... yes. Or something like that ..

[deanosbeano 0]

hmm, from what you asking, where and how would it effect ofp.well it would happen at the point of starting flashpoint @mod= blah x5, would take longer than @mod=x1. after that all things are equal ,until your mission uses all or most of the addons from @mod blah x5. this will cause lag during playing, however ,if you only use 1 car from 1 pbo in 1 mod folder, there will be no lag ,no matter if you initially loaded 1.3gig of addons on start up, but as said before, the initial start up ,will be affected.

[ebud 18]

Thanks again.

I was just using those as simple examples.

What I'm doing is running a mod in OFP, but I'll never look at or use 1/2 of the stuff that is in there, and the missions I play all the time barely use that 1/2. So what I'm hoping to figure out is if I took out the 1/2 I'll never use, will I get better performance.

Or down the road, later if I release a huge addon, would the performance be better if it were broken up into say... 6 pbos rather than 1.

Another example. Say someone made a pbo with US troops in it and you wanted to include every era, every camo scheme, every branch of the military. Now that 1 pbo would be huge. If you threw one unit on a map from this pbo, say a Modern era Navy Seal, would it make a difference if it was from that 1 huge pbo that contained over 500 models and textures, or if you had broken down that one pbo into many smaller pbos and now the only pbo being used was one of the smaller ones that just contained the Seal and needed textures?

See what I'm wondering?

[ebud 18]

however ,if you only use 1 car from 1 pbo in 1 mod folder, there will be no lag ,no matter if you initially loaded 1.3gig of addons on start up, but as said before, the initial start up ,will be affected.

What if that car used a tiny texture from each of those addons that add up to 1.3 gig?

Would each of those addons be "loaded into memory"? If so, what exactly is happening? Only the minimum files are found and used like I think was mentioned above?

I'm just trying to figure out how to optimize addon usage and figure out EXACTLY what will cause FPS loss.

One last example and I'll let this die.

Tonal. It's a hog, but why? Does every plant model+texture, every building model+texture and every ground texture get loaded into ram right when you load up the island?

Or an island using 1 barrakan model one AGS model and one model from the Tonal island object pbo. Does OFP load all of those addons into memory, each and every file? Or does it open the pbo, pick out what is needed and only loads those tiny files from each pbo into memory?

[deanosbeano 0]

nope only on initial startup of ofp. as for tonal, there are something like 250 - 300,000 objects in that map all loaded. all defined and all being calculated. the addons from the pbo are simply waiting to be called and whilst in ram waiting cause no problem, but once they extracted and placed on map, they become a source of cpu power.

viewdistance is theonly lag saver on tonal and play near the airport . but why dont you test , make a start up with multiple folders and make simple 4 tank againgst 4 tank on des island. calculate fps.

then make another startup , only using the pbo that has the tank.and see the diff

[Shinkansen 0]

Thanks for those quick replys.

Let me put it as simply as possible.

I have a pbo with 20 vehicles. I place one vehicle on the map and it uses, say 10% of the pbo as far as model size and textures. What happens to the other 90%?

If I had 20 single pbos for each of those 20 vehicles would it make a difference?

I want to know if one huge pbo is less efficient that several pbos, especially since there would never be a time when everything from the pbo would be used in the same mission.

Yes and no,

in one big pbo file several addons can use the same textures, but big pbos will be bad for performance. You can see this sometimes on maps that uses only a small number of trees, houses or other objects from a group of big pbo files. If all the map objects are regrouped in only one pbo file, deleting all the stuff that is allocated but not used the map will run faster.

[vektorboson 8]

well im no coding expert but i think ofp will have to extract the textures from each pbo

and a 500 mb pbo wil prolly take some more time than a 10 mb one...

Well, depending on how BIS implemented the PBO access, there wouldn't be a difference.

If OFP rereads the PBO-header everytime a file is desired, then it will probably take longer.

If OFP caches the offsets to every file, then there should be no difference between a 500MB an a 10MB file, because all irrelevant data is just skipped and only the desired file is read.

[-HUNTER- 1]

I dont have anything usefull to add to this topic, but I think this is a great idea, and well worth looking at.

I also use many big .pbo's from which I only need a little amount of addons, so a solution would be good for many players. And I think if you need something from a large .pbo it will load the whole thing, instead of just the file you need... So making lots of smaller .pbo files would increase performance!

*goes off to talk a bit more about this on msn...

I love this game, after 5 years ppl still get brilliant ideas

[ebud 18]

So I'm seeing conflicting answers here. I figured there would be, and no real definate yes or no answers.

Maybe I'm still just not being very clear, I do that alot.

I'm messing with several pbos for VTE, some of them quite huge. Some of them have things in there I will never use, especially the island pbo which has many nice islands, but I'll maybe use one or two. Same with the avatar pbo I'm making. It has expanded enormously and not everything will be or should be used in every mission, but since they are all in one pbo does that affect performace?

For my personal sp playing time I thought I might just either take out the unneeded files, or in the case of the avatar pbo, break it up into smaller pbos which only contain the models and textures needed, such as having only Aircav models and textures in it's pbo, so that if a mission uses the Aircav only then only those will be loaded into memory and not SF units, marine units, Seal units, etc. I could test this forever and never be sure if what I was doing made any difference unless it was a HUGE difference. I just don't know if it would make any difference other than saving HD space. And from the looks of the posts many others aren't totally sure either.

Since there are seemingly conflicting answers I'm still not sure I know the answer. Or if there even is one.

[nephilim 0]

Well its seems to be your choice..

however id say if you want to share that addon with the community it would be better to include the textures from teh big pbos in the new one just for the sake of 56k users..

its pain in the arse to dwl a cool addons and need to find out you need 3 huge ones for some tiny textures..

and after all moving some tex ans redoing paths isnt a big deal..

good luck man

[ebud 18]

But like I said, that was just an example as I would never in a million years do that.

[Shinkansen 0]

To bring up a new suggestion, if you want to regroup all the material in two or three new pbos for islands, missions, sound files and addons you have the change to control and/or redo/ reassign the lod numbers, fire and geo lods to save performance.

This of course is very time consuming...

[Jackal326 1182]

...

Maybe I'm still just not being very clear, I do that alot.

...

Since there are seemingly conflicting answers I'm still not sure I know the answer. Or if there even is one.

Its not that you're being unclear, its just that the people with the correct answer (BIS themselves) are yet to respond. From a community point of view its mostly just guess-work and supposition.

[ebud 18]

True, true.

I guess I'll just assume that for my personal addons, if I take out all the stuff I don't need or will ever use, somehow, someway, it'll help performance.

<edit> maybe someone can answer this simple question. When ofp starts with or without the -nomap option (that is another option that escapes me) how are addons handled? Then you load a mission, and that mission uses 1 addon pbo, what exactly happens with that pbo and it's contents?

[action man 0]

Well here's my ten pence worth I have my own Y2K3 customised config,like E bud say's sometime's say for instance a new lsr special forces addon comes out and i only want to use 1 unit of the addon, i do so and have done for multiple other addon's (vehicles plane's choppers ect) Now in my expirience when in game their is no additional lag or memory usageby adding the whole PBO into my addon's.However when loading up the game and you see that bar thing that move's from side to side,now that does take longer.

[Gökhan - OFP TURKISH UNION 0]

So;

1) Big pbo's cause start-up lag. "you see that bar thing that move's from side to side, now that does take longer."

-action man

2) Ofp loads things into memory "when" they needed not once. Besides It wipes out unused stuff rapidly. This is a great mechanism running big pbo's smooth. I made many tests and big pbo's are not bad during playing but start-up. The difference is not that big... Putting the reference textures from someone's big pbo into your own little is of course better for both the loading issue and the community.

3) Most of the times big pbo's (let's say 500MB pbo) get fragmented and this increases the waiting time referred to start-up time mostly.

So don't worry

Gökhan

Turkish Union Mod

www.turkishunion.com