SVN/CVS Discussion

[hoz 0]

I don't know much about this. But I think we have to make some assumptions to keep the progress moving.

My first assumption is:

*BI will not likely host the cvs/svn.

[raedor 8]

Meanwhile the assumption is proven wrong.

BI has a server for us Probably with SVN.

[feersum.endjinn 6]

Next thing to do is to agree on repository structure, in my opinion it should be same as PBO file structure will be.

As what the PBO structure will be, I'd propose structuring it in so that every vehicle type has their models and textures in one pbo (tracked, wheeled, characters, everon.pbo), sounds in another (sounds.pbo) and configurations in separate .pbo. That way each part of a team can do their changes in one specific directory and test their parts only by rebuilding single .pbo at time.

[raedor 8]

I'd suggest to use the same structures as BI did with the ArmA content. So basically what you said, but also the configs in the respective pbos, as then even the config work can be split easily.

[kegetys 2]

Single pbo with config would imo. be better, as it would make working with the config easier (no need to repack the large pbo's every time), it would allow having it all in one addon such as Cold_War_Rearmed instead of a dozen different cryptic ones, and it would make small patches easier to release as you do not necessarily have to include the whole large pbo if there's only some config tweaks made.

The config can (and should) be split into multiple files to make working with it easier.

[raedor 8]

Well okay. Then we'll do it that way. I already started making the base configs. Which splitting makes most sense in your opinion? Wheeled, tracked, air... or West, east, ... or some other method?

[shinRaiden 0]

Lessons learned :

You will have bugs, it is guaranteed. Nobody does perfect work. If you want the content to be reasonablely managable and patchable, A) do not use the CWC data3d model, and B) do not use the ArmA shallow ArmA data model.

Arma content is only stacked one level deep, ie p:\ca -> p:\ca\weapons

The result there is all weps are dumped in together, and to change one property on one line of config code (initial tonal release et al), would require a full new pbo to meet signability.

Using the alternate deeper content model would be perhaps more manageable. In this case, you'd have something to the effect of p:\ca\vehicles\air\helicopters\mil_mi24\cz_mi24a.

However, if the BIS tools will be made available in their present form, the OFP development evolution of seperating configs from data in seperate pbo's (BAS's pavehawks, tonal, CBT's work, etc) is not directly possible due to packing requirements. The configs must be present with the content that they describe. However, this can be dealt with by utilizing the deeper structure as listed above.

[feersum.endjinn 6]

For OFP/Resistance MLODs there was limitation of 32 characters for texture path which was one reason why OFP addon directory structures tended to be pretty shallow (and nondescriptive). Is this limitation still there in Oxygen 2.0?

Alternatively, do P3D's, rvmat's and textures have to be in same PBO? Would it be possible to have structure where config, P3D and rvmat's were stored in single PBO and textures in another, as textures tend to be most space consuming part of addon (especially more so in ArmA). At least for sounds I don't see why they should be in same pbo as p3d's or configs...

[shinRaiden 0]

The path string limitation appears to have been resolved in next gen content. One case example I'm aware of in testing is 58 characters long expanded, not counting individual file names.

Configs and models must be together, if the textures and rvmats are located elsewhere and not below the model location, a reference textures.p3d index model is required.

Sounds are generally more common shared assets, and would thus be located higher up at shared common hierarchy levels.

[feersum.endjinn 6]

Hmm, completely I forgot about binarize requiring PAA header info for proper alpha channel blending, scratch that idea then.

What we did with FDFmod was that during binarize process we copied config.cpp into p3d directory and removed it before packing model as pbo. Of course that approach has it problems as well, if for some reason script failed to remove config.cpp you sometimes ended up with multiple versions of same config in different pbos and it was bit random which one got loaded first...

[raedor 8]

So due to the new signs it is impossible to patch a pbo, you always have to red/l the whole pbo? If that's true -- and I guess it is when you say that -- many small pbos aren't a bad solution. That'd also make it ultimately easy to split work...

[feersum.endjinn 6]

So due to the new signs it is impossible to patch a pbo, you always have to red/l the whole pbo? If that's true -- and I guess it is when you say that -- many small pbos aren't a bad solution. That'd also make it ultimately easy to split work...

You can patch the .pbo just fine, you just need to supply new signature file (I assume bisign files hold the authentication signature).

I'm not sure if current pbo tools like cpbo handle SHA-1 authentication block at the end of the pbo properly yet but it shouldn't be too much of an issue.

Whatever structure we go with only affects development process, final releases are better off made as full complete downloads since patching system just confuses users unless you have idiot proof installer but writing one is a lot of work (just witness trouble people are having installing 1.01 at times).

[raedor 8]

So due to the new signs it is impossible to patch a pbo, you always have to red/l the whole pbo? If that's true -- and I guess it is when you say that -- many small pbos aren't a bad solution. That'd also make it ultimately easy to split work...

You can patch the .pbo just fine, you just need to supply new signature file (I assume bisign files hold the authentication signature).

Are you sure about that? As if it is that way, everybody can pack pbos without needing a new key... or not

[shinRaiden 0]

The issue is that if differential patches are done client-side, then they need to be done in a manner than will guarantee the results every time to be matched against the server signs. The dubiousness of that successfully happening without strong-arming the client is marginal enough that pre-packed distros is imho a much more reliable option.

[suma 8]

patchable, A) do not use the CWC data3d model, and B) do not use the ArmA shallow ArmA data model.

Arma content is only stacked one level deep, ie p:\ca -> p:\ca\weapons

Not exactly. The pbos are stacked one level deep, but there are still additional folders hidden inside of the pbos.

There are two separate questions:

1) folder structure

2) pbo structure

While the two are related (you cannot have a pbo without a corresponding folder) they are not exactly the same (pbo can hide multiple folders)

From performance point of view it is good to have only a few pbos, as this minimizes number of handles which need to be open.

[suma 8]

You can patch the .pbo just fine, you just need to supply new signature file (I assume bisign files hold the authentication signature).

Are you sure about that? As if it is that way, everybody can pack pbos without needing a new key... or not

Not everybody. The creator of the pbo is able to patch it, as he is able to provide a signature of the patched pbo. If anyones packs pbo without updating the signature, he will fail the signature test when enabled on the server.

Note: as we did not release PBO packing tools yet, it is quite possible the unofficial tools you might be using now produce something which is assume a "hacked" PBO - a PBO which pretends to be signed, but its Hash value stored in the PBO is wrong. This is also checked by the server when enabled, however the check may be slightly delayed and the results of the check may be different (server may be notified about you as a suspested cheater).

PBO packing tools will be one of the first tools we plan to release. Until then, I guess this project can do without any signatures / hash values.

[raedor 8]

So what we have to discuss about is patchability (many small pbos) vs performance (some big ones)? Or did I overlook something? Well, if that is not the case, it only depends on how big the performance loss is and how big the big pbos would become (as if they're smaller than 10? 20? 50?MB patches are useless...).

[shinRaiden 0]

patchable, A) do not use the CWC data3d model, and B) do not use the ArmA shallow ArmA data model.

Arma content is only stacked one level deep, ie p:\ca -> p:\ca\weapons

Not exactly. The pbos are stacked one level deep, but there are still additional folders hidden inside of the pbos.

There are two separate questions:

1) folder structure

2) pbo structure

While the two are related (you cannot have a pbo without a corresponding folder) they are not exactly the same (pbo can hide multiple folders)

From performance point of view it is good to have only a few pbos, as this minimizes number of handles which need to be open.

I know there's other misc folders embeded at each level (like data, anims etc) my point was more abstract for explaining the differences for people more familiar with the OFP content organization, and less familiar with BIS's data organization.

If you use a deep tree, you can pack at any level, ie at air, or at helicopters, or at mi-17. If you use a shallow tree like ArmA, you're stuck in that model.

[suma 8]

So what we have to discuss about is patchability (many small pbos) vs performance (some big ones)?

It depends on what kind of patchability you mean. We are quite able to provide patches for those 300 MB PBOs we have in ArmA, and I guess there is no reason Rearmed could not do the same. Smaller PBOs might make data packing and work distribution somewhat smoother. Moreover, the borders of the PBOs can be changed later - you can merge multiple PBOs into one (collapsing folder one level).

[raedor 8]

So what we have to discuss about is patchability (many small pbos) vs performance (some big ones)?

It depends on what kind of patchability you mean. We are quite able to provide patches for those 300 MB PBOs we have in ArmA, and I guess there is no reason Rearmed could not do the same. Smaller PBOs might make data packing and work distribution somewhat smoother. Moreover, the borders of the PBOs can be changed later - you can merge multiple PBOs into one (collapsing folder one level).

Okay, thanks for clarification. So we'll start with the system shin proposed, many small pbos.

[mikero 79]

FYI the eliteness tool currently allocates a 21 byte ending block to arma pbo's. Zero Filled. Not sure if Kegs does same.

It is assumed (by me at least), that signing could be a separate process to the creation of the pbo itself. A two stop shop.

It would allow fractured pbo's, multiple authors, to merge into one, final product. So called, patching, or derivates of that idea.

Both Eliteness and Kegs account for the prefix, or folder depth we're discussing here

http://andrew.nf/ofp/tools/eliteness2.08.rar

[feersum.endjinn 6]

To get things going forward, I propose following directory structure. It allows us to split things into smaller PBOs in development phase but with option combine everything into single huge .pbo or ~9-14 separate pbo's in release.

I'm not sure if Oxygen 2.0 still has 31 character limit for texture names but even then path like "\cwr\units\tracked\common\" leaves 5 characters for texture name, which should be enough for somewhat descriptive name.

Quote[/b] ]\cwr

Main root. This directory should have nothing but subdirectories as each subdirectory is

builts as its own pbo (with option of using single CWR.pbo in release if so desired)

\cwr\units\

All units go under this directory. This directory should not contain any data.

\cwr\units\tracked\

.p3d's and config.cpp for tracked vehicles goes here

\cwr\units\tracked\common\

all textures shared by multiple models go here

\cwr\units\tracked\bmp\ (example)

all BMP-specific textures go gere

\cwr\units\wheeled\

\cwr\units\air\

\cwr\units\men\

\cwr\units\weapons\ (both handguns and vehicle weapons are defined in config.cpp located here like in ArmA)

These follow same structure as tracked directory (config and p3d in the root, textures in subdirectories).

\cwr\static\

For buildings and such used by islands

\cwr\sounds\

\cwr\sounds\wheeled\

\cwr\sounds\tracked\

\cwr\sounds\air\

\cwr\sounds\men\

\cwr\sounds\weapons\

\cwr\sounds\music\

\cwr\sounds\environment\

\cwr\sounds\impact\ (for explosions, hit[] sounds etc)

Sounds directory structure mostly follows same division as p3d's and textures.

\cwr\anims

For whatever animations that cannot be reused from ArmA.

\cwr\islands\eden

\cwr\islands\abel

\cwr\islands\cain

Island data. These can be built either into big cwr.pbo, islands.pbo or separate eden/abel/cain directories using prefix system. In development phase it's probably best to use separate pbo for each island.

[Jahve 0]

Sure we shouldnt be using tags?

CWR_Eden

CWR_Abel

CWR_Cain

CWR_Noe

CWR_Intro

etc..

CWR_SoldierWB

CWR_OfficerW

CWR_Civilian2

[feersum.endjinn 6]

What I am talking about above is directory structure (and PBO prefix system). What classes are called is irrelevant regarding CVS/SVN as it happens completely inside config.cpp.

But yes, classes should probably be tagged with CWR_ .

If you are talking about PBO names - what actual PBO names are doesn't really matter either as they are not related to in game paths anymore either due to PBO prefix system, all paths used in .cpp's are "virtual paths".

[raedor 8]

I like feers suggestion, some small changes (eg \data instead of \common) and we're ready to go...