PBO Hell

[Synide 0]

Since reading Celery's post on a naming convention for AddOn pbo's it's resurfaced and old issue I've always had with AddOn makers.

That being version control.

This problem doesn't surface as much with Mod's as they are a little more robust and managed a little better than the average standalone AddOn.

What I'm proposing here is a possible method.

Problem.

An AddOn maker produces a standalone AddOn say for example an Aircraft (tag_Aircraft.pbo) on a Monday.

A user downloads this AddOn and uses it. Over the rest of the week the AddOn maker makes several updates to the AddOn and does a 'release' on the Friday.

Rather than having to micro manage keeping up to date with the latest version of that AddOn and what it works with and doesn't work with we need a more automated method.

Solution. (Possibly)

Inside the tag_Aircraft.pbo the AddOn maker would place a text file.

The name of this text file would have to be a community agreed upon format.

I'd like to see it as just a guid with no filename extension.

eg.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

5AB917B6-0C0C-40CD-B572-6F90386D203E

the '5AB917B6-0C0C-40CD-B572-6F90386D203E' file would contain an agreed upon set of information about the pbo.

eg.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

This=5AB917B6-0C0C-40CD-B572-6F90386D203E;

Maker=2C6ED9C1-019A-45EA-9E3C-1060AE187C80=AAW;

Version=1.06;

Description="This AddOn is for a JumJum Aircraft"

There could be as little or alot of information placed in this file by the AddOn maker. But, it would have to be in a agreed upon format.

Also, this AddOn would be signed with the DSSignFile tool. Remeber, the DSSignFile tool will only define a pbo as being legtimate and un-altered. It, is not a mechanism for determining the uniqueness of this pbo.

Why.

So, why use GUID's.

Well, because they are easy to generate. The likelyhood of someone generating the same GUID for their ArmA AddOn is literally millions to 1.

Also, having a unique id within the given pbo will be immensely valuable.

Also, the benefit of having a text file inside the .pbo with a guid as it's filename is that one or more community SQL databases could store (based on the above example) the following information...

You could have a table of AddOn makers with a primary key on the AddOn makers guid.

eg.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

AddOnMaker                                                Name

D2403490-1FBF-4671-977F-BF8FD4428567       BAS2

2C6ED9C1-019A-45EA-9E3C-1060AE187C80      AAW

You could have another table of AddOns with a primary key of the AddOn guid.

eg.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

AddOn                                    Name                  CurrentVersion     Maker

5AB917B6-0C0C-40CD-B572-6F90386D203E     AAW_Aircraft.pbo      1.07               2C6ED9C1-019A-45EA-9E3C-1060AE187C80

A tool could be created that could easily look inside of a an AddOn pbo and if there is a file that looks like a guid with no extension the tool could just display the contents of that file.

Also, it could query a list of community DB's for that AddOn's guid and discover if the version of the AddOn it's looking at is the latest version.

It could also, display a list of available download sites of where to get the later version from.

It could show that this new version of the pbo now has a dependancy on another pbo and show a link of where to get it.

I know this all sounds a bit complicated but I'm trying to detail it in simple terms and really for those of you who are programmers by trade you know this would be blindingly easier to create.

The only hard part is having community websites willing to host an AddOns Database that can be queried by a standalone tool.

The AddOn maker would also run the tool and when they create a new or alter an existing AddOn pbo they would just fill in the appropriate details which would write them into the guid file stored inside the pbo. then they would pack it up and upload it to an AddOns hosting site.

The community SQL Databases that would be storing the AddOn Signatures could be one in the same as the AddOn hosting site, but not necessarily. It would probably be best if they were.

When the AddOn maker uploaded them, the AddOn Signatures DB could also be updated.

Summary.

In summary having at the very least a community register of guid's of AddOns would be hugely beneficial on so many levels.

It would mean all AddOns would have an instant version control right across the board.

Cheers, Sy.

Comments and or ridicule is expected.

PS. Apols for using BAS as example text... it's just an example.

[abs 2]

...OR you could just name the addon Tag_Addon_v2.pbo, and skip all the malarky of the middle steps.

Abs

[KeyCat 131]

...OR you could just name the addon Tag_Addon_v2.pbo, and skip all the malarky of the middle steps.

And then you have to redo any missions that where using the Tag_Addon_v1.pbo

Naming a addon-PBO with version number is not a good option IMO.

/KC

[abs 2]

Good point.

Well, I guess it's up to the readme to tell us which version number is being used. *nods*

Abs

[Synide 0]

@Abs... true, but for an AddOn hoster and for end users they are completely reliant on the AddOn creator having good management of there AddOns.

And, based on the OFP days this isn't a strong suit.

Also, if the AddOn maker ever wants to change the naming convention for 'Tag_Addon_v2.pbo' they could do so without impacting anything.

As the AddOn hosting service would be referening the guid of the AddOn and it wouldn't make any difference what you called it.

I just did a test. I created a file called '5AB917B6-0C0C-40CD-B572-6F90386D203E.cpp' and in the file I placed the following text.

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

class 5AB917B6-0C0C-40CD-B572-6F90386D203E {

Name = "aawUI.pbo";

Version = 1.06;

Description = "This is a User Interface AddOn for use with the AAW Mod.";

};

I then 'rapified' this file with Eliteness and it became '5AB917B6-0C0C-40CD-B572-6F90386D203E.bin'. And, then un-rapified it and the text stayed in perfect condition.

So at the very least it would be great if AddOn makers could put a similar file in the root folder of their AddOn, it would'nt need to be rapified.

The only thing is what exactly to place in the file.

The only 'management' on the part of the AddOn maker would be keeping the guid of the file un-altered for the given pbo and altering the version number within the file.

I just think that having a tool that when you start it up and point it at a folder of AddOns and it can then query AddOn Hosting Databases and show you the AddOns that have updates would be extremely handy.

[Synide 0]

Here's another example... Ok, it's a MOD example and not a standalone AddOn example... but heh...

On a MOD I'm part of, the naming convention I've adopted to date is the following... The MOD prefix is AAW.

aawUI.pbo = The User Interface pbo.

aawWpns.pbo = The Weapons pbo.

aawTracked.pbo = The Tracked Vehicles pbo.

aawAir.pbo = The Air Vehicles pbo.

aawMisc.pbo = Misc. models.

aawCore.pbo = Core functionality for the MOD.

aawBodies.pbo = Character models.

Inside each of these I was planing to place a GUID file.

With the MOD I was planning to release a small .net App that could be run intermitently by the end user.

This tool would first query the above .pbo's and then query our MOD website Database to see if the version the user was running was up to date.

This was the original idea but I thought why not at least broach the subject of a community versioning mechanism to see if it would be of any benefit or not.

[DBO_ 0]

is it not the case that when the tag system is in place and bis sign

etc along with filesize checker all this will be moot ?

I am sorry if i am wrong ,but always had this nagging that Bis said people who use the tag system will get good/preferential support.

Thus leading to a system where , if you joined a server with xxx_addonname and altho it was the same name but not upto date ,then the server would give a message saying "_x uses modified xxx_addonname.pbo" at which point your either kicked or politely asked to update your pbo ?

[Synide 0]

DSSignFile signs the given pbo as being authentic.

If you have an authentic AddOn but it's an older version than the one on the Server the Admin for that server would have to have...

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

verifySignatures=1;

onDifferentData = "kick (_this select 0)";

The biki states that the 'onDifferentData' event should be used sparingly. It makes no reference as to how one would go about using this event sparingly.

I'm rightly or wrongly assuming that this event is a hungry beast, but as I don't have the DSSignFile utility I have no way of testing how hungry it may or may not be.

Indeed you are correct though it could be a mechanism to highlight to a connecting user that a number of AddOns are authentic but out of date. However, it possibly may place a heavier burden on network traffic - who knows?

So, you get kicked from the server. Then you've gotta go through the process of identifying which AddOns you got kick for.

So, you goto your ArmA.rpt file and discover x number of AddOns (made by possibly ny several different AddOn makers) need updating.

So, you goto each of there websites and navigate around looking for their update. And, then the next, and so on and so forth.

Seems a bit time consuming.

Whereas, if I could just start up a tool and point it at the Mod Folder or AddOns folder and it displays x number of AddOns have updates available and where to get them seems to me like a good time saving thing. Maybe?

The only way to achieve this would be to uniquely identify a pbo. Yes, you could just base the uniqueness on the pbo's filename. However, the filename is just a person friendly naming convention that is meant to make it easy to identify what the pbo does and, maybe changed by the AddOn maker.

The '$PBOPREFIX$' inside the AddOn is what points the Engine to the contents of the pbo not the filename.

A much better method to uniquely identify a pbo is either perform a checksum on the file (the likelyhood of 2 pbo's having the same checksum is astronomical - even pbo's that are the same but of different versions will produce a different Checksum) or embed within the pbo an agreed upon file that can uniquely identify it.

The later method in my mind would be preferable because it could also contain relevant information about the pbo's contents and identify it uniquely.

Once you can identify a given pbo uniquely you can check a remote database listing AddOns.

All I'm discussing here is thrashing about possible different ideas so as to make it as easy as possible for end users to manage their pbo's.

[Col. Faulkner 0]

A lot of makers obfuscate their .pbo headers so de-pbo tools

cannot open them. Would this be a problem to your scheme?

[Synide 0]

Mikero's Eliteness allows 'obfuscated' pbo's to be opened.

A tool that reads the contents of an 'obfuscated' pbo to extract the '5AB917B6-0C0C-40CD-B572-6F90386D203E.cpp' file so it can query a remote DB as to whether there are any updates for said pbo is entirely 'doable'.

'Most' addon makers that try to 'obfuscate' their pbo's only munt the header. a few try to be difficult and modify the .paa layouts also but Mikero's tool takes care of that also. (i think).

But for the purposes of pbo versioning one would only be interested in the existence (or not) of a file therein that would look like a 'guid.cpp'.

If for some reason in the future it was impossible to 'look inside' a pbo.

Then what I'm proposing could still be done based on checksuming the pbo and then querying the remote AddOn's database with the 'checksum'.

The benefit of this method is it would mean the AddOn maker would not have to include any information whatsoever with the pbo they would just register that information once with the AddOns Database and when a user uses the 'tool' it retrieves all it needs to know about the pbo.

If the AddOn Maker alters the AddOn or information pertaining to the use of the pbo they would sign into the AddOn's DB or submit a modified AddOn/info to the DB.

A network of AddOns DB could be spread around the community with push/pull subscriptions updating each other with the latest AddOn info.

This would require the AddOns Databases to be available 24/7 which i'd imagine would be desireable for an AddOns DB anyway.

edit:

I only suggested the method of embedding a unique guid.cpp file inside the AddOn because it could also double as a 'pbo details file'.

[UNN 0]

Quote[/b] ]And then you have to redo any missions that where using the Tag_Addon_v1.pbo

In some cases this is desirable, if the addon modifies or removes some functionality used in the mission.

The CfgPatches class name is the name checked by the mission and reported to the user when missing, not the filename. If you made the addons filename and the CfgPatches name, the same. So they both include a version numbers. Then all someone has to do if told that MyAddonV1_2 is missing, is go search on the net for MyAddonV1_2.pbo.

Unless the issue here is keeping all missions updated with the latest versions of an addon, rather than ensuring all players are using the same (mission compatible) version of an addon? If that’s the case you will need to inform the mission maker of what changes you have made so they can decide if they need to re-compile that mission.

From experience the only problems I had when trying to locate an addon used by a specific mission. Was if the mission was an old mission and the older version of the addon was no longer available from the makers site, you could spend ages trying to track it down. Especially if they used a generic name in the cfgPatches section of a config.

All you really need to solve the problem is proper naming conventions for the filename and cfgPatches name, along with someone who is willing to host links or copies of all versions of any addon, and ensure it comes up first in any Internet search engine.

[Synide 0]

Quote[/b] ]All you really need to solve the problem is proper naming conventions for the filename and cfgPatches name.

Maybe, I prefer the stance that AddOn makers should be able to call their AddOns pretty much whatever they want.

I mean the 'only' reason I'm calling 'aawUI.pbo' it's current name is so I and anyone else who looks at the file sort of gets the impression that it might be an AddOn that modifies the ArmA UI.

but really it's irrelevant what the filename is, what's important to the engine is the prefix (or class name) inside the pbo and whats important to the end user is 'what does it do'? and 'is it up to date'?

yes true, a decent readme included with (or inside) the pbo would cover off the 'what does it do'? question.

As far as the 'is it up to date' and 'where can i get the up to date version'? questions.

Based on my experience, most of the time it's not too hard to track down the latest version 'cause I know what I'm doing and where to look.

Alot, of people are not as capable or are just lazy (or are away on deployment or work related stuff) and just want to 'click a button' and have something tell them what is out of whack and where to get it.

Also, so many times I've helped out people try and sort out there mods and or addons with issues such as the right files but incorrect versions, missing files and even corrupt pbo's that it seems to me there has to be a better way.

anyhoo, i've exhausted myself on the subject for the moment - i think!

[UNN 0]

Quote[/b] ]but really it's irrelevant what the filename is, what's important to the engine is the prefix  (or class name) inside the pbo and whats important to the end user is 'what does it do'? and 'is it up to date'?

Well comming from the point of veiw of someone who only plays MP missions, the most important thing for me is. Will I be able to connect to the server and play the mission, if I download half a gig's worth of addons. I just want to play games online, I really don't care if it's the very latest version that comes with tartan seat covers or whatever. As long as I can connect to the server.

Quote[/b] ]Based on my experience, most of the time it's not too hard to track down the latest version 'cause I know what I'm doing and where to look.

Yeah, most of the time it's easy enough. Especially when missions are hosted alongside the required addons. But not everyone has the resources to do that.

Quote[/b] ]Alot, of people are not as capable or are just lazy (or are away on deployment or work related stuff) and just want to 'click a button' and have something tell them what is out of whack and where to get it.

True, there some people who will throw a mission away simple because they have to download an addon. That leads to the numerous "Some many addons and not enough mission" threads we see on the forums. Whether that’s most people or not, I'm not qualified to say. There are already programs out there for OFP that tell you what addons are missing e.t.c Yet the same old discussions like this keep crawling into the forums. So they don't appear to offer the best solution?

The worse case I can think of was a popular and large (700+ mb) Mod used in CTI's. I ended up having three different Mod folders each containing slightly different versions, but even then I still found other servers hosting different versions to the three I had on my HD. Simple because of the missions hosted on the server and the lack of proper version info.

Quote[/b] ]anyhoo, i've exhausted myself on the subject for the moment - i think!

Lol…Oh well, that was a short lived initative. You never know, you might get lucky; perhaps someone else will come along and do all the work for you. If not, it certainly helped kill half an hour and padded the forums out bit

[Synide 0]

Quote[/b] ]Lol…Oh well, that was a short lived initative. You never know, you might get lucky; perhaps someone else will come along and do all the work for you. If not, it certainly helped kill half an hour and padded the forums out bit.

oh, so true... lol.

I created a tool for use by the clan last year that produces a text file of the md5 checksums of 1 or more files within the OFP/ArmA folder and then compares these checksums to a remote version of the same. I made this tool in 2 parts. 1 part was/is run by the Server Admin and it detailed all the pbo's in the MOD and embedded the MOD server password into the text file as well.

The 2nd part of the program the clan members used. When they connected to the passworded server and couldn't get on they'd goto the forums grab the latest checksum file it would show them what was stuffed up on there computer and they went away and just fixed those pbo's highlighted.

Once fixed it presented them with the new Mod server password.

Was great for dealing with clan members where they're OFP/ArmA setup was 'acting' up and it could easily identify what they were missing, what they had too much off, and what was different. So they didn't have to go and download so much to get back in sync with the rest of us.

edit: it would not be hard to modify this tool to either continue using the checksuming method to uniquely identify the pbo or look inside the pbo and grab a guid.cpp file and then have it query an AddOns DB.

So, most of what i've detailed here I already have in place or would not be much work to get it up and running.

If AddOn makers did nothing apart from place a guid as a file inside there pbo's this would at least be a start.

It's quicker for a tool to extract a file from the pbo that looks like a guid.cpp than for the tool to md5 checksum the pbo( in most cases) as some pbo's can get quite large.

Any AddOn hosting service that isn't checksuming the content they are hosting should be doing so imo.

[UNN 0]

Quote[/b] ]If AddOn makers did nothing apart from place a guid as a file inside there pbo's this would at least be a start.

It's quicker for a tool to extract a file from the pbo that looks like a guid.cpp than for the tool to md5 checksum the pbo( in most cases) as some pbo's can get quite large.

I'd be happy to include a guid in the stuff we plan to release. But that won't guarantee everyone (server, clients and addon makers) will use the method. So I would still want to make it as easy as possible for someone to identify which addons were mismatched, and locate the version used in the mission they were trying to connect to. It's safe to say the one thing everyone will use is their eyes. If a mission says you need xxx_yyyyyyV1_2. Then it's easy enough for someone to locate that version using a search engine, if the filenames match the message.

OFPWatch let you select the server you wanted to play on and it downloads the addons used on that server. Regardless of the current version and without requiring any cooperation from the addon maker. But it still wasn't embraced by every server admin or player.

Anything that helps coordinate addons and missions has to be a good thing, but without direct intervention from BIS. There will always be someone who slips through the net.

[Synide 0]

hmmm yes I remember that OFPWatch feature now... there is a ArmA command line parameter that I put on the biki at one stage called '-download' but was never able to get the thing to do anything...

might give it another whirl

[Col. Faulkner 0]

Mikero's Eliteness allows 'obfuscated' pbo's to be opened.

Not in all cases.

[sdoc 0]

if this would become standard, addons by people who 'obfuscate' their pbo's would have a strong evolutionary selection process to let them die out :-)

seriously, trying to make their work unavailable to other modders is something they should be isolated for. their work is also based on the hard work of the ofp modding community.

i strongly support Synides idea. it's one of the suggestions very easy to learn and implement for modders. it has no bad sideffects, and it allows people to create quite a lot of pretty usefull tools.

go for a few more standard tags however, most importantly comments, dependencys, support/update url and a clear version of creators name. could be pretty usefull to display in addon managing tools.

[Synide 0]

@Col. Faulkner... maybe, dunno, there's alot of pbo's out there and i certainly haven't seen every single one of them. However, over the years there hasn't so far been a pbo i haven't been able to look at one way or another.

As soon as someone tries that I get very suspicious. Games are not above being utilized as vehicles for 'virus like' activity.

But, that's really not a big 'show stopper' as I said if it were the case that at some point it was impossible for a community pbo tool to not be able to interrogate the contents of said to extract the guid.cpp file then the 'tool' would simply fall back on checksuming the file as a method for determining it's uniqueness. This checksuming an entire pbo can be a bit more time-consuming depending on the size of the pbo, but just as effective.

[mikero 79]

>>Mikero's Eliteness allows 'obfuscated' pbo's to be opened.

>Not in all cases.

In *all* cases Colonel.

[.kju 3244]

you might wanna check this out:

ArmAWatch

Quote[/b] ]ArmA Watch is an open source project in the footsteps of OFP Watch.

http://armawatch.net/

i hope this one is getting back on track !

[zaphod 0]

agree with Q

[Flat!!! 0]

Why not call it the same name and include a readme?

[.kju 3244]

(possibility of) automation.

less hassle on the user / consumer side.