Addon Serial Database

[bn880 5]

Since we have this Addons At Ease forum here and it doesn't get 100% attention I thought I'd post this here for BIS eyes:

We have a lot of people playing a lot of missions or trying to, where they get missing addon warnings and error messages. This is followed by them not knowing where to get the addon and which version it might be etc.

Auto Addon Server resolves this slightly, honestly not much more than that. And AAE has too many requirements to scoop up the lazy addon makers, such as myself.

Now, here is my idea; if we could have a central addon database, maintained at a single site like BIS or OFPInfo or elsewhere, where each addon, each variation of an addon, and each mod, has a unique numeric serial number or index assigned to it.

The serial number would function like a MAC address does for NIC's, no duplicates allowed ever, everyone must sign up for it at the same place.

Now, in the cfg patches section of a config, people could type in that serial number at the end of the name of the addon, so that people can write this number down, type it in to the OFPEC addon cross reference, find it, click on the link and DL the addon from a server.

This is how I think each row for each addon may look like:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

<index/serial>, <name>, <version>, <maker/team> , <date>, <released>,<relaced by index>,<DL locations>

unique integer, string, string, string, date, boolean, unique integer, string(s)

0000001, "Bn Tracers", "1.24", "bn880", 08/09/2005, true, 0000000, "http:\\blah blah"

^ this is all a rough draft and idea. Now, if each addon and mod has a seperate and unique serial, we can fairly easily decentralize the download locations to dozens or hundreds of servers. SQL queries can easily be run on the entries to search for addons.

I do not have any of the web server for this written, nor do I plan to work on this personally. However I think it is a valuable thing to consider, to have ONE central place to find every addon ALL the time.

The addon makers could apply for their addon serial number before they publish the addon, so that they have time to insert the serial in the cfgPatches section. Each entry in the DB would need to be administered by someone, as with most other submissions. An admin could take in a request for a serial, create the entry, and send back to the addon maker the number.

This can extend to Armed Assault, and probably as far as Game2... :-* Thoughts? Actuallty remember, when ArmA comes up a lot of addons will be repacked for the new updated engine, a great time to serialize them.

The main thing I am concerned with is that this be kept without any requirements for registration besides: author or valid team member submits their addon/mod, it is not an exact duplicate

This is not about requiring things of addon makers, for approval, this is to give them a tool to identify their addon globally, to make it easier for lazy users such as ourselves.

[bn880 5]

A previous discussion about this that may be relevant:

An automated Registration Site

Addon maker registering would be required to fill out a simple form

Registration Form

<span style='color:Red'>NAME:</span>

<span style='color:Blue'>VERSION:</span>

<span style='color:Red'>Previous versions avail:</span> (YES/NO)

<span style='color:Blue'>MAKERS NAME:</span>

<span style='color:Red'>EMAIL:</span> (With option to hide it)

<span style='color:Blue'>TYPE:</span> Pull down list eg (MOD / Air / Armour / Weapon / Units / Combination / Misc / Island / Objects)

<span style='color:Red'>DESCRIPTION:</span> simple text entry field

<span style='color:Blue'>LINK:</span> (Verified as a working link during registration Process)

<span style='color:Red'>FILESIZE:</span>

<span style='color:Blue'>ADDITIONAL ADDONS REQ:</span>

......................................... Serial No

......................................... Name

......................................... Version

_

something simple but time consuming enough to deter timewasters and spammers

Some form of anti spam protection, only 1 entry per IP per XXXX amount of time

Emailed Registration Number and requiring validation something on those lines

As an OFP user searching for the addon, they would log onto the site and be able to search either by

Serial Number

OFPEC tag, (which ideally would be in the addon name anyway)

Name of addon

Type

this will then pull up a simple window, with the content filled out in the registration form and the link highlighted to them

An automated link checking system, would monitor the condition of the link and inform an admin, if it went down

(A highlighted message box in the search process would inform the user that the link was down)

But as BN stated, only thing an addon maker has to do to the addon is add a serial number to the Required addons field nothing more than that, or it just simply starts to create unescessary complications

and as we all know, the more complicated it is, the more likely it is to fail

<span style='color:Red'>I think its a great idea,</span> and if those technically able to do such a thing, get it together, this could be running before Arma is released, which will be great for the community

a good idea, but too late i'm afraid...

how would you handle already released addons which most missions rely on? would you want the addonmakers to re-release their addons with such a serial, or would you wanna assign the serials to addons ex post?

anyway, this would be a hell lot of work given the sheer number of addons. and what about the addonmakers that have disappeared - or are out of reach for the english-speaking community?

if realized right from the start then it's a good idea for ArmA i think; in close cooperation with BIS preferably.

Its never too late

From the outset of the OFP Release, of course it would have been better, however thats hindsight.

A good addon making tutorial would have also been welcomed, but that didnt happen either

I am sure that when the OFPEC tag system entered its initial discussion phase, the same was said

just because previously made addons may not be upgraded by their makers to utilise any such system, does not detract from the fact that overall it would be a worthwhile system

And better a system like this too late, than never at all

The OFP community still flourishes, even in OFP1's twighlight days, and it will continue to grow, so please dont think about the past, think about the next 5 years and the hundreds more addons that will be created that could utilise such a system

<span style='color:Red'>MANPOWER REQUIREMENTS</span>

For the web based scripting system, i have no idea, but from a registration point of view, nothing more than a minute or so, and to add the serial to the addon config, even less

Once the system was incorporated, it would i imagine be fairly easy tro manage, as most of it will be automated

<span style='color:Blue'>ARMA RELEASE</span>

I would also expect, that when Arma is released, that a lot of existing addons may be upgraded to utilise any new entries in the comref or new engine attributes

so preparing such a system in advance would be a very sensible approach

[Jinef 2]

Bravo! Encore Encore!

Darling you are so talented.

I want your baby!

[bn880 5]

Damned weirdos!

Nothing to add to the topic?!

[Spinor 0]

An absolute dream (for AA etc.) would be if this unique addon identifier is included in the mission file, and missing addons can be automatically downloaded from the BIS server.

[bn880 5]

Well yes, the plan is for the identifier to exist in the mission file, as it will be appended to the current cfgPatches of the addon, such as:

addOnsAuto[]= {

"bn_tracers_<serial>",

"finmod_<serial>",

"coc_ce_<serial>"}

(as an example)

But for this to be automatically linked inside AA to a download location would be amazing indeed. I figure only the lookup database has to be centralized and administered at BIS, the actual download space can be decentralized and offloaded.

[blackdog~ 0]

If someone else does all the work I'll host it and gladly take all credit for the idea.

[tug2000 0]

Funny you should say about this. I've previously posted about a similar idea. And have started the programming work in C#.

But mine does not use anything in the config file for the reference instead using a MD5 hash of the addon.

Also, this means existing addons can be ported into the system.

The GUI would have the ability to auto-download updates, could detect missing addons and download.

Planning on making this modular to allow support lots of different games.

Any thoughts?

[bn880 5]

Well I don't know exactly about your idea as it's really a similar way to solve something different?

What I am aiming for here is for the user to know 100% which addon he/she needs to get when they get a missing addon message (currently it's not that easy). This by the unique serial number in cfgPatches, and it requires no additional software to be run. With an MDS, I am unsure as to how the user will know what they require. However, I am not saying your project is a bad idea nor am I trying to deterr you from working on it.

I would be happy if BIS could pick up on the idea of 100% addon identity match via mission.sqm, and a download/info database which is linked to that via AA GUI.

[crashdome 3]

Sounds like a super idea. Glad people are thinking of the add-on issue.

Just want to get your gears grinding:

1) What if an author (or someone) makes a change to an add-on?

If it is enough to break a mission how will you know? In the case of MD5, what if the change is a simply a small bug fix like text change or something that doesn't affect anything? won't the hash change?

Sounds like we would need a team in place to control either when a serial number should change or if it's truly a new version of an add-on.

In that event, who and what if it gets difficult?

2) What if a mission uses only a single unit or weapon? Is it still required to download the whole add-on pack?

IMO, the whole concept of add-ons should be re-thought. Why must we pack everything together as we have? We should be further breaking down add-ons and making them multiple files (one for sound, one for each weapon, etc..) and simply have a way to organize them better.

If that were the case, could we not pack most of this into the mission directly??? Most people will download 200mb of add-ons for the hell of it. Having the ability to (as an option) compile the add-ons directly into the mission (or as I would like - mission packs part-by-part or unit-by-unit would increase mission size (e.g. 5-10mb) but for most people this is chump-change compared to the amount of add-ons downloaded w/o missions. Sure you may double up some add-ons but I have so much space from regular un-used add-ons that if I just downloaded the missions I liked, I would save most of my HD actually. It also solves when we get the JIP feature. I Imagine trying to JIP w/o the proper add-ons is going to be the next big disappointment for most. Compiled into the missions solves that (Just an idea)

[bn880 5]

Good to see people being warm to the idea.

1) My view on this would be, every time you make a change you get a new serial, this is kind of a hard fact of the solution. This IMO is the only 100% reliable and fool proof way of ensuring people have the right stuff, with the minimum of confusion. It will be up to the author to decide whether or not the addon is backwards compatible, and to flag the old serial as depricated. The old entry in the central DB will have a pointer to the new serial, and it will be a linked list in that way. The thing to keep in mind is that the process will be painless and a serial is just a number, we would probably have an enormous capacity in terms of serials, like several billion numbers in 10 base.

2) Certainly I see your point, however as it looks right now, it will be up to the addon makers to decide if they wish to break up their addons into several standalone PBO's and to request serials for each. That's the problem of allowing a breakdown of an addon/mod, you often need to grow each individual piece, and need a lot more testing to ensure each eas truly capable of running without the other. Having said that, it is theoretically in BIS's capability to write an addon/mod interpreter which automatically has a capability of breaking down an addon into standalone pieces. (scripts being the only immensly unpredictable factor, especially if an addon maker does not consider that the pathscould change) Theoretically. I am certain BIS doesn't have the time to do this, but I hope they implement a centralized serialization and distribution/acquisition of addons!

[Spinor 0]

Just to clear this up: A MD5 hash is a number constructed from the content of a file uniquely identifying it, right?

One advantage of using MD5 would be that the system can be less centralized. Using serial numbers, the system work only if there is really just one server that has the authority to give out the number. In addition, with a MD5, there would be no need for including the identifier within a file itself.

Quote[/b] ]1) What if an author (or someone) makes a change to an add-on?

If it is enough to break a mission how will you know? In the case of MD5, what if the change is a simply a small bug fix like text change or something that doesn't affect anything? won't the hash change?

Sounds like we would need a team in place to control either when a serial number should change or if it's truly a new version of an add-on.

I think this is better decided by the addon author itself: When uploading a new version of an addon that is backward-compatible, the new hash is treated as an alias to existing ones (with the additional feature that updates can be automatically identified and downloaded).

If the new addon is not backward-compatible or differs much in functionality, it should get a completely new entry.

Quote[/b] ]2) What if a mission uses only a single unit or weapon? Is it still required to download the whole add-on pack?

The system would indeed work best if each separate addon has itw own file. There would be no need to organize addons in packs if everybody would adopt the system. I think Addon Packs can still be managed, by using aliases as above, i.e. a single unit/weapon can be uploaded separately or within a pack. The hash of the pack is an alias for the hash of the separate addon (but not vice versa, if I do not get confused here).

One way to realize the system on the user-side is to combine the addon-manager with an OFP (or AA) launch utility. The addon-manager can test the dependencies of all missions and addons prior to starting OFP (mod folders can also be taken into account properly).

For every addon and mission (missions can also be assigned hashes):

1) Calculate hash and send to database server(s)

2) If a new version (hash alias) is available, inform user

3) Retrieve dependency hashes from server or from a local dependency file

[Missing dependencies]=[All dependencies found]-[Local addon hashes]

For every missing dependency:

1) Retrieve download location(s) from database server and download addon from there

2) Optional: Automatically unpack addon to appropriate folder if necessary

This procedure will probably take some time for dozens of addons and mission, so result of a previous scan should be cached.

This system should even work quite well decentralized. There could be several database servers from which to retrieve the data.

[bn880 5]

I wanted to quote Spinor here on the same issue, as it is very clearly showing the need to separate two aspects of addon hosting in the community:

bn´s suggestion is only concerned with retrieving addons and other resources on which a mission depends on as easily and reliably as possible. Tagging and reviewing are both concerned with the quality of a mission, addon, etc. and are a different matter and should be separated. This is not only about mission makers being too lazy to include links, its also about links in old missions becoming obsolete, etc.

The optimum would be if BIS would support such a unique addon serial number in AA and subsequent games, i.e.

- The serial number is contained within the addon/mission/etc. file along with serial numbers of all dependencies.

- When starting the mission, AA tries to automatically download the missing dependencies.

The next best thing is separate application that does the downloading as suggested by tug2000

Whatever solution, I would find a central database if not file server (or server mirrors) to be desirable to ensure retrival of a resource. Island solutions via community sites will always result in holes where a given addon can be found at site A but not site B.

As an analogy, this is how its done in the scientific community (physics):

1. Publications can be uploaded at a central server (actually multiple servers, but all are mirrored, e.g. www.arxiv.org), and you get a unique ID for it. This number has become the standard way for references and citations.

There is no quality control for uploading at this server, everybody can upload, only formal/technical editing such as checking file consistency, and removing fake stuff.

2. For quality assurance, you can send your publication to one of a series of journal, where your work is reviewed by one or more referees. If found suitable, your work is published in that journal. This is where all the funny stuff happens, e.g. one journal is more 'elitist' than the other and harder to get your publication through...if you fail, you can send it to a less prestigious journal, etc.

Reviewer can also demand changes before the journal publishes the work.

Applying this idea to the AA+ community (for OFP, I think all this discussion is too late):

1. The big community sites agree to host mirrors of a central mission/addon database where anybody can easily upload resources.

2. Addons/Missions can be sent to community sites such as OFPEC for reviewing. Each community site can apply its own rules/requirements such as on variable tagging or quality. Make your rules strict, you will have few but high quality addons and missions. If a mission/addon does not reach the required quality level, reject it. I would even be inclined to suggest to get rid of the rating system: if you have enough of such 'reviewing journals' the quality will automatically adjust, e.g. the best mission makers publish at the best community sites and vice versa. In the end this might even reduce the reviewing work.

Just a thought, nothing more...

[tug2000 0]

Thanks Spinor. Took the words right out of my mouth

I agree it would be ideal to have the download support built into AA but that may or may not be practicable.

The reviewing etc. was not something i was initially planning on building in. So purely registration, version and depandancy info of each the addons.

Also, I'm planning to have plugin support for multiple games.

For the geeks : In addition, the system will have Web Service interfaces so if BI saw fit they could build support directly into AA.

@BN880 & Spinor if you guys fancy helping out with some advisory discussion etc outside of the forums i would appreciate it. Let me know on PM Also anyone else who has some related ideas drop them here or PM me.

[vektorboson 8]

I have made some tools a while ago, I never announced here, which were recently improved with very simple version control.

I use a special tool I called "Recon", that does the following tasks: Checks the config, binarizes the addon, binarizes the config, PBOs that thing up and copies it to my special addons-directory.

In that whole process it creates a file which is named "build.num", it contains the build number (how many times recon was called successfully on this addon) and the according date.

You may take a look at Recon (and some config-tools) here:

CfgCheck.zip 560 KB

I think that would be the quite correct way: Include the version information into the PBO (I think BIS does something like that with their vssver.scc). You could also include the dependencies into PBO; best thing would be, that a tool like Recon created the addon dependencies automatically.

[Sui 0]

I'll re-post here what I posted in the OFPEC thread...

Quote[/b] ]

I think Spinors view on a centralized solution is correct... I reckon to cover the whole community we'd need to get BIS involved.

Ideally, by submitting an addon to one of the top sites, it would automatically get registered in the 'master' database.

I'd definitely be keen to work together with ofp.info and the community at large to come up with a viable system to get something like that off the ground. We're also planning for changes and improvements to OFPEC. Maybe now would be a good time to discuss how we can work together and come up with a more united system of how addons are handled around the major sites?

So ideally, some sort of co-operative would exist between the major sites, whereby registering an addon with any of them automatically enters it into the master (BIS?) database.

Whether this is done by serial number, OFPEC tag or md5's is something that would need to be worked out, but I think the principle of a centralised database maintained by a community authority (ie BIS *hint hint*) is what's needed for the whole community to start using the system.

[crashdome 3]

Quote[/b] ] When uploading a new version of an addon that is backward-compatible, the new hash is treated as an alias to existing ones (with the additional feature that updates can be automatically identified and downloaded).

If the new addon is not backward-compatible or differs much in functionality, it should get a completely new entry.

I like that.

I also like vektorboson's idea.

What about playing an old mission that needs an old version of an add-on? Does it download and over-write the newer one automatically?

tug: Will there be source available?

[tug2000 0]

I'm devising a few plans to enable older addons to be avilable for old mission etc. Use of mod folders and integrating a launcher etc.

Client wise it will have at least an open API. Server wise i don't feel it's really any need to make it availble.

[bn880 5]

Well after long discussions, it also appears to me that the MD5 hash will be the way to go with this.

Main point being, serialized tags: backwards compatibility will suffer. OFP has some weird behaviour with making backward compatible tags so that missions do not report missing XYZ addon V1.0 or 1.2 etc.

A global solution can be made quite well, if we take your MD5 work tug2000 and coulpe it with Mikero's AddonScanner, plus some centralized DB for MD5 hashes and dependencies.

Mikero's addon scanner would need to be modified to work like AAS, open all PBO's in OFP subfolders, detect all the necessary tags from their configs to match tags to PBo's. Once a single PBO is matched to a tag from mission.sqm, it's MD5 has can be obtained, and all dependencies can be found at the hash DB. Additionally we would store the PBO data once it is scanned. And only re-scan new or modified PBO files, so that we do not waste people's time.

We are 40% done a unified solution this way anyway. All it takes is for people to work together to a common goal.

Also, the best way to make sure the mission does have all the PBO's it needs is to store an additional MD5 list file in the mission folder/PBO. This file would be managed by the external exe launcher.

[M9ACE 0]

I rarely visit this forum area, but I happened to check out what was going in here to kill some time.  After doing a little bit of reading, I believe I understand what you are trying to accomplish bn880.  I would like to compliment you on your idea and encourage you to continue with what you are trying to do.

[.kju 3245]

is this still being worked on right now with ofpec down ?  

[tug2000 0]

Yup, I'm working on the system architecture.

Note: i'm not affiliated with OFPEC or CoC. But have been talking to BN880 and Spinor for ideas and support.

Hopefully by end of this month I should have a site up so we can start importing data.

The desktop application / test will take longer.

[Sui 0]

We're keen to support this in any way we can, though at the moment we're pretty busy rebuilding.

We'd like to incorporate it into our site, not necessarily hosting it ourselves (though we probably could if no-one else is willing or able), but I'd definitely be willing to ensure we 'comply' to any community standard we can come up with

This would be great if we could get a system in place before ArmA comes out... it may already be too late for OFP addons, but something along the lines of this should be compulsory for future ones.

[tug2000 0]

Great to hear.

As for hosting I'll probably end up hosting the master database and having a nice integrated mirroring system for site owners to provide most of the downloads.

This is something that will have to be worked out with mirror sites once I have a preview up.

[ag_smith 0]

I'm not really sure if enforcing usage of automaticaly generated MD5 hash is such a good idea. I very often modify addons for my personal use (tweak, fix 'n' stuff) thus MD5 will change as well. Implementing MD5 checking on mission start would make me unable to do so. There should be at least an option to turn that feature off.

Just my 2 cents. I hope I understand it right what you're trying to do.