RPT file checker idea.

[Yoma 0]

I just had the gazillionth crash, most likely due to some bad/conflicting mod on a multiplayer server.

Up came a small idea.

What if:

I created some kind of wrappertool around Kegetys CPBO.exe and then made some kind of database with:

-a table with Modfilename + MD5sum

-a detail table with all filenames that are in this addon

-a table storing all configfiles in the addon in searchable text

-maybe something else, but i honestly don't know what (i've never created an addon)

So you could browse to a directory and have the tool create your database with "inside" addon information.

Next to that some kind of arma logfileparser that looks for strings and points you to the mods causing errors in arma.rpt

I've never run a dedicated server but maybe it's logfiles would also have interesting stuff to look at/compare.

Maybe it could even store these errors to provide feedback to the makers of the addon.

Would this be

-legal (i don't wanna get into trouble)

-worth the efford?

What further tips could addonmakers/serveradmins give me to create some kind of Arma Addon Debug Tool

It might be something completely useless, but it might also be an invaluable tool to set standards on addon quality.

If i ever get started on making something like this i would really need help from a top notch addonmaker. Someone who knows every kind of file a mod can carry and what information is stored in it.

[Synide 0]

An interesting idea... Sounds similar to something I created late 2006 early 2007 called 'cow' & 'sheep'...

A Server Admin used one part of the program called 'Cow' which he could produce a text file of various subfolders of an OFP or ArmA root folder containing md5 checksums of the contents of those folders.

He'd pass out the text file only...

On the user end they ran 'Sheep' intermitently and pointed it at the md5 text file provided by a Server admin and it would show the User whether their versions of the files specified were different to the server versions or whether they had more than is necessary or whether they had missing ones...

It was written in .net which I realize you prefer c++ (i think, don't you? ) if you'd like the source to muck about with drop me a line...

Some screenies...

Cow...

Cow Output...

and Sheep...

Cheers, Sy.

[.kju 3245]

Maybe it's me. However I find it hard to understand what you

really want to do / achieve. Can you please outline your goal(s)

more clearly please. Thanks!

[Yoma 0]

Maybe it's me. However I find it hard to understand what you

really want to do / achieve. Can you please outline your goal(s)

more clearly please. Thanks!

For example:

You have several modfolders loaded, you start a mission and get a message missing blabla.paa or cannot load blabla.paa

I want to provide a means to quickly find what addon is causing this.

Or you get all kinds of errormessages in arma.rpt (i guess errormessages there are not a good sign) If you could scan this file and compare it to a moddatabase you could better find stuff that inheritely causes errormessages. My guess would be: if it causes errormessages, it might cause crashes.

I may be way off here. I don't know.

[Yoma 0]

An interesting idea... Sounds similar to something I created late 2006 early 2007 called 'cow' & 'sheep'...

A Server Admin used one part of the program called 'Cow' which he could produce a text file of various subfolders of an OFP or ArmA root folder containing md5 checksums of the contents of those folders.

He'd pass out the text file only...

On the user end they ran 'Sheep' intermitently and pointed it at the md5 text file provided by a Server admin and it would show the User whether their versions of the files specified were different to the server versions or whether they had more than is necessary or whether they had missing ones...

It was written in .net which I realize you prefer c++ (i think, don't you? ) if you'd like the source to muck about with drop me a line...

Some screenies...

Cow...

Cow Output...

and Sheep...

Cheers, Sy.

My addonsync solves most of the stuff that you mention.

And it has download and launch functionallity.

And it has modsequence functionallity, which can also be controlled by serveradmin

[Synide 0]

My addonsync solves most of the stuff that you mention.

And it has download and launch functionallity.

And it has modsequence functionallity, which can also be controlled by serveradmin

Yes, I am aware of what your sync program does... it would have been difficult to miss your OFPWatch ArmA replacement...

So...

-a table with Modfilename + MD5sum

- I've personally always thought that it would have been a good community initiative to have a DB of pbo's and their md5's.

-a detail table with all filenames that are in this addon

- Can't see the benefit in this... maybe you could give an example...

-a table storing all configfiles in the addon in searchable text

- Keeping a running DB of text versions of a given pbo's config's might be beneficial, dunno...

-maybe something else, but i honestly don't know what

- I guess one of the holly grails would be something that could tell you which pbo's conflict with each other and specifically what portions... A gargantuan job though...

[Yoma 0]

I'm not even talking about conflicting mods.

When i open my arma.rpt after a night of playing on modservers its allways has plenty of errors. I don't know alot about the internal techics of addons, but i'm pretty sure that if one keeps outputting errormessages, this cannot be a good thing.

A public database is even another step that i didn't even mention, allthough it might be an option.

But just a local database having all mods you use and a crossverification Arma.rpt errors vs mods might lead to some interesting results.

Stuff like hey maybe i'm NOT gonna use this addon because Arma keeps reporting errors on it.

An example where filenames could come in handy:

I connect to my favorite server

I get the message

Warning Message: Cannot load texture data\hang_bordel.pac

How are you going to fix this?

Maybe a missing/corrupt texture could lead to the client game crashing?

If you could generate a database of all mods loaded you could at least find out WHAT mod causes this without a lot of manual work.

Again, i'm completely freewheeling here.

It would be nice if a dedicated serveradmin could send me a logfile so i can have a look at it. Maybe that too would be something nice to parse/crosscheck.

[sickboy 13]

I personally don't see the use of this.

If there's a missing .paa in an addon, then the addon creator can simply test himself and debug the arma.rpt and supply a fix.

If the tester wants to be helpful, all he has to do is remember what the .paa is called (write it down? lol), open arma.rpt file, use the text editor search option, search for the xxxx.paa, and copy the matching line to the developer. The developer should know what to do

Other than that, by using ModFolders, I believe one can pretty quickly figure out which addons are causing his problem. Also using windows search in the addons folders and search for "data\hang_bordel.pac" wouldn't take more than a few minutes aswell, even on like 10GB of addons :-D

Quote[/b] ]Warning Message: Cannot load texture data\hang_bordel.pac

How are you going to fix this?

Maybe a missing/corrupt texture could lead to the client game crashing?

If you could generate a database of all mods loaded you could at least find out WHAT mod causes this without a lot of manual work.

The problem is that textures are defined inside models (.p3d), either binarized or unbinarized, maybe you can read these also into the database, or not?. If not, then when a model from XYZ_Vehicle.pbo uses a texture: data\hang_bordel.pac, then there's no way you will find in your database, which addon/model uses this texture. (However, you can know where the texture is located)

The problem is basicly that a model doesnt have to take textures from the same pbo, it can take textures from anywhere, outside pbo's, or inside other pbo's.

There's also a problem with different versions of addons, how will you distinguish? If you use md5 hash or similiar, how will this work with modified versions?

What IMHO might be useful is an arma.rpt log parser utility:

[*] Ability to search for a string, the utility would then display all matching lines. (maybe an array of strings would be nice)

[*] Option to enable/disable 'copies display'. If copies display is disabled, all double/multiple copies of error messages are not displayed. So only 1 occurence of every error would be visible

[Yoma 0]

Ah ok as mods can share textures, this completely changes the picture.

Wow your windows search looks inside pbo's? Mine doesn't, unless i go through the effort of manually dpbo'ing 20GB of mods in different folders... ;-)

Maybe vista search DOES look inside them?

Different versions would not be any problem (i could take an md5 as and link to other tables using a primary key) this way you would only see the files/stuff that's linked to that md5.

Modified versions will simply have a different md5.

So in short it could then come up with a result md5 / exact location of the file.

Anyway the fact that mods can cross use data does away with most of the idea, unless i could indeed unbinarize the p3d files which is probably illegal anyway.

Damn.

The logparser idea i can do, i guess.

Sickboy could you send me a big multiplay logfile?

(Don't run a server myself, but i just want to look at what is displayed in there, if i make an arma.rpt parser, i might as well make a multiplay log parser)

[sickboy 13]

Here you go:

http://pb.6thsense.eu/pb/56

The rpt isnt so big cause since it was cleaned, not much has been going on

Basicly, the rpt of a server is just the same as a client, except for the "Server: ..............................." messages

[Yoma 0]

I made a smallish logparser, it's far from perfect, but might come in handy.

Download it at http://www.yomatools.be/ArmaLogParser/ArmaLogParser.rar

[.kju 3245]

link doesnt work here

[Yoma 0]

Seems my host is upgrading/f*cking up.

Temporary mirror here

yomatools.be is back online.

[.kju 3245]

Checked it. Thanks for the effort Yoma!

However its not useful for me at least.

EditPadPro with regular expressions is far more useful for me.

One problem is that error messages can be multi line.

The core prob is that most of the time its hard to track down

the source as the text of the error messages is only limited in

usefulness. So BI had to improve them instead.

EPP supports LC, sorting, RE, search and such. Much more

flexible as you can edit it all.

To locate the source of a problem its always best to remove

all PBOs and put them in one after another.

Thats why I asked. I can hardly think of any additions to make

ArmA.rpt output more useful. Only BI can make that.

Anyway again thanks for your effort!

[Yoma 0]

Still i'm toying around with something.

I'm just doing some tests on how far i can take this.

What i've added (not published yet) is a wrapper around kegetys cpbo.

You can now unpack all pbo's in a subfolder in one go. (cpbo does however not like some addons (TrueView etc won't extract here)).

My though is that somehow having a searchable database of addons/files they contain might come in handy at one time or another. If only for solving stuff like: An admin puts a map on his server,  but doesn't play arma a lot. Some scripts will put out errormessages, logged in arma.rpt. Perhaps if one had a database one could compile from all your addons/missions/etc, it would be easier for a server admin to find bugs in maps he didn't write.

I know this will most likely all lead to nothing, but there's not hurting in trying ;-)

I also noticed that the p3d files are binary but do contain readable text (the .paa files i can read at least), maybe that's not the case with all addons, but it may lead to detecting some .paa path problemns.

Anyway: expect nothing, but i'll keep you posted.

To me the tool was allready usefull, now i can at least quickly judge some of the quality of some addons.

Warning: sd_btr80\sd_btr80.p3d:0.5 Error while trying to generate ST for points: 325, 319, 327

(a couple of hundred times, this cannot be good?)

Or does this say nothing about the quality, but is it just some kind of state where the game-engine lost it's wheels. I'd seriously doubt that if the errors keep coming back.

Also using the new "extract entire armafolder"-stuff i came across some potentially bad addons (cpbo gives crc errors), i don't know if it's cpbo or the addon that's to blame, but it's worth an investigation. I can imagine the engine crashing on a corrupt pbo.

Quote[/b] ]To locate the source of a problem its always best to remove

all PBOs and put them in one after another.  

Somehow there must be a better way to at least solve/detect *some* problems.

I think we should seriously consider RATING addons on quality, and not just niceness in game. I mean there is this hind addon that causes massive errors in arma.rpt all the time.

Sure i want a hind in arma, but i don't want it to crash/slow down the friggin game.

Maybe BIS should consider some kind of detailed logging mode, just to sort out problems.

I guess i'll keep toying a bit.

[.kju 3245]

Warnings is "not a problem". Errors are. At least they are

different in severity.

So the ST warning are not an issue. The addon should be binarized though to hide the warning.

ST warning will always appear.

ArmA.rpt flooding can be an issue in MP, yet it has to be a

message that is debugged into the arma.rpt several times a

second again and again. This happens VERY rarely.

CPBO cannot unpack PBOs in OFP PBO format I think.

Most likely these addons have been packed with OFP tools.

The core problem I see is that even if you detect the error in

a mission or addon, there is a low chance to get it fixed. Most

people either delete the mission or try to fix/workaround it by

themselves.

P3Ds (model containers) contain the path to textures yes.

PAA/PAC are textures. They only contain color information.

RATING is hard. Takes a lot of time for in depth. Also in effect

it would be better to have the addon/mission improved than

only rated.

I can see what you are after. Yet I am unsure if there are good

ways to combat the problem.

[Yoma 0]

Just out of curiosity: if logging brings next to no decent debugging info, can it be switched off alltogether clientside/serverside? Might just bring that extra minimini bit of performance.

One thing i truely hope for in Arma2 is an easy way for serveradmins to create a blacklist of addons. This way admins can at least ban addons in an easy fashion.

What ofp tool was probably used to pack those mods? (if i remember correct you had something to do with Truemod?)

[Synide 0]

The pbo's may have been packed with an older OFP style pbo tool. Or, the addon maker in question possibly could have created the addon with the ArmA BinPBO/Filebank tools but then afterwards they sometimes 'play' with the header in the pbo to make it more difficult for oppotunistic extraction of there work.

Mikero's pbo.dll will open all versions of pbo's, OFP Elite and ArmA... ask him if you can link his dll to your little program - he might, ya never know....

I suggested a while ago that all addon makers should package a small text file with their pbo's which contained additional meta information so that there was no need to physically open the pbo to have a complete listing of it's dependancy's etc. I also, suggested that this text file contain a guid and a md5 of the pbo. It met with little support at the time... I think Q might have seen the value in it...

Anyway, having an additional metafile with community agreed upon infomation inside would have lead to all sorts of wonderful management senarios for pbo's for both SA's and clients alike...

You can use Mike's latest Eliteness to unpack them... unless they creator has played with the header... in which case some very simple Hex Editing of the pbo header will put it back into a useable state...

[Yoma 0]

I think the metafile would never work, as addonmaker would have to MAKE that.

I'll look into using something else but cpbo to extract the pbo's. Maybe i'll take a stab at an auto database creator too.

Not quite sure on which database to use yet either.

It would need to be something that's easily deployable and takes little to no resources. I'm currently thinking along the lines of SQL lite.

If people fiddle with headers to make files unextractable, then i'm not going to take the efford to try unfiddle them. By the way out of 1500 addons on my HD CPBO only had trouble extracting about 10 i think.

[Synide 0]

I think the metafile would never work, as addonmaker would have to MAKE that.

Creating a tool to automate the production of the majority of the info in a metafile is not difficult... And, personally I think it would work very well indeed... however, I will admit that I'm in the minority with this view or the majority of the ArmA community are disinterested... either way I moved onto other areas... I'm a strong believer in meta-information, for too many decades the world's had god-zillions of gigabytes of under utilized data sitting about... anyhoo...

I'd thought I'd mention the idea here as you seem to be keen on providing the community at large with pbo/addon related management tools and may of been of interest to you, obviously not... nvm.

With respect to data storage in your idea... I haven't much looked at stand-alone DB softwares for a number of years. Maybe, BDE, Borland Database Engine might be a go?

There are quite a few free C++ libraries around with substantial DB functionality that can operate on your own flat binary datafiles too... s'another option...

[Yoma 0]

I'm currently toying around with SQLlite.

It seems to work pretty well.

I don't have the means to provide a true online database where users can store their generated data realtime.

What i plan however is to have some kind of function so that people can do stuff like take someone elses generated db and have it imported in theirs.

It's not that i'm agains metadata at all. The main problem is that it would mean some kind of support by all addonmakers, which i'm pretty sure will never happen.

The next best thing in line is a tool that can "generate" the data for you and an easy means to share this info with others.

That's just my 2c.

I intend to provide some kind of service where people (mostly server admins) can post their db's to me and i can add them to my repositry. I could then easily provide something that "get's the latest addon repositry" online. Maybe we could get to a point where people can get pointed to missing addons via this system. (there's no harm in dreaming now is there)

At the moment i have part of this working, in a proof of concept style thingy, but i don't see any main problems with implementing the stuff i mentioned.

It could even be integrated with my addonsync tool at some point. Imagine this: what if we had several addon repositry servers (with stuff like directoryname=MD5) and gameservers simply have an xml with all addons listed in them. Then my tool could get them from the repositry instead of getting them from a server setup by the game admin. I'm affraid we won't ever get to that point, it would take loads of webspace and loads of bandwidth to host a repositry server.

More to come, soon.

[Synide 0]

I'm currently toying around with SQLlite.

It seems to work pretty well.

I don't have the means to provide a true online database where users can store their generated data realtime.

What i plan however is to have some kind of function so that people can do stuff like take someone elses generated db and have it imported in theirs.

It's not that i'm agains metadata at all. The main problem is that it would mean some kind of support by all addonmakers, which i'm pretty sure will never happen.

The next best thing in line is a tool that can "generate" the data for you and an easy means to share this info with others.

That's just my 2c.

I intend to provide some kind of service where people (mostly server admins) can post their db's to me and i can add them to my repositry. I could then easily provide something that "get's the latest addon repositry" online. Maybe we could get to a point where people can get pointed to missing addons via this system. (there's no harm in dreaming now is there)

At the moment i have part of this working, in a proof of concept style thingy, but i don't see any main problems with implementing the stuff i mentioned.

It could even be integrated with my addonsync tool at some point. Imagine this: what if we had several addon repositry servers (with stuff like directoryname=MD5) and gameservers simply have an xml with all addons listed in them. Then my tool could get them from the repositry instead of getting them from a server setup by the game admin. I'm affraid we won't ever get to that point, it would take loads of webspace and loads of bandwidth to host a repositry server.

More to come, soon.

I had similar thoughts to all this several months back and spent a week discussing it publicly with the community... well mostly I blathered on about shit and people made suggestions why they didn't think it'd work...

Yes, a centralized informational DB about addons would be nice wouldn't it... nvm.

You could get whizzy and throw some torrenting of standalone DB's into the mix...

When I refer to metadata I basically mean largely automatically generated stuff based on whats in a pbo...

I've thought of many similarly related things regarding addon managment....

Out of all of it I do know one thing for sure... The management and lifecycle of pbo's for both server admins, client installations, addon sites etc. has to evolve... the management of everything to do with addons has stayed pretty static for several years... for optimal benefit to all there has to be a planned evolutionary path to addon management. And, I believe metadata is one of the cheap wins that can relatively easily be implemented right throughout the addon community.

I'm sorry for blathering  on in your thread about a .rpt tool... I'll let the topic go now... good luck with your endeavors...

Cheers, Sy.

[Yoma 0]

No problem, the thread has taken this direction allready from the start.

I need someone that's real good with addons to give me more info on

-All types of files an addon can contain is there some kind of comprehensive list somewhere?

-The datatype in which these files can exist (binarised, text only, "media file")

I can obtain this list by trial and error, but maybe someone hase a complete list lying around. If my work doesn't intervene (we have a holiday tomorrow but i'm on call), i'll work on it a lot tommorow.

So far i think the idea will spawn some usefull stuff.

[Yoma 0]

The thingy can now:

- look for all pbo files on a disk

- extract them all (with a few exceptions)

- inserts all pbofiles (md5, path on disk, pboname) into a table

- inserts all extracted filenames (path in addon, filename, key to pbo file table)

- the database size with this info in it is about 8MB

 , extracted size is 18GB

It has a few downsides:

-takes a lot of time / diskspace (about 1 hour! to generate database for about 1550 files)

-uses SQLite (has very bad locking, you cannot have the db open while creating it)

I will now look into inserting sqs/sqf scripts text into the db too

Anyway, the resulting db is quite fun to query.

My useless efford will continue  

Anyone have any megaserver with 100GB of space and no bandwidth limitations lying around?

[Yoma 0]

Lol i was amazed to find that sakakah even has an exe inside the pbo  

Pal2PacE.exe (i guess it's some kind of texture converter?)

I may have been in an older version (i have 2 versions and only one has it)

04E833C0CCD358F5182D54750D91B002 McN_hmmwv_des.pbo

has a zipfile HMMWVTextures.zip which is 1.5 MB.

I don't know if stuff like that matters, probably not.