Data pipeline - packing

[suma 8]

I suggest now, before any real work is started, to dedicate some time doing some dull work: creating a data pipeline, so that anyone can at any time know how to get the CWR into the ArmA.

If someone is willing to undertake this, please answer here.

More detailed overview follows.

The packing pipeline ideally should be:

- get one batch file from some prominent location in the SVN (install.bat at the root of the repository)?

- run the batch file

- as a result you will get a bunch of PBOs packed into some defined location (e.g. temp folder in the root of the working copy) or even copied into your ArmA location.

As an intermediate step a manual process may be documented instead, how to pack data, but it is likely you will soon find it is easier and more fruitful to create the bat file itself.

Some very basic content should be tested first, perhaps only a config base addon (e.g. an editor extension, or some island variant with different grass, or a unit derived from some ArmA one, with only some parameters changes)

As this is something everyone will be doing quite frequently, having implemented this will be a big time saver for everyone, especially for those of us who are more artist oriented.

I have no time for implementing this, but if anyone needs, I can give an advice, review proposed packing bats, etc.

One thing you definitely need for this and we will provide it promptly is the FileBank tool to have a reliable way how to create pbos.

[Pennywise 0]

Guys, I think I might already have this taken care of this. I recently installed cruisecontrol on the svn box. For those unfamiliar with cruisecontrol, it is an automated build tool that uses ant to bring tasks together and automate them, http://cruisecontrol.sourceforge.net/. We use cruise control at work with Clearcase and it works great! There is also a plugin for SVN. Essentially, what will occur is when you checking your source and binary file, cruisecontrol will detect the change in SVN and kick off a build. The build will currently create all pbos for each corresponding component and place them in a distro directory. This distro directory will be available via web after each new build. I can provide further details when I near completion. Otherwise, let me know if somebody already has something in the works.

[raedor 8]

Guys, I think I might already have this taken care of this. I recently installed cruisecontrol on the svn box. For those unfamiliar with cruisecontrol, it is an automated build tool that uses ant to bring tasks together and automate them, http://cruisecontrol.sourceforge.net/. We use cruise control at work with Clearcase and it works great! There is also a plugin for SVN. Essentially, what will occur is when you checking your source and binary file, cruisecontrol will detect the change in SVN and kick off a build. The build will currently create all pbos for each corresponding component and place them in a distro directory. This distro directory will be available via web after each new build. I can provide further details when I near completion. Otherwise, let me know if somebody already has something in the works.

We can try that, it sounds very good.

As soon as we have the tools I'll try out FileBank anyway, it works reliable and is written exactly for that job We'll see.

[berghoff 11]

Hopefully it will avoid problems like this:

Happens when a newbie like me tries to get stuff working =)

[raedor 8]

I doubt that this has anything to do with the packing. I guess you renamed the model... then you have to write a new CfgModels

[berghoff 11]

I doubt that this has anything to do with the packing. I guess you renamed the model... then you have to write a new CfgModels

Well I downloaded the west infantry (CWR_west) and packed it and a lot of things were messed up after loading it guess I did something wrong.

[raedor 8]

I doubt that this has anything to do with the packing. I guess you renamed the model... then you have to write a new CfgModels

Well I downloaded the west infantry (CWR_west) and packed it and a lot of things were messed up after loading it guess I did something wrong.

Whoops, no, that way my fault. I uploaded an outdated version...

[suma 8]

Essentially, what will occur is when you checking your source and binary file, cruisecontrol will detect the change in SVN and kick off a build.

Very nice, glad to hear that. Still, it would probably be good to have some easy and standard way how to test my data in game before I commit them to SVN. Such testing should be a common practice, avoiding buggy versions appearing in the repository.

[berghoff 11]

Ok now it gets more complicated than packing one folder.. My first attempt to pack all that data currently on SVN and run ArmA resulted into a error at startup .

[raedor 8]

Ok now it gets more complicated than packing one folder.. My first attempt to pack all that data currently on SVN and run ArmA resulted into a error at startup .

Until we get the tools from BI (including Filebank), you have to do it manually.

Quote[/b] ]If you pack each folder (in addons\ which does not have subfolders (beside data\ and put the pbos in your ArmA addons directory, you should be able to play the two Demo missions.

[berghoff 11]

Until we get the tools from BI (including Filebank), you have to do it manually.

All tools have been released. =) I think filebank works without some sort of activation.

[raedor 8]

Pipeline done.

[berghoff 11]

Nice work.

[raedor 8]

Ah, forgot the link with soem useful notes about it:

https://cwr.xdev.bistudio.com/wiki/PBO_Packing

[Pennywise 0]

Hey Guys, I'm trying to run this PBOPacker.bat script and nothing seems to be happening except the script pipes out a message:

Quote[/b] ]

Executing script with "[]"

Packing...

<result><null></result>

This is a rather peculiar tool as all it seems to be doing is executing other tools in b:, if that. The pbopacker.bat seems to take parameters. However, non are provided in the the wiki. Contents of PBOPacker.bat:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

@echo off

t:\Oxygen\O2Script.exe -d -a "P:\CWR\BatchFiles\packing.bio2s" %1 %2 %3 %4

Should we be passing in parameters to this batch file? Could someone please clarify there isn't something else that needs to be done? Here is my armapath.txt:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

project= "p:\CWR";

UseSS = false;

SSLogIn ["\\core1\vssdatap\datap\srcsafe.ini","$/OFP2","p:\OFP2"];

UpdateAddonDir = "p:\CWR\addons";

UpdateDtaDir = "p:\CWR\data";

UpdateDist = "x:\OFP2Dist";

I appreciate any help you can provide

[raedor 8]

All seems correct. Do you have your SVN checkout in p:\ as it should be?

Also, do you have oxygen in C:\BIS\Oxygen\? This is not a must, as you can also change the paths in the packing script(s).

[Pennywise 0]

Hey Raedor,

We're still having no luck with this I'm wondering, I was looking through the bio2 script and we don't have this file in the repository:

<table border="0" align="center" width="95%" cellpadding="0" cellspacing="0"><tr><td>Code Sample </td></tr><tr><td id="CODE">

if (L1Pack) then {

shellCmd ("cmd /c for /r p:\CWR\temp\bin\L1"+ mainPath@[2] +" %F in (*.dep) do del %~F");

} else {

shellCmd ("cmd /c for /r p:\CWR\temp\bin"+ mainPath@[2] +" %F in (*.dep) do del %~F");

};

Is this 'L1' file needed? Should it be in the repository?

[raedor 8]

Did you follow these instructions?

Usually the packer should go through all models (which are chosen to be packed), binarize them, put them in the temp directories (and create them, if needed) and pack the pbo after that (so that the models only have to be re-binned when they have been changed).

Tomorrow I'll try to make a vanilla installation here to see if I encounter similar problems, but currently it is working here.

[Pennywise 0]

Yes, I have been through those instructions several times, step by step. I'm wondering if it has something to do with placement of files, but everything else seems to be working and Oxygen loads up fine. I'll go back and recheck everything again.

[raedor 8]

Sorry for the delay. I now tested it on another comp, it is working. Also it is working for Planck. I try to catch you on ICQ the next days.

[Pennywise 0]

Raedor, I appreciate you checking things out. I'm going to do a clean install and checkout a new working-copy. Hopefully that will clear things up.

[raedor 8]

The main Problems Planck had were

a) P: has to be the root of the SVN checkout.

b) He had spaces in the ArmA path (this can be workaround'ed by using eg c:\test\ as UpdateDir in armapaths.txt, you have to copy the files manually then).

c) If you run the script the first time, you have to accept that "it is a folder", the script will ask you for that

After that, it worked...

[Planck 1]

Quote[/b] ]c) If you run the script the first time, you have to accept that "it is a folder", the script will ask you for that

Yes, when it found my path to the game had a space in it (Bohemia Interactive) it screeched to a halt and asked if that was a file name or a directory.

Once you gave it an answer however it carried on, or it seemed to carry on.

What it actually did was:

The path to the game I gave it was C:\Games\Bohemia Interactive\ArmA

It got to the Bohemia Interactive bit and stopped and asked if Bohemia was part of a file name or a directory name, as explained.

I selected 'D' for directory and it then skipped off into the distance, happy once more.

Unfortunately it then produced a folder in C:\Games called 'Bohemia' and put the pbo in there.

Moral of the story 'Spaces in pathnames are best avoided'.

[ebud 18]

This may seem like a silly question, but why does this process need to be 10x more complicated than it needs to be? I can barely get the tools setup, my Visitor worked for a whole 2 days and now for some reason crashes each time, I have no idea what to do with my svn folders as I'm scared to touch them after uploading all of them on accident (I don't have the time to learn the whole svn process for one project, just don't), I have no idea what half the tools do let alone how I should use them, same within the tools. I'm at the point where all I want to do is make the edits or files I need to and upload them to an ftp and tell you all where the files are and let someone else integrate them, lol.

This whole setup and process to me is insanely more complicated than it needs to be, but I know there are good reasons for it.

Just my rant since I lost a few irreplaceable files after trying to setup the bis folders, a new bulldozer folder in the x folder or whatever and then moving folders from my old bulldozer folder to the new one for the new O2. My fault and this is a good example I'm sure of the need for backups like the svn revisions, lol, but this is all just overcomplicated where I can't see much of a reason for it, but that's just me being old fashioned.

Sorry for the rant, and I'll just stick with making what I have to the old way and post things the old way as needed.

[raedor 8]

This may seem like a silly question, but why does this process need to be 10x more complicated than it needs to be? I can barely get the tools setup, my Visitor worked for a whole 2 days and now for some reason crashes each time, I have no idea what to do with my svn folders as I'm scared to touch them after uploading all of them on accident (I don't have the time to learn the whole svn process for one project, just don't), I have no idea what half the tools do let alone how I should use them, same within the tools. I'm at the point where all I want to do is make the edits or files I need to and upload them to an ftp and tell you all where the files are and let someone else integrate them, lol.

This whole setup and process to me is insanely more complicated than it needs to be, but I know there are good reasons for it.

Just my rant since I lost a few irreplaceable files after trying to setup the bis folders, a new bulldozer folder in the x folder or whatever and then moving folders from my old bulldozer folder to the new one for the new O2. My fault and this is a good example I'm sure of the need for backups like the svn revisions, lol, but this is all just overcomplicated where I can't see much of a reason for it, but that's just me being old fashioned.

Sorry for the rant, and I'll just stick with making what I have to the old way and post things the old way as needed.

Yep, I know that the tools are rather complicated... SVN does the rest

As long as you don't want to check things ingame immediately, this is okay. As soon as we have some more things done, I'll upload packages with pbos for testing.

If you have something done you can send it to me or someone else who works on the same project.

BTW, which tool are you using for SVN?