Mod Combiner+Loader

[azrapse 10]

Hi.

I have been looking at some mod combos out there, and it gave me the idea of writing a program to combine and load mods automatically.

I saw how easy was it to combine the Deadly Islands with the NoGrainyFilm mod, but other mods would prove to be more challenging.

I'm halfways through it, but I wanted to survey if there would be any interest for this. What do you think?

This is the basic functionality:

• The program is able to index the packed file tree in the data*.cc and patch*.pak files (that is, the packed game data files) to identify which folders inside the game folder are mod folders. For each subfolder in the game folder, check if it has any file or directory in the same places as the internal file system in the game data files. If it does, it's a mod folder. (100% completed)
  
• The program lists all mods in a checkbox list. The user can select 0 or several mods to be loaded.
  
• The user can give higher priority to the mods by sliding them up or down in the list. Higher priority mods are patched into the original files first, lowering the chance of them triggering a combination error.
  
• When the user clicks the big button to launche the game, the program does this:
  
  • Create a new mod folder with a temporary name ABC. (100% completed)
    
  • Mimic the file structure of both selected mods by copying all non conflicting files to the ABC folder. Alternatively, make symbolic links to save up disk space instead. (100% completed)
    
  • For each conflicting file, that is, a file that has been modified in more than one mod, calculate the diff between the original file (extracted on-the-fly from the game data files) and the modded file. (100% complete).
    
  • Copy the original file into the ABC folder, then patch it successively with the changes made to it by every mod. This should work fine as long as not two mods have modified exactly the same line in a text file, or they aren't binary files.
    
  • If any error occur, highlight in yellow the mod that provoked the error while combining the files and warn the user that some mod isn't combinable with the rest.
    
  • When all mods are combined, the game is launched with "-mod ABC" appended to the command line.
    

The program can already extract the original files on the fly (without having to unpack anything first), and uses the Google diff_match_patch library to calculate the individual patches. The program chooses the most recent original file if there exists several copies of a particular file (for example, a file in data00.cc can be overriden in patch00.pak, in that case, the program picks the one in patch as original file).

Opinions?

[species1571 64]

There are a couple of possible problems I can see:

First of all, there may be a situation where one of the combined mods is modding one line that the other mod assumes is not modded, or the other mod replaces that line with a whole other section.

The second possibility is that two mods might be able to run combined, but one is doing the exact opposite thing that the other is doing, or just something it wasn't expecting to happen, causing unpredictable results or crashes.

I would suggest just limiting it to giving an easy way to select mods and combining mods that don't mod the same files. Would the launcher also be able to integrate with the Steam version, which uses a different method of launching the game?

[azrapse 10]

Whenever two mods modify the same files, I was planning to consider those two cases.

[*] If they cannot be combined because they modify the same area of one file, then the launcher stays in a "Uncombinable mods" warning, and the game is not launched until the user de-selects one of the conflicting mods.

[*] If they can be combined because they modify different areas of one file, or different files, then the launcher attempts to launch the game. If the user selects two mods that do the opposite, well, it's his fault.

The launcher would run the carrier.exe program to launch the game. In Steam, running that program will run Steam first if needed, then launch the game. I have the Steam version and I can launch the game fine from a shortcut in my desktop with the usual -mod switch.

[disorder 1]

I think it's best to have a "code snippet injector". This was what I thought would be good for a mod manager. The code injector would insert chunks of programming into existing modable files, most of which are plain text. The injector could insert or replace strings at a given line number or char index on an and/or principle , that way you can write your code as small minimods and not have to distribute the whole file.

As for binary files, a scripted byte patcher could be good. Taking what newtie has developed and automating it so that it can inject small chunks of binary data(or converted plain text) into whatever files you want to mod would be awesome. Technical but awesome anyway.

The modding community is snowballing quite well now.

Edited August 5, 2013 by disorder

[azrapse 10]

I have thought about doing a mod manager. I thought it would be easier to inject code based on either replacing or inserting strings at certain line numbers/ char index.

What I'm trying here is a more advanced version of that. There is the concept of Diff between two versions of a file. The diff describes how a file changes from version 1 to version 2. That description of changes is something like this.

File 1:

AAABBBCCCDDDEEE

File 2:

AAABBBXXXDDDEEE

The diff for that file would be

- Go to position 4.

- Around that position, look for the partter BBB, that won't change but marks the start of the change.

- Find CCC after that.

- Replace that with XXX.

- Find DDD after that, that marks the end of the change.

The patch tool can use those instructions to patch a file with those modifications. But it has a twist, because the patch tool can try to find the best match that doesn't necessarily need to be exactly at position 4, but around it. This allows to patch changes to files that have been modified since the original.

For a perfect example of what I'm talking about, and for the source code of the library I'm using, go here.

I wouldn't extract and the rebuild the pak files since there is already mod folder structure to override anything in them. If you want to make one, ask newtie who has already cracked the dbexport.bin file.

I won't extract and repack the data files. The launcher is able to read the packed data files because it needs to find the original unmodded copy of every possible file that the modder has changed, to know which particular changes he or she performed. With this knowledge, the launcher creates a new mod that is the combination of

- all modified files from every mod that don't collide with each other...

plus, for every file that has been modified in more than one mod

- the original unmodded file, then patched with the changes made by mod A, then patched with the changes made by mod B, etc...

This creates a file structure in a subfolder. Basically a new mod folder. Then the game is launched with that ad-hoc mod.

[disorder 1]

I think I get what you mean, but you said "This should work fine as long as not two mods have modified exactly the same line in a text file"

If you do it and you replace XXX based on char/line index then there is a chance you can overwrite parts or all of other peoples code(or exclude it completely) just because it resides in the same area of the file. Most of the tweakable code is object based psuedo-c parsed by the game. You need to inject what you want into a new position, adding to what was already there. The only time you would need to patch is when edits are made to the existing functions.

What you want to do is first get the original most up to date version of the file for a reference then gather all the new lines from your modded file(s) into a stringlist and in the order they were written. Parse the created stringlist and see which entries are new and which entries match an existing line by iterating through it.

You can use an approximation algorithm based on checking each character and then incrementing a number based on a match in same char position. If you want to a second value, also add the chars appearance in the whole line.

Divide the numbers into 100 to get a percentage to get an approx matching chars/probability. Set the base minimum matching chars to 0% so if there are no matching chars, insert the line, if there are a certain percentage of matching chars then check the line for differences.

Edited August 6, 2013 by disorder
cleaner, easier to read

[azrapse 10]

Hi, Disorder. First of all, thanks for your interest and comments. :)

In respect of what you say, that's why I'm not reinventing the wheel, but instead, I am using (or so far, researching which to use) a well-known and stable algorithm, such as diff3 for doing this.

Code repository programs have been using three-way-merge algorithms since around 2000 and I think most of our problems can be solved with one of those algorightms that are well tested and ready. And if these algorithms cannot solve it automatically, the only way would be the user himself to resolve the conflicts. But you can aswell just show an error message stating that the mods are incompatible.

To make it clearer what the program already is doing, this in an example I have just fed it, and the output it has produced. Again, the merit is not mine, but of those that invented this three-way-merge algorigthms. This is what the diff3 tool can do by itself. Let's say that the original game file is this one

Original:

function OnUpdate()
{
   DoThis();    
   XYZ = "Original Value";
   DoThat();
}

function OnCarrierAttacked()
{   
}

This original file can be extracted on-the-fly from the datafiles by my launcher to feed it as the original common file for both mods. Currently, the launcher considers that the files in the patch??.pak datafiles are more recent than those in the data??.cc files. Please someone correct me if this assumption I make is mistaken.

This is the same file in after modder A changed it for his modA.

modA:

function OnUpdate()
{
   DoThis();
   XYZ = "Original Value";
   modAxxx = "custom modA value";
   DoThat();
}

function OnCarrierAttacked()
{   
   print("ModA Report: We are under attack!";
}

And this is the same file after modder B changed it for his modB.

modB:

function OnUpdate()
{
   DoThis();
   CallMyModdedRoutine();
   XYZ = "Original Value";
   ABC="ModB Modded thing";
   DoThat();
}

function CallMyModdedRoutine()
{
   ABC = ABC+" / "+XYZ;
}

function OnCarrierAttacked()
{
   print("ModB Output:"+ABC);   
}

Well. After feeding the diff3 algorithm in my program, with the original, the modA version, and the modB version, it already produces this merge version:

function OnUpdate()
{
   DoThis();
   CallMyModdedRoutine();
   XYZ = "Original Value";
   modAxxx = "custom modA value";
   ABC="ModB Modded thing";
   DoThat();
}

function CallMyModdedRoutine()
{
   ABC = ABC+" / "+XYZ;
}

function OnCarrierAttacked()
{
   print("ModB Output:"+ABC);   
   print("ModA Report: We are under attack!";
}

That is quite good. Don't you think?

I insist have not programmed the diff3 algorithm myself. I have just borrowed it from some open source code revision system, and I have wired it to the rest of the launcher infrastructure, that it the one that provides it with the original file, the modded files, replicates the combined file structure of all mods in the combined mod folder, etc.

Right now I'm just tweaking it to increase the performance with the size of files the game mods usually deal with. I'm trying out different implementations of the three-way-merge algorithms to finally pick the one that throws the best results for us.

When I said that the program would fail if two modders have modified the same line of text, I didn't mean line of text as of char/line number based. But by content and context.

If, for example, both modders would have changed the line in OnUpdate that is

XYZ="Original Value";

to something else... for example, modA changes that to

XYZ="Value1";

and modB changes it to

XYZ="Value2";

Then the merge algorithm still can find that line in both versions even if it has been moved down or up because of previous content added to the file (that is no problem), but it doesn't know how to put both changes together because they are modifying the same line and it doesn't know which mod is right and which is wrong. In that case, it would be a conflict not automatically resolvable, and the launcher would claim the mods are incompatible. Or maybe one mod should always have preference over the rest (I was thinking for a while on letting the user select the precedence of the mods by vertically reordering them on the list).

Once I'm happy with how non-incompatible files merge, I will move onto that, but so far I'm still trying to increase performance, because some bigger files are taking several seconds to merge, and that is bothersome.

[disorder 1]

The pak files are overriding the cc files, if I am not mistaken.

I think that merge might work well. Sorry I misunderstood you about the patcher. I think the loading order will be good for working out the priority, top of the list is first. As it goes down the list it can check each one to see if any conflicts come up and then highlight them. A few seconds is ok, just as long as it isn't minutes.

Good job anyway

[azrapse 10]

Greetings!

I'm happy to announce that I have managed to increase the performance of the merging tool and now it does everything in less than two seconds!

I'm now working on the interface, that is just barebones right now, but I wanted to show you the results already.

This is the prototype of the launcher:

http://azrapse.es/carriercommand/sshot05.jpg (113 kB)

I have selected there a bunch of mods, and clicked on the Launch button. That starts the process of making a combined mod. There are no conflicts, so the game starts okay.

Deadly Islands running at the same time as Deck Views (special camera in this screenshot), as well as GripMod, and NoFilmGrain:

http://azrapse.es/carriercommand/sshot01.jpg (251 kB)

You cannot see it here, but the grip mod makes the island controllable walruses turn on a dime! :)

http://azrapse.es/carriercommand/sshot02.jpg (341 kB)

Funny effect. When you use the special cameras added by Deck Views while looking at a manta...

http://azrapse.es/carriercommand/sshot03.jpg (444 kB)

...and then press K, Deadly Islands will put you in TimeWarp and it will look this awesome. :D

http://azrapse.es/carriercommand/sshot04.jpg (270 kB)

When the mods don't work because they collide with others, the launcher indicates it:

Optionally you can tell the launcher to indicate in the combined code where the collision with the problematic mod occurred:

http://azrapse.es/carriercommand/sshot07.jpg (181 kB)

Next I will work on making the launcher prettier and attempt to deal better with conflicting mods.

Any suggestions?

[disorder 1]

Nice work. I noticed it is conflicting on text files. Maybe you can ignore them or just auto-combine them into a new text file if it is called readme* I don't think there are many text files used by the game anyway.

[azrapse 10]

Thanks, Disorder.

It's not conflicting on the readme.txt file. It says "compatible". That means that the merge is successful without any conflict. It just puts the contents of the readme of a mod at the begining of the file, and the contents of the other at the end.

However I wonder what would happen when combining xml or other kind of structured text files. I have not had the chance to test that because so far no mods have collisions on those files.

In respect to the binary data files, I don't think it's possible to merge them using a standard diff routine. I guess I would need to use some kind of special reader adapter for every format. And in some cases it wouldn't be possible at all, because... how you merge two different sounds or two images?