This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=385

michalss, posted Sun Nov 16, 2014 9:59 pm (1826)

---

Hi all,

Can anyone fix the aluigi script for SoM ? Thank you

Examples:

Code:

https://dl.dropboxusercontent.com/u/38234344/discdata1.rar

Echelo, posted Mon Nov 17, 2014 3:09 pm (1834)

---

michalss wrote:

Hi all,

Can anyone fix the aluigi script for SoM ? Thank you

Examples:

Code:

https://dl.dropboxusercontent.com/u/38234344/discdata1.rar

quickbms must use standard zlib as the default ComType, cause I don't see it being set in his script for the pc files.
Xbox360 is using ComType XMemDecompress, the files inside the .arch05 are big endian too. So types are like LDNB, not BNDL.

I don't think it would take too many modifications to get it to work for xbox360, aside from the fact the the xmem chunks have no clear decompressed size in the chunks, rather, their total decompressed and merged size is stored in the main table. Currently, in C# I'm using the main table size and subtracting 65536 from it as I process a chunk, then once the last chunk that is less than 65536 bytes is reached, I know I'm at the end.

Side note, PS3 version is using Deflate, and just adding ComType deflate to his current script and changing the endianness should work.

Echelo, posted Tue Nov 18, 2014 10:41 pm (1843)

---

Here is my unpacker so far. Only PS3 and Xbox360 support for now. It uses quickbms to handle the XMem chunks, so quickbms needs to be in the same folder as this exe.

This will unpack the .arch05 files to a folder named after them, and unpack any .embb inside them.
I've cleaned up the output to dispose of the temporary compressed files and only keep the decompressed final results. (Saves space on extraction).
Also the .embb files are now unpacked to layerX_unpacked_bundles, and are merged back to the original file structure the game engine uses.
I verified that all md5 match for overwritten files throughout all archives on both ps3 and xbox360. So now the tool doesn't write the file if it already exists. (Saves a little more time)

Code:

usage: middleEarthArchiveUnpacker.exe  
example: middleEarthArchiveUnpacker.exe PS3 layer01.arch05
example: middleEarthArchiveUnpacker.exe XBOX360 layer01.arch05

Now that the file structure is sorted, a repacker may be possible.
Its weird how this game has multiple copies of the same files throughout different bundles, (kind of wasteful it seems), but with that now known, I can make a repacker soon.

Attachment removed, see the repacker further down in this thread

TON, posted Wed Nov 19, 2014 8:15 am (1845)

---

Echelo wrote:

Here is my unpacker so far. Only PS3 and Xbox360 support for now. It uses quickbms to handle the XMem chunks, so quickbms needs to be in the same folder as this exe.

This will unpack the .arch05 files to a folder named after them, and unpack any .embb inside them to _d/_b. (was trying to keep unpacked paths short)
The files in the _d folders are the decompressed files from the .arch05, the files in the _b folders are the files from the .embb.

Usage:

Code:

usage: middleEarthArchiveUnpacker.exe  
example: middleEarthArchiveUnpacker.exe PS3 layer01.arch05
example: middleEarthArchiveUnpacker.exe XBOX360 layer01.arch05

Thanks Echelo!
Any chance for repacker?

michalss, posted Wed Nov 19, 2014 10:32 am (1847)

---

yeah repack would be great

aluigi, posted Wed Nov 19, 2014 12:16 pm (1853)

---

Script updated to version 0.2.2, thanks.

raykingnihong, posted Fri Nov 21, 2014 4:15 pm (1898)

---

aluigi wrote:

Script updated to version 0.2.2, thanks.

Hello, my friend, thank you for sharing

michalss, posted Sun Nov 23, 2014 9:09 pm (1947)

---

aluigi wrote:

Script updated to version 0.2.2, thanks.

Lots of the things cannot be exported It only create header and files are always 1Kb of size, so it des not work correctly Also main arches does not work, it extract only 1 chunk 64kb

Examples :

Code:

https://dl.dropboxusercontent.com/u/38234344/embb.rar

Archives:

Code:

https://dl.dropboxusercontent.com/u/38234344/SoM_X360.rar

michalss, posted Mon Nov 24, 2014 6:28 am (1957)

---

Ok i figuret it out. All files are chunked with different size... Here is template for 010....

Code:

//--------------------------------------
//--- 010 Editor v3.1.1 Binary Template
//
// File:
// Author: michalss
// Revision:
// Purpose:
//--------------------------------------
local byte kompr;
local uint checker,trueer;

BigEndian();
    do {
        struct compfilestart {

            uint packed;
            trueer=FTell() packed;
            if (trueer < FileSize()) {
                checker = ReadUShort(trueer);            
            } else {
                checker=0;    
            }
   
            if (checker==22616) {
                byte *SPAM*[packed 2];    
            } else {
                byte data[packed];
            }    


        } record;

    } while(FTell()!= FileSize());

aluigi, posted Mon Nov 24, 2014 10:26 am (1959)

---

Well done michalss, indeed it's a chunked compression with 4 bytes alignment.
The alignment is probably per-file but, considering that all the stored files are aligned, I opted for a per-archive alignment which is exactly the same in this case.
Script 0.3
http://aluigi.org/papers/bms/others/sha ... mordor.bms

I removed also the creation of 0 bytes files.

michalss, posted Mon Nov 24, 2014 10:46 am (1960)

---

aluigi wrote:

Well done michalss, indeed it's a chunked compression with 4 bytes alignment.
The alignment is probably per-file but, considering that all the stored files are aligned, I opted for a per-archive alignment which is exactly the same in this case.
Script 0.3
http://aluigi.org/papers/bms/others/sha ... mordor.bms

I removed also the creation of 0 bytes files.

thank you, but still not there This chunk method is not normal. In some chunks in the end of the is XX number 22616. It is only on some of them and this breakpoint brake your extraction and decompression. If you look at my template you can see that im always checking if that breakpoint number is presen in the end of the chunk, coz if it is, then size of the chunk is about 2 bytes bigger. Not sure why this is like this, but ....

aluigi, posted Mon Nov 24, 2014 11:00 am (1961)

---

I tried the script with the samples you provided and it worked, the "XX" is not a break but is just padding for the alignement.

michalss, posted Mon Nov 24, 2014 11:13 am (1962)

---

ok try this file.... you will see https://dl.dropboxusercontent.com/u/382 ... r01.arch05

What you mean alligment pls?? Can you a bit explain pls ? How to calculated it?

aluigi, posted Mon Nov 24, 2014 12:48 pm (1963)

---

Some chunk sizes have AND 0xf0000000 instead of 0x80000000, script 0.3.1 out.

The padding/alignment is the amount of non-used bytes necessary to have data aligned.
If I have alignment 4 and I'm at offset 0x33 it means that the correct offset where reading data must be 0x34 so 1 byte unused.
If we are at offset 0 then it's 0, offset 1 is 4, offset 6 is 8 and so on.
The 'X' is the just unused byte and in fact you can see it only when the compressed data terminates in a non-aligned offset.

aluigi, posted Mon Nov 24, 2014 12:50 pm (1964)

---

In QuickBMS this is the math used for the calculation of alignment:

Code:

if(var2 && ((u_int)var1 % (u_int)var2)) {
    var1  = (var2 - ((u_int)var1 % (u_int)var2));
}

where var1 is the offset and var2 is the alignment (for example 4).

Echelo, posted Mon Nov 24, 2014 3:49 pm (1968)

---

Here is my full unpacker and repacker. Have fun!

Code:

usage: middleEarthArchiveTool.exe   
example: middleEarthArchiveTool.exe unpack layer01.arch05 PS3
example: middleEarthArchiveTool.exe unpack layer01.arch05 XBOX360
example: middleEarthArchiveTool.exe repack layer01.arch05 PS3
example: middleEarthArchiveTool.exe repack layer01.arch05 XBOX360

This newer version will unpack all of the .arch05 files to a directory named after them.
Their internal .embb files will be unpacked to a "all_layers_unpacked_bundles" directory that is created when the first .embb is unpacked.
On repack, any files modded in the "all_layers_unpacked_bundles" are included in the repacked .embb before its packaged into the .arch05.

I've also tested with a few texture mods, and the game still loads the repacked .arch05.


At this time, it will repack XBOX360 .arch05 one to one if no files were modified.
For ps3, I think edge zlib 1.2.3.0 or just zlib1.dll version 1.2.3 is needed to make the repacked .arch05 one to one if no files were modded. Atm, I'm getting slightly better compression then the game. Will tinker with the zlib flavor later..

I can add PC support as well but I don't have a PC .arch05 to look at, I assume I can re-use my ps3 code and just change deflate to standard zlib with a few changes to endianness.

Remember to delete the install data and install it again with your repacked .arch05 files.

middleEarthArchiveTool.zip

aluigi, posted Mon Nov 24, 2014 5:16 pm (1969)

---

You can find various samples here:
viewtopic.php?f=9&t=185

michalss, posted Mon Nov 24, 2014 6:47 pm (1974)

---

Echelo wrote:

Here is my full unpacker and repacker. Have fun!

Code:

usage: middleEarthArchiveTool.exe   
example: middleEarthArchiveTool.exe unpack layer01.arch05 PS3
example: middleEarthArchiveTool.exe unpack layer01.arch05 XBOX360
example: middleEarthArchiveTool.exe repack layer01.arch05 PS3
example: middleEarthArchiveTool.exe repack layer01.arch05 XBOX360

This newer version will unpack all of the .arch05 files to a directory named after them.
Their internal .embb files will be unpacked to a "all_layers_unpacked_bundles" directory that is created when the first .embb is unpacked.
On repack, any files modded in the "all_layers_unpacked_bundles" are included in the repacked .embb before its packaged into the .arch05.

At this time, it will repack XBOX360 .arch05 one to one if no files were modified. I've also tested with a single texture mod and the game still loads the repacked .arch05.

For ps3, I think edge zlib 1.2.3.0 or just zlib1.dll version 1.2.3 is needed to make the repacked .arch05 one to one if no files were modded. Atm, I'm getting slightly better compression then the game. Will tinker with the zlib flavor later..

I can add PC support as well but I don't have a PC .arch05 to look at, I assume I can re-use my ps3 code and just change deflate to standard zlib with a few changes to endianness.

Remember to delete the install data and install it again with your repacked .arch05 files.

Thx mate gonna test it Soon All i need to repack correctly layer01 there is text and fonts ... Can i repack with with bigger size of text then original?

Echelo, posted Mon Nov 24, 2014 7:10 pm (1975)

---

michalss wrote:

Thx mate gonna test it Soon All i need to repack correctly layer01 there is text and fonts ... Can i repack with with bigger size of text then original?

Yes, the file tables for both the bundles and the archives are re-written on repack to reflect changes in file sizes and addresses.