This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=3251

lsaint, posted Sat Oct 15, 2016 5:12 pm (18068)

---

Hey guys,

I have a question, I'm looking at a game archive and the structure resembles lz77, RLE.

I know this is broad..but am I on the right track, it appears that the compression is starting around x15







LZ77 compression scheme.

The compressed data will be in the pattern of 1 bite that is gathered by 8 flag bit and 8 chunks.
Ex) [Flag][chunk0][chunk 1][chunk2][chunk3][chunk4][chunk5][chunk6][chunk7][flag][chunk0][chunk1]....

Each bit of the flag indicates whether the chunk was compressed or not.1 means compressed chunk, 0 will be the chunk has not been compressed and has a size of 1 bite.
#0bite represents # 0 chunk, while #7bit represents #7 bit.


There are 4 types of compressed chunk.

If the first bite of the chunk's lower nibble has a value of 0, thats mean it is a long compression.1 means RLE, 2means Copy, the rest means compressed.
Long compressed chunks are formed with 3 bites.

If the 3bite comes in the order of [H0:L0][H1:L1][H2:L2] ,( H is upper nibble,4 bit), L is lower nibble. They are enclosed by bite. L0 is 0.

The enclosed 12 bit which is the order of H1 L1 H0, helps you to understand how much backwards you have to refer in advance starting from current cursor.

Add 0x10 to 8bit which is enclosed by H2 L2, that tells you how much bite you have to bring.

If it is 30 12 24 , after seeing L0 which is 0, you know it is a long compressed chunk . 0x123 tells you how much you have to refer backward from previous.
and (0x24 0x10)is how much bite you have to bring from there.

RLE chunks are formed by 2 bite.
L0 has to follow 1. Add3 to H0 tells you how many times you have to repeat.[H1:L1]tells repeating bite.
If it 21 78, that means 0x78 words was repeated 2 3 times.

Copy chunks are formed by 2 nbite.
In case L0 is 2, add0x12 to H1 L1 H0, you will know how much bite you have to copy. You just have to copy the data shows after that.

Compressed chunk is formed by 2bite.
The value of L0 same as previously brought data's bite. H1 L1 H0 tells you how much you have to go backward from the current cursor.

There are 3 points in total.
1. The point that indicates from previous to present .
2. The point that indicates the present of compressed input file.
3. After decompressing and saving, the point that is used for output.


It reads one bite from the input point. Input point will move automatically.
After seeing the bite, it will make an assumption that it is a copy chunk.

After reading another bite, it will calculate the length of n. input point will move automatically.
for(i = 0; i c = fgetc(input point);
fputc(c, output point);
Previous [previous point ] = c; previous point ;
}

Previous size was 4kb. Since the range of previous point was 0~4095, after4095, instead of 4096, it will move back to 0.
Previous point= (previous point 1) % 4096;

aluigi, posted Sun Oct 16, 2016 9:11 am (18084)

---

Are you sure this algorithm is not implemented yet in quickbms?
Have you tried the compression scanner?
Maybe provide the original file.

lsaint, posted Sun Oct 16, 2016 6:26 pm (18093)

---

Here is an SD0 archive I've removed from the main file.

aluigi, posted Sun Oct 16, 2016 7:15 pm (18094)

---

The CLZ77 algorithm of quickbms gives a possible result but I don't think it's the correct one.
Why don't you make a C or Java or .NET implemenation of what you figured?

lsaint, posted Sun Oct 16, 2016 7:34 pm (18095)

---

Thank you for the input, I have a program from a Japanese game development website called westside, that extracts the files...
its just a headache, because everything I've tried doesn't bare fruit, both programming and bms wise

Also, for your own records, I'll give you a copy of the program.

aluigi, posted Mon Oct 17, 2016 7:21 am (18106)

---

I just dumped the whole decompression function to use it directly in quickbms, anyway it's easy to decompile or converting it to C by hand:

Code:

set MEMORY_FILE10 binary "\x83\xec\x18\x55\x57\x8b\x7c\x24\x24\x33\xc0\x33\xc9\xba\x0c\x00\x00\x00\x8a\x67\x0a\x8a\x4f\x08\x8a\x47\x09\xc1\xe0\x08\x0b\xc1\x89\x44\x24\x1c\xb8\x00\x00\x00\x00\x88\x44\x24\x0b\x88\x44\x24\x0a\x8b\xe8\x0f\x86\x72\x01\x00\x00\x53\x8b\x5c\x24\x2c\x56\x8a\x44\x24\x12\x84\xc0\x75\x0d\x8a\x04\x3a\xc6\x44\x24\x12\x08\x88\x44\x24\x13\x42\xf6\x44\x24\x13\x01\x75\x0d\x8a\x0c\x3a\x42\x88\x0c\x2b\x45\xe9\x19\x01\x00\x00\x8a\x04\x3a\x8a\x4c\x3a\x01\x88\x44\x24\x18\x83\xc2\x02\x24\x0f\x88\x4c\x24\x14\x88\x44\x24\x20\x75\x3a\x8b\x74\x24\x14\x8b\x44\x24\x18\x81\xe6\xff\x00\x00\x00\x25\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe8\x04\x03\xf0\x33\xc0\x8a\x04\x3a\x83\xc0\x10\x74\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x42\xe9\xc5\x00\x00\x00\x3c\x01\x75\x48\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\x8d\x3c\x2b\x8a\xd8\xc1\xe9\x04\x8a\xfb\x83\xc1\x03\x8b\xc3\x89\x4c\x24\x1c\x8b\xf1\xc1\xe0\x10\x66\x8b\xc3\x8b\x5c\x24\x30\xc1\xe9\x02\xf3\xab\x8b\xce\x83\xe1\x03\xf3\xaa\x8b\x7c\x24\x2c\x8b\xc6\x03\xe8\xeb\x79\x3c\x02\x75\x3f\x8b\x44\x24\x14\x8b\x4c\x24\x18\x25\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe0\x04\xc1\xe9\x04\x8d\x34\x3a\x8d\x3c\x2b\x8d\x4c\x08\x12\x89\x4c\x24\x1c\x8b\xc1\xc1\xe9\x02\xf3\xa5\x8b\xc8\x03\xd0\x83\xe1\x03\x03\xe8\xf3\xa4\x8b\x7c\x24\x2c\xeb\x36\x8b\x74\x24\x14\x8b\x4c\x24\x18\x8b\x44\x24\x20\x81\xe6\xff\x00\x00\x00\x81\xe1\xff\x00\x00\x00\xc1\xe6\x04\xc1\xe9\x04\x03\xf1\x25\xff\x00\x00\x00\x76\x0f\x8b\xcd\x2b\xce\x45\x48\x8a\x0c\x19\x88\x4c\x2b\xff\x75\xf1\x8a\x44\x24\x12\x8a\x4c\x24\x13\xfe\xc8\x88\x44\x24\x12\x8b\x44\x24\x24\xd0\xe9\x3b\xe8\x88\x4c\x24\x13\x0f\x82\x9e\xfe\xff\xff\x5e\x5b\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3\x8b\xc5\x5f\x5d\x83\xc4\x18\xc3"
idstring "SD0\0"
get ZSIZE long
get SIZE long
savepos OFFSET  # unused because the function wants offset 0
comtype calldll "MEMORY_FILE10 0 stdcall RET #INPUT# #OUTPUT#"
get NAME basename
get EXT extension
string NAME   "_unpack."
string NAME   EXT
clog NAME 0 ZSIZE SIZE

lsaint, posted Mon Oct 17, 2016 3:14 pm (18117)

---

Dude,

you are amazing

Thank you!

lsaint, posted Tue Oct 18, 2016 12:28 am (18130)

---

Thank you,

I was wondering if I am doing something wrong? I assumed quickbms could open the MRG files, the SD0 Files dumped.
the one for legend of the dragon I changed it to "MRG\0"

I'm new to bms, are they just totally different?...hmm wait..I answered my own question.

lsaint, posted Tue Oct 18, 2016 12:53 am (18131)

---

Ok,

I should see if I can figure this out, bms seems pretty amazing...let's see if a Newb can war game through this

aluigi, posted Tue Oct 18, 2016 5:19 am (18138)

---

The script was just a decompressor.
The MRG format doesn't seem so simple, it's not a classical archive.

lsaint, posted Tue Oct 18, 2016 6:46 am (18152)

---

Yes,

with a little work I figured out that the MRG's are a variant of an animation format know as MNG,
thank you for your help.

This project just gets, more and more indepth

lsaint, posted Wed Oct 19, 2016 3:02 pm (18168)

---

I'm kind of at Empasse, good it be a graphics format "TX" represents individual Textures, and PSET sets the dimensions of
the textures.

I can't seem to find a program that can open it.

lsaint, posted Fri Oct 21, 2016 5:33 pm (18209)

---

I have one more question

how would I loop the bms script to go through the complete file?

aluigi, posted Fri Oct 21, 2016 5:43 pm (18211)

---

What you mean?
Isn't just one compressed file?

lsaint, posted Fri Oct 21, 2016 6:07 pm (18217)

---

It comes from a larger archive..
meaning there are several SD0's in one MKD (the Main file), but the archive is large so I couldn't post it here

so have it pull all of the SD0's out of the main file

aluigi, posted Sat Oct 22, 2016 7:10 am (18228)

---

Post it on mega.nz and paste the link here.

lsaint, posted Sat Oct 22, 2016 10:19 am (18233)

---

Hello,

here is a link to a full MRG File.

https://mega.nz/#!DRNHQJbT!eA_5HADq1ykh ... ZENH_JJ4BQ

aluigi, posted Sat Oct 22, 2016 12:36 pm (18234)

---

http://aluigi.org/bms/gundam_gihren.bms

Now it's clear that these MKD archives are just raw containers and there are other information (including names probably) in smaller index files.
You should check the list of your files, probably there is a small file called ZZZPSP1.*

lsaint, posted Sat Oct 22, 2016 3:34 pm (18239)

---

Yup,

aluigi, posted Sat Oct 22, 2016 3:43 pm (18240)

---

I don't see names or useful references.
Anyway it's not necessary, the scanning feature of the script does the job so no problem

lsaint, posted Sat Oct 22, 2016 3:49 pm (18242)

---

Oh,

I attached the wrong file... (I don't think there are any names)

for some reason in other MKD files it crashes with this code

aluigi, posted Sat Oct 22, 2016 6:59 pm (18250)

---

Can you upload the MKD file that crashes?
Remember that I don't have the game so everytime you notice a problem you have to upload the file causing it.

lsaint, posted Sat Oct 22, 2016 7:45 pm (18255)

---

Sorry,

Common sense escaped me for that moment

https://mega.nz/#!WVMHiYqI!bq6Heid3zFQe ... 11mGRgdtrI

also because it uses chunks...the reimport function doesn't work, correct?

aluigi, posted Sat Oct 22, 2016 8:18 pm (18256)

---

My fault, it was a stupid bug I fixed some seconds after the upload of the script but forgot to reupload it
Script 0.1.1