This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=16123

wood5568, posted Thu Nov 25, 2021 11:57 am (67681)

---

Hi all, I need help extracting the dialogue files from the game and edit them.
The uses the same game engine as Tokyo Xtreme Racer 0.

The character dialogue texts in this game are textures, I found the textures being contained within the .dat file of the game. The .dat contains multiple TM2 files.


I tried splitting them into individual TM2 files and opening them in TM2 Rainbow and Noesis. Rainbow is unable to read the texture and Noesis just crashes upon opening them.


Appreciated if anyone could help out on this one, would be useful for making a English patch for it.

Below is the sample file of the .dat that contains the textures (requires splitting) :

BloodRaynare, posted Fri Nov 26, 2021 12:54 am (67695)

---

The TM2 files are compressed, that's why it crashed in Noesis/doesn't open in Rainbow. Unfortunately, the compression algorithm is unknown, so nothing can be done ATM. In order to crack the compression algorithm, you'll have to reverse engineer the game which is not an easy task, especially if the game didn't leave any debug symbols intact, by using Ghidra or PCSX2 debugger.

wood5568, posted Fri Nov 26, 2021 3:43 am (67697)

---

BloodRaynare wrote:

The TM2 files are compressed, that's why it crashed in Noesis/doesn't open in Rainbow. Unfortunately, the compression algorithm is unknown, so nothing can be done ATM. In order to crack the compression algorithm, you'll have to reverse engineer the game which is not an easy task, especially if the game didn't leave any debug symbols intact, by using Ghidra or PCSX2 debugger.

I see, thanks for the info. Probably will have a look at it again some other time, still a noob on reverse engineering.

Thanks again.

Allen, posted Mon Dec 06, 2021 5:23 pm (67992)

---

I am interested in the compression algorithm of this game. I tried the reverse decompression algorithm these days and finally made progress.
I found that there are 5 custom compression formats, but only 4 are used in the game.
I have completed 4 decompression algorithms, and I will release the tool later.

The decompressed file is in standard TIM2 format.
The structure of .dat is as follows:

Code:

//------------------------------------------------
//--- 010 Editor v12.0.1 Binary Template
//
//      File: Wangan Midnight PS2 .dat TIM2 textures pack format
//   Authors: Allen
//------------------------------------------------
DWORD   numFiles;
DWORD   offset[numFiles];

local DWORD size[numFiles];
local int i;

for (i=0;i{
    if (i!=numFiles-1)
    {
        size[i] = offset[i 1] - offset[i];
    }
    else{
        size[i] = FileSize() - offset[i];
    }
}
for (i=0;i{
    FSeek(offset[i]);
    struct{
        byte    CompressType;//0=NO COMP,1=Unknown COMP1,2=Unknown COMP2,3=Unknown COMP3,4==Unknown COMP4 NOT USE,5=Unknown COMP5
        ubyte   Size[3];//Big endian 3bytes / 24 Bits DWORD.
        local DWORD DecmpressSize = ((DWORD)Size[0] << 16) | ((DWORD)Size[1] << 8) | (DWORD)Size[2] ;
        byte    data[size[i]-4];
    }file;
}

I think that after decompressing and modifying the texture, you only need to modify the compression type to 0 in the file header, and change the decompression size to the file size, and the modification will take effect. I found through the code that when the compression type is 0, copy the data directly. No need to decompress the data.

Allen, posted Wed Dec 08, 2021 5:39 am (68023)

---

This tool supports unpacking and decompressing TIM2 files in .dat.
It also supports decompressing a single file, unpacking the original file without decompressing the data.

Requirements: .Net Core 3.1

*EDIT*:
Fixed Type2 decompression.
Upload source.

wangan_midnight_compress.zip

wood5568, posted Fri Dec 10, 2021 5:43 pm (68142)

---

Allen wrote:

This tool supports unpacking and decompressing TIM2 files in .dat.
It also supports decompressing a single file, unpacking the original file without decompressing the data.

Requirements: .Net Core 3.1

Thanks a bunch for looking into this! Works well for the PS2 version (also tested with the arcade versions). Only a few textures that doesn't seem to be working, but most of the decompressed properly. Saved up a lot of time.

I think the UI part of it isn't fully compatible for the PS2 version but for works with the arcade version however (not fully but it got most of them right compared to the PS2 ones.) They're usually next to the character texts. I could provide the elf and samplesfor the arcade version if it has any difference needed to look at.

Also regarding to applying the patch, do I just concatenate the TM2s files into a single .dat file, then use aluigi's txr script to reimport it?

aluigi, posted Fri Dec 10, 2021 8:27 pm (68158)

---

@Allen
Are those 4 compressions some known algorithms?

They look like a mix of rle and lzss but I want to be sure if they are already available in quickbms with other names.

Allen, posted Sat Dec 11, 2021 2:25 pm (68180)

---

@wood5568

I checked the file and found that there were indeed some Type2 compressions that were not decompressed correctly, which has now been fixed.
If there are still unsupported formats, you can upload samples.
For the PS2 version, I used dynamic debugging and static analysis assistance. If the arcade version only uploads ELF, I may not be lucky enough to find the relevant decompression code.

.dat is a container of TIM2 compressed files. You need to repackage the modified TIM2 file as .dat, and then use aluigi's txr script to re-import the .dat.
I am here to provide a quickbms script to unpack .dat.
The script has the same effect as my program "-o" mode unpacking(only unpack without decompression), and it is only used for re-importing here.
When you finish modifying the TIM2 file, you need to insert 4 bytes into the header of the TIM2 file. The first byte is the compression type, write 0, and the last 3 bytes are the file size is big-endian storage.
For example, 01 00 08 60. Compression type 1, decompression size 0x860. But we need to change the compression type to 0. Then change the decompression size to the file size because there is no compression.
Then rename the file according to the corresponding index name to the name after unpacking my program or quickbms. Then replace the corresponding file. Then re-import the file to .dat.
But whether it works, I think you need to test first.

Code:

#Quickbms script
#Wangan Midnight PS2 .dat unpack

get numFiles long
xmath maxIndex "numFiles - 1"

for i = 0 < numFiles
   get OFFSET long
   savepos nextOfs
   if i != maxIndex
      get SIZE long
      math SIZE - OFFSET
   else
      get SIZE asize
      math SIZE - OFFSET
   endif
   goto offset
   get CmpType byte
   
   goto nextOfs
   string NAME p "%d.cmp%d" i CmpType
   log NAME OFFSET SIZE

next i

---------------------------------------

Hi @ aluigi

Type 1 is a variant of LZSS
Type 2 is a variant of RLE.
I have not verified the other types.

I uploaded the source code, you can check it out.


I think re-compression is also feasible. Type2 is a variant of RLE encoding.
I think it should be possible to modify the compression source code to achieve compression.

aluigi, posted Sat Dec 11, 2021 6:21 pm (68184)

---

Thanks, very appreciated.

I already tried adding those functions decompiled from the exe to the quickbms beta and they worked perfectly, so the source code (and the fix for algo 2) is even better