[PSP] Gundam Battle / Macross Ace Frontier compressed files

[ScornMandark]

Hi all!  I'm running into an interesting problem while perusing some old PSP games.  Gundam Battle, Macross Ace Frontier, Macross Ultimate Frontier, and Macross Triangle Frontier were all made by Artdink between 2008 and 2011.  They mostly use the same engine, and basically the same structure.  There's a quickBMS script for unpacking the main .dat files (macross_idx), unpacking most of the models and textures into .gmo and .gim files that Noesis can easily read.  Both games use 2 main .dat files though - a gundam.dat/macross.dat and a fileset.dat.  The files in the gundam/macross.dat are stored relatively uncompressed, and extract very easily.  The files in the fileset.dat are apparently stored encrypted and compressed, and the main quickBMS script assigns file names and dumps them with no file type.  There is a second quickBMS macross_fsts script that takes those files, scrapes them and assigns file extensions, but they are still compressed and encrypted.

I've gone through and I'm pretty sure they're char XOR'd with 0x72, but also compressed.  I don't really know the first thing about actual encryption or compression methods, but I've attached a file with a few .gim examples.  ef_me01.gim is one of the compressed/encrypted files, ef_me01_XOR.gim has been char XOR'd with 0x72.  2d_ope_42.gim is an uncompressed .gim from the same game, with what I think is the same overall file length?  (I included the .bms scripts if you're interested.)    (  is the .png conversion of that .gim in Noesis.)

https://www.psdevwiki.com/ps3/Graphic_Image_Map_(GIM) is the spec I'm using for sorting things out - the .gmo files are probably compressed in a similar fashion, but the .gim spec came up first on my internet search 😛 

Once I performed the char-wise XOR of the whole ef_me01.gim file, some interesting things stood out to me.  All .gim files (little endian) start with MIG.00.1PSP, but there was a new section at the front for the ef_me01_XOR.gim: RAIC.qrrÿMIG.00.1×PSP 

Most of the encrypted/compressed files have some variation of RAIC.qrr - some have bits flipped, but I don't know what that actually means.  r is also the ASCII representation of 0x72, so maybe just RAIC.q?  q.CIAR in big endian? 52 41 49 43 2E 71 in little endian?

Then there's an 0xFF.

Next is the first 8 bytes of the .gim header.  It's interrupted by a 0xD7, followed by the last 3 bytes (PSP).  That's followed by 0xE9 F2, then the 02 that's the next line in the .gim spec (usually 5 bytes 00 padding).  4D 49 47 2E 30 30 2E 31 D7 50 53 50 E9 F2 02

after 02 there's an EB, which I think is a 3 byte 00 buffer indicator?  so something like FF is 8 bytes of data, E9 is 5 bytes of buffer, EB is 3 bytes of buffer.

After EB, there's an F0, followed by 4C 03 C7 00 00 10 EB.  That's along the right lines of the .gim root block.  On 2d_ope_42.gim, that looks like:

0x 02 00 00 00 4C 03 00 00 10 00 00 00 10 00 00 00    

On most of the .gim files, the block_size might change, but the block_header_next and block_data_offset don't - it's a one line data block followed on the next line by the next block.  Then it gets into the picture and image blocks, and I start losing the thread 😕 

I can usually find the Fileinfo block at the end of the file pretty easily - it starts with the file name, and is mostly interrupted by the inserted FF marker every 8 bytes, and contains the original graphic file name, the user, the date, and the version of GimConv they used to convert the graphic file to .gim in the first place.

 I don't even really know what kind of compression was used, it seems like most of the headers were ripped out. The PSP and PS3 used some mix of ZLIB and RLZ, according to the rcoxml model entry on the ps dev wiki, but I don't know what I'm even looking at.

Am I looking at this the wrong way?  Should I be looking at the compression before the decryption?  help us, oh wise gurus, you're our only hope 😄 

ms.zip

[ScornMandark]

I've tried the comtype_scan2 from aluigi, on both the xor and raw data, and none of the output dumps appear to line up with anything specific.  I'm just not sure what to do next, besides reverse engineer a custom solution.  I don't have a before and after, unfortunately, so it's going to be a lot of guessing I suppose.

[DKDave]

2 hours ago, ScornMandark said:

I've tried the comtype_scan2 from aluigi, on both the xor and raw data, and none of the output dumps appear to line up with anything specific.  I'm just not sure what to do next, besides reverse engineer a custom solution.  I don't have a before and after, unfortunately, so it's going to be a lot of guessing I suppose.

It looks like it just uses LZSS compression on the decrypted file from offset 8 - the "RAIC.qrr" is just some header, no idea what it means.

 

 

[ScornMandark]

Oh phenomenal!  That would explain why the comtype scan wasn't working, since the actual weird header was throwing it off.

It very nearly worked flawlessly - the comtype scan lzss dump filled all the 0x00 padding spots with 0x20 for some reason, but it might just need a different decompression tool.  Definitely some kind of image there, though.  Thanks very much!

[DKDave]

Yeah, should be LZSS0 for the compression and that should work now.

 

 

[ScornMandark]

Well, there ya go!  Thanks very much, I appreciate your help!