This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=7149

valkyrie-ace, posted Fri Dec 29, 2017 7:06 am (31100)

---

STAR OCEAN anamnesis is Mobile Game from Tri-ace.
I try file extract, but I can't
Files extensions looks like "Heaven x Inferno", however, header's not "SLZ", there's "ADLD"

I attached some sample files.
have any idea and can you help me?

aluigi, posted Mon Jan 01, 2018 3:29 pm (31185)

---

It uses a xor obfuscation with a 32bit number (converted to a 8 bytes hex string), I don't know how the key is generated but we can easily guess it
I have updated the script to automatically doing the job but remember that SLZ type 7 used in 2 of your samples is not supported yet.
Script 0.2.4:
http://aluigi.org/bms/slz.bms

valkyrie-ace, posted Tue Jan 02, 2018 12:28 am (31201)

---

great work aluigi.
I have question, FIA and FIS header files, can open any softwares?
They are still unique algorithm compression files?
and will you support type 7?

aluigi, posted Tue Jan 02, 2018 10:32 am (31208)

---

I don't know about FIA/FIS.
I don't have the code for the type 7 compression, once someone will provide it I will implement it in quickbms.

akderebur, posted Tue Aug 07, 2018 9:18 pm (37291)

---

The compression method for type 7 seems to be zstd. I have disassembled the game library to check the type 7 code, and there are calls to methods like "DecompressZSTD". Here is the pseudocode : https://pastebin.com/J39hFdr2

. I hope you can take a look at this sometime. In the bms script I have tried setting the comtype to ZSTD/ZSTD_COMPRESS, but it didn't work. New algorithm?

Edit : The zstd version that is used is 1.3.4.

aluigi, posted Wed Aug 08, 2018 2:03 am (37295)

---

zstd streams have a recognizable 4 bytes magic so it's easy to guess.
"comtype zstd" automatically supports all the existent zstd versions in case you want to try.
If you don't see the zstd magic it probably means it's encrypted

akderebur, posted Wed Aug 08, 2018 4:05 am (37307)

---

aluigi wrote:

zstd streams have a recognizable 4 bytes magic so it's easy to guess.

Yea, you were right. There are many occurrences of the magic, so the file is divided into multiple chunks with zstd streams. Before each magic there is also a 2 byte offset to the next chunk.

I don't know how to handle this in bms. I ended up finding a zstd library and writing a program to read and decompress the chunks. The output is correct, so I can confirm that type 7 is indeed zstd. Thanks for the help.

aluigi, posted Wed Aug 08, 2018 12:12 pm (37315)

---

It looks like a customized zstd, in fact the magic is not 0xFD2FB528 but 0xFD0FB528.
Even fixing the byte and using comtype zstd with type 7 will not work.
So first you have to solve this mistery

akderebur, posted Wed Aug 08, 2018 12:23 pm (37320)

---

aluigi wrote:

So first you have to solve this mistery

Already did

The problem is with the slz script actually. It doesn't fully decrypt the "ADLD" files. You have to xor the every 5th byte of 8 byte iterations with 0x20 to have the correct output. I noticed this while working on the textures, you can see the full discussion here : http://forum.xentax.com/viewtopic.php?p=142494#p142494

After doing the xor stuff, the file is decrypted properly and the zstd magics will be correct too. It is standard zstd.

aluigi, posted Wed Aug 08, 2018 12:31 pm (37325)

---

That's exactly what I tried before posting using one of the provided files for forcing the magic to match but it still gave errors and the obtained key had an invalid char at byte 5 (a '4' become 0x14)

akderebur, posted Wed Aug 08, 2018 12:41 pm (37326)

---

Weird. Here is how I do it. I have attached 2 bms scripts. First I run the "slz.bms" on the "ADLD" file to get the slz output. Then I use the "slzxor.bms" on the slz file. Maybe you can pack them into a single script.

Anyway at that point you should have the correct output. Still like I said I couldn't get it to decompress with quickbms, but I thought it is because of my lack of knowledge. I managed to get it decompress with a zstd library, so I can guarantee that the slz scripts produce the correct output.

slzscripts.rar

aluigi, posted Thu Aug 09, 2018 1:28 am (37340)

---

Eh no, as already said it still doesn't work:

Code:

Info:  algorithm   478
       offset      00000022
       input size  0x0000302e 12334
       output size 0x00004440 17472
       result      0xffffffb8 -72

zstd works perfectly as tested with many other zstd compressed streams so if quickbms returns an error I'm sure there is something wrong with the data. Additionally the key has no sense with that invalid byte at position 5.
Keep me update.

akderebur, posted Thu Aug 09, 2018 3:08 am (37342)

---

I think it will be easier if we both look at the same file. I am working with models atm, so I have just tested this with asf model files. Here is the model file of one of the characters (Myuria) that uses type 7 : http://www.mediafire.com/file/4jhficeh9 ... p.rar/file

I have included various states of the file numbered (1) to (4) :
1 - Original file with ADLD header
2 - Output of "slz.bms", not xored yet
3 - Output of "slzxor.bms", just xored
4 - Output after zstd decompression

I thought maybe this way you might be able to see in which step there is a problem.

Can you also post the file that you tried to decompress, and the script?

aluigi, posted Thu Aug 09, 2018 4:05 am (37343)

---

ZSTD_decompress_usingDict (without dictionary obviously, so it gets ignored) returns -72 so there is something invalid in that stream.
quickbms uses zstd 1.3.4
I can get the data only if I ignore the error with the -e option of quickbms.
Script used:

Code:

comtype zstd ""
goto 0x20
get ZSIZE short
savepos OFFSET
clog "test.dat" OFFSET ZSIZE 0x10000

What zstd function are you using for decompression and what's its return value on the first chunk?

akderebur, posted Thu Aug 09, 2018 4:25 am (37345)

---

I am using this library : https://github.com/bp74/Zstandard.Net, which is basically a wrapper for the native zstd library. Instead of invoking direct decompression methods, it uses stream related methods like "ZSTD_createDStream" and "ZSTD_decompressStream". I don't know much about zstd, but do you think that might be the main difference?

aluigi, posted Thu Aug 09, 2018 5:44 am (37346)

---

I tried to write a simple tool from scratch that uses ZSTD_decompress on the 0x2b28 bytes of the sample and the result is the same: -72.
I got the same result even with the other one-chunk samples so it's excluded anything related to streams, chunks, quickbms, api and so on.
-72 means ZSTD_error_srcSize_wrong

So what I did was very simple, tried to subtract 1 from the chunk size in a loop till I got a valid return value and it worked:
chunk_size - 1

Probably that .NET wrappers ignores errors or you didn't check them or the state of the decompression in your program, that's why you didn't see it.

Now the mistery is finally solved... except for the meaning of that 0x00 byte at the end of each chunk that means nothing.

Personal opinion: why this *** zstd library returns an error if the input size is bigger than the real compressed size? mah senseless...

akderebur, posted Thu Aug 09, 2018 8:04 am (37347)

---

aluigi wrote:

Probably that .NET wrappers ignores errors

No, it actually checks for error codes and throws an exception if there is one.

aluigi wrote:

except for the meaning of that 0x00 byte at the end of each chunk that means nothing.

Not every chunk ends with 0x00 though. Check out the chunk starting at 0x4AD2 (last byte at 9CFD), it ends with 0x01.

For the chunks that end with 0x00 I can also use chunkSize - 1, but otherwise the data won't get decompressed properly or not at all. I am not sure if chunkSize - 1 is the right way to go, but if it works out that way in quickbms then it is fine I guess.

aluigi, posted Thu Aug 09, 2018 8:27 am (37348)

---

In fact for those chunks that have the last byte different than 0 zstd wants the whole chunk size.
Long story short: zstd is acting in a very confusing way (*edited*).
The .NET wrapper uses ZSTD_decompressStream instead of ZSTD_decompress that probably is not affected by the "bug" of the size.

Unfortunately there is no way to bypass that error returned by ZSTD_decompress because ZSTD_error_srcSize_wrong is returned in any case the compressed size is not the exact one it expects, so both if the provided data is shorter (correct behaviour) and longer (wrong behaviour that leads to this bug).

I will evaluate if using ZSTD_decompressStream may be a good solution for the future but I don't like to waste time in this way.
More probably I will just ignore the -72 error and return the full decompressed size which is wrong too.

I'm curious to know how the game developers obtained that compressed data with the additional 0x00 byte at the end.

akderebur, posted Thu Aug 09, 2018 8:58 am (37349)

---

Yea, zstd seems a bit weird. Still it seems to perform really good, or at least that is what it says on its site

I feel like it might get more popular. For example with SOA most of the older character asf files (like a year ago) were using slz compression types from the earlier games, but now almost all character asfs are compressed with type 7/zstd. It might indeed be a good idea to implement ZSTD_decompressStream for the long run.

aluigi, posted Thu Aug 09, 2018 9:12 am (37350)

---

yeah zstd is gaining popularity in games.

The problem is that ZSTD_decompressStream and ZSTD_decompress are not exactly the "same".
ZSTD_decompress is a wrapper for the multiframe function and internally their 2 low-level functions ZSTD_decompressStream->ZSTD_decompressContinue and ZSTD_decompress->ZSTD_decompressFrame have no other low-level function in common.
Basically the code is redundant as clearly visible from the 2 occurrencies of "bt_compressed"... very bad idea.

chrrox, posted Fri Aug 10, 2018 11:59 pm (37406)

---

Quickbms seems to work fine with the -e option on files I tested.
(including 0 ending)
I attached 2 samples I decrypted on my own (have correct key).

Here is crude decompress script I used.

Code:

comtype zstd
get NAME filename
string NAME  = .dec
getdstring SIGN 3
get TYPE byte
get VER long
get TZSIZE long
get TSIZE long
goto 0x20
set CHUNKS TSIZE
math CHUNKS  = 0xFFFF
set REMAIN CHUNKS
math REMAIN % 0x10000
print "%REMAIN%"
math CHUNKS - REMAIN
math CHUNKS / 0x10000

    if TYPE == 0
        comtype copy
        math SIZE = ZSIZE
    elif TYPE == 1
        comtype slz_01
    elif TYPE == 2
        comtype slz_02
    elif TYPE == 3
        comtype slz_03
    elif TYPE == 4
        comtype xmemdecompress
    elif TYPE == 5
        comtype unzip_dynamic   # it's both zlib_noerror and deflate_noerror depending by the game!
    elif TYPE == 7
        comtype zstd
    else
        print "Error: unsupported type %TYPE%, contact me"
        cleanexit
    endif
append
for i = 0 < CHUNKS
get zsize short
savepos OFFSET
clog MEMORY_FILE OFFSET zsize 0x10000
math OFFSET   zsize
goto OFFSET
next i
append
log NAME 0 TSIZE MEMORY_FILE

akderebur, posted Sat Aug 11, 2018 12:31 am (37407)

---

Yep, it is just that quickbms returns the -72 error code. Otherwise the output is same as the one decompressed with "ZSTD_decompressStream".

aluigi, posted Sat Aug 11, 2018 12:48 am (37408)

---

As far as I remember even with the -e option the decompressed data of the sample in the first post FootSE_cm408.spk didn't work, but maybe it was a mistake on my side due to all the tests I was doing in that moment

Ah the final decision I have taken about the next version of quickbms is editing the zstd code for removing the 2 occurrencies of "if (srcSize) return ...", so I will be able to use the ZSTD_decompress function (which has multiframe support so it's bad to not use it) and it will automatically work with any stream

Rem, posted Sat Aug 11, 2018 10:22 pm (37434)

---

I unfortunately cannot help with this but I thought I would add another ADLD encrypted type that is not able to be decrypted, even with the updated method described by akderebur or with chrrox' script

I've confirmed that most files can be decrypted using the method described here, except some files such as the one attached to this post.