This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=8545

Demonslayerx8, posted Sat Sep 15, 2018 4:33 pm (38418)

---

Would anyone be willing to make a script/tool to unpack the files from Black Clover Quartet Knights?
I'd upload the files, but they're pretty big and would take too long to upload, but looks like the DATH files are for file directories I think, but can be grabbed down below.

Quote:

XDathz, posted Sat Sep 15, 2018 5:36 pm (38422)

---

Hello!
I need some help to extract the files of this game.
Can anyone help me, please? Some script for this type of archive?

I need to found the texts, and i think it is on that folder.


Thank you!

aluigi, posted Sat Sep 15, 2018 6:46 pm (38425)

---

The format is very simple but there are no filenames stored so be ready to some headache browsing about 13000 nameless files (13000 only for the first dat, so double that number):
http://aluigi.org/bms/black_cover_gdath2.bms

akderebur, posted Sun Sep 16, 2018 12:44 am (38442)

---

Chrrox's bms script for another game seems to be relevant for this archive too : http://forum.xentax.com/viewtopic.php?p=104389#p104389 It is possible to at least set a file extension based on "FLDRHASH".

Also are you sure that the files are not compressed? I was looking at one of the model files : http://www.mediafire.com/file/46cqv2woi ... l.rar/file . I am not sure, but looking at the bone names around 0xB0, they seem like compressed. Like some variation of RLE maybe?

Edit : Indeed the files are compressed with lz4.

aluigi, posted Sun Sep 16, 2018 11:27 am (38458)

---

Yes it's lz4:

Code:

comtype lz4
goto 0x80
get DUMMY long
get SIZE long
get ZSIZE long
get DUMMY long
savepos OFFSET
get NAME basename
clog NAME OFFSET ZSIZE SIZE

akderebur, posted Sun Sep 16, 2018 1:54 pm (38462)

---

There is still something off though. The output is fine for small files, but with bit larger ones it seems inaccurate. Vertex/index buffer sizes don't match the size given in the header, and you can also see that indices are weird (compared to correct output on smaller files). Maybe there is padding (unnecessary bytes) that messes up the decompression?

I am not sure how can I find the source of the problem, but I will give it a try.

Edit : It wasn't unnecessary bytes, but chunk sizes. The file is separated into chunks. The smaller files with single chunk were working fine, but multiple chunks were failing. So the second "DUMMY" long in your script is the chunk size. After you read that many bytes there will be another long which is the size of the next chunk.

aluigi, posted Sun Sep 16, 2018 6:58 pm (38485)

---

Upload that new sample.

akderebur, posted Sun Sep 16, 2018 7:22 pm (38490)

---

Here, a file with 4 chunks.

aluigi, posted Sun Sep 16, 2018 8:19 pm (38493)

---

There is something weird about the chunks because apparently the chunks are ignored and the decompression must be applied on the whole data (basically it's like maintaining the "context" during the decompression of the chunks) instead of decompressing each chunk separately.
That's something that can't be supported by quickbms so I tried to collect all the chunks in a buffer and decompressing that buffer which is indeed what's expected, but test.mdl failed:

Code:

comtype lz4
goto 0x80
get CHUNK_SIZE long
get SIZE long
get ZSIZE long
get NAME basename

log MEMORY_FILE 0 0
append
for MEM_SIZE = 0 != ZSIZE
    get CHUNK_ZSIZE long
    savepos OFFSET
    log MEMORY_FILE OFFSET CHUNK_ZSIZE
    math OFFSET   CHUNK_ZSIZE
    goto OFFSET
next MEM_SIZE   CHUNK_ZSIZE
append
clog NAME 0 ZSIZE SIZE MEMORY_FILE

Just for reference, this is the test script in case the chunks were decompressed one-by-one as it usually happens (I leave it here just in case):

Code:

comtype lz4
goto 0x80
get CHUNK_SIZE long
get SIZE long
get ZSIZE long
get NAME basename
log NAME 0 0
append
for MEM_SIZE = 0 < SIZE
    get CHUNK_ZSIZE long
    savepos OFFSET
    clog NAME OFFSET CHUNK_ZSIZE CHUNK_SIZE
    math OFFSET   CHUNK_ZSIZE
    goto OFFSET
next MEM_SIZE   CHUNK_SIZE
append

chrrox, posted Sun Sep 16, 2018 9:55 pm (38513)

---

if i use the -e option and just read the whole file starting at 0x90 as the zsize it extracts the large files.

akderebur, posted Sun Sep 16, 2018 10:06 pm (38515)

---

The problem is that the last lz4 block before the new chunk isn't complete, the 2 byte offset is missing (offset for copying from the output buffer). In my own implementation the 2 byte offset was necessary even though the token is 0, so it caused problems. I haven't checked how you implemented lz4 in quickbms, but maybe quickbms also needs it?

I had to change my implementation, so that for the last block only the literals are copied from the block, copying from output buffer (with offset and token) is ignored. I am not sure how this can be achieved in quickbms. Closest I can get to correct output was with inserting 0 as 2 byte short (like dummy offset). It will run on test.mdl without error, but still the output is not exactly correct :

Code:

comtype lz4
goto 0x80
get CHUNK_SIZE long
get SIZE long
get ZSIZE long
get NAME basename
set DYN_SIZE ZSIZE

log MEMORY_FILE 0 0
append
for MEM_SIZE = 0 != ZSIZE
    get CHUNK_ZSIZE long
    savepos OFFSET
    log MEMORY_FILE OFFSET CHUNK_ZSIZE
    put 0 short MEMORY_FILE
    math DYN_SIZE   2
    math OFFSET   CHUNK_ZSIZE
    goto OFFSET
next MEM_SIZE   CHUNK_ZSIZE
append
clog NAME 0 DYN_SIZE 20000000 MEMORY_FILE

Anyway I got this working with custom lz4 code, but just wanted to help with the bms script. I am not sure if this can be done without changing the actual lz4 code though.

chrrox wrote:

if i use the -e option and just read the whole file starting at 0x90 as the zsize it extracts the large files.

Is the unpacked data correct? It might seem fine at the beginning of the file but I suspect it might become nonsense as you go further.

chrrox, posted Sun Sep 16, 2018 11:16 pm (38522)

---

yeah you are right data looks wrong.

aluigi, posted Sun Sep 16, 2018 11:55 pm (38531)

---

implementation of lz4 in quickbms is very simple:
size = LZ4_decompress_safe_partial(in, out, zsize, size, *outsize);

and this is an important comment about why I opted for safe_partial:

Quote:

// hard choice here:
// LZ4_decompress_safe returns errors if there are additional bytes after the compressed stream (because it's raw)
// LZ4_decompress_safe_partial returns no errors if the stream is valid
// currently I opt for the second one because gives more freedom to quickbms and its scanner

If interested in how the various algorithms are implemented in quickbms you must check the perform_compression function in perform.c

akderebur, posted Mon Sep 17, 2018 12:45 am (38535)

---

aluigi wrote:

If interested in how the various algorithms are implemented in quickbms you must check the perform_compression function in perform.c

I should really do that some time. The amount of algorithms that quickbms supports is insane I could definitely learn a few things.

I have attached a little program for decompressing the files. Drag and drop the file on the exe. Only tested it with mdl and tex files.

BC_Dec_U1.rar

Demonslayerx8, posted Mon Sep 17, 2018 9:58 am (38553)

---

I say, ur BC_Dec is failing to download for me, chrome, IE, Edge, and JDownloader just doesn't want to download it due to a virus being detected, should get that checked out.

akderebur, posted Mon Sep 17, 2018 10:06 am (38554)

---

Try this : http://www.mediafire.com/file/q3q19k5qj ... 1.rar/file

Demonslayerx8, posted Mon Sep 17, 2018 10:17 am (38556)

---

it still failed even with that, had to disable windows anti-virus to grab it, sadly it goes away after turning it back on.

edit: so what's the best way to unpack those DAT files, aluigi's script then the BC_dec?

akderebur, posted Mon Sep 17, 2018 10:24 am (38557)

---

Demonslayerx8 wrote:

it still failed even with that, had to disable windows anti-virus to grab it

Weird, seems like false positive for some anti-virus programs.

Demonslayerx8 wrote:

edit: so what's the best way to unpack those DAT files, aluigi's script then the BC_dec?

Exactly.

Demonslayerx8, posted Mon Sep 17, 2018 10:34 am (38558)

---

akderebur wrote:

Demonslayerx8 wrote:

edit: so what's the best way to unpack those DAT files, aluigi's script then the BC_dec?

Exactly.

gotcha.. anyway to make it do it to the whole folder instead of 1 by 1? Would be easier for me lol

akderebur, posted Mon Sep 17, 2018 10:40 am (38559)

---

Yea, I can do that, will update it when I have the time.

Demonslayerx8, posted Mon Sep 17, 2018 10:43 am (38560)

---

Alright cool~

edit: damn, trying to figure out these textures are difficult
edit2: welp, figured out the texture data, pretty much little endian, sadly haven't found anything else that's not BC1 or BC7

Quote:

0x00 - width
0x02 - height
0x04 - unk (seems to be the same in every texture data I've came across)
0x06 - texture type
~002d = BC1
~402d = BC7
0x08 - padding
0x0C - texture data

akderebur, posted Tue Sep 18, 2018 1:04 am (38575)

---

Demonslayerx8 wrote:

Alright cool~

I have updated the tool with folder support. Drag and drop folder or file on the exe. Download from my previous post.