This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=16030

masagrator, posted Mon Nov 01, 2021 10:27 am (67293)

---

It seems they updated VFS3 format without changing header, so script for Killer7 is returning reading error.



I have only figured out that something was added at 0x8 which is not taken into account in script.

0x0 - VFS3
0x4 - VER (isn't this (also) header size?)
0xC - folder count

And so on. Maybe there is already tool somewhere that can extract it? Best would be open source.

Archive
https://drive.google.com/file/d/1Ao_0bn ... sp=sharing

Edit: It seems to be similar to file from this thread:
https://zenhax.com/viewtopic.php?t=15413#p66367

masagrator, posted Mon Nov 01, 2021 5:48 pm (67303)

---

So after checking out whole file, I think I know now how this works, but I am lost how to decompress stuff.

Main issue is that VFS contains 10 compression dicts that needs to be used for decompression of files.
which dict to use is probably stored at the file table (there is one 16-bit value that has number between -1 and 9).
I am lost what to use to apply dicts to zstd compressed data without frame header.

Currently written script you can find here:
https://github.com/masagrator/NXGameScr ... npacker.py

spiritovod, posted Mon Nov 01, 2021 6:39 pm (67304)

---

@masagrator: If you mean quickbms, dictionary is supported like this "comtype zstd dict dict_size" where dict is binary data - which is implementation of default ZSTD_decompress_usingDict function from api, I think. You can check such implementation in Frostbite script, for example.

masagrator, posted Mon Nov 01, 2021 7:53 pm (67307)

---

spiritovod wrote:

@masagrator: If you mean quickbms, dictionary is supported like this "comtype zstd dict dict_size" where dict is binary data - which is implementation of default ZSTD_decompress_usingDict function from api, I think. You can check such implementation in Frostbite script, for example.

Thanks for tip. Sadly it doesn't work for me. (error -10 while uncompressing)

By looking at files i am assuming that if file has -1 in DEC_DICT then that means compression dict is not in use.
Even by looking at data it seems to be a correct stance.



But even that fails. I guess it may be related to that it doesn't have frame header (although dicts have them).

I have uploaded script that has hardcoded first dictionary and tries to unpack two files from archive in first post:
first has DEC_DICT = -1 (so I'm assuming that it doesn't use DICT at all)
second has DEC_DICT = 0 (thus why first dict)

EDIT: When adding manually header 28 b5 2f fd I'm getting error -72 instead of -10

Script.bms

spiritovod, posted Mon Nov 01, 2021 8:11 pm (67308)

---

@masagrator: I never seen usage of headerless zstd blocks - you can probably strip magic from it, but not the rest, otherwise you'll need to modify library like, a lot. And data from the picture looks more like lz family stream (most of them can use dictionaries as well). Did you try to look for some string references of used compression in binary?
Also, your archive in the first post is not available for public, in case if someone else will decide to look into the matter.

P.S. List of zstd error codes for better understanding: link

masagrator, posted Mon Nov 01, 2021 8:23 pm (67309)

---

spiritovod wrote:

@masagrator: I never seen usage of headerless zstd blocks - you can probably strip magic from it, but not the rest, otherwise you'll need to modify library like, a lot. And data from the picture looks more like lz family stream (most of them can use dictionaries as well). Did you try to look for some string references of used compression in binary?
Also, your archive in the first post is not available for public, in case if someone else will decide to look into the matter.

P.S. List of zstd error codes for better understanding: link

oops, fixed.
Still dictionaries have correct magic for zstd dictionaries. So at least files using them must be zstd.

Executable is also mentioning lz4 and zlib. It's not zlib obviuously.
Tried lz4 and also doesn't work.

I'm assuming all files are zstd because filename has "zstd"

masagrator, posted Tue Nov 02, 2021 10:15 pm (67344)

---

So for now figured out that files not using DICTs can use standard frame_header, but I need manually update it with compressed size in binary form.

Edit: All files not using dictionaries with only 1 frame decompressed successfully. It's using stanard zstd frame_header with --no-check.

masagrator, posted Thu Nov 04, 2021 5:12 pm (67366)

---

Figured out how to decompress files with chunks not using dictionary.
Now time to take dictionary attack.

masagrator, posted Sat Nov 06, 2021 8:32 am (67396)

---

Ok, I have now 100% success rate in extracting and decompressing files in all VFS3 archives for Ni no Kuni 2 Switch version.

Python script can be found here:
https://github.com/masagrator/NXGameScr ... npacker.py

It requires Python 3 and installed libraries: zstandard, numpy