This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=14162

Inuk Syooperstar, posted Sat Sep 26, 2020 7:54 pm (58888)

---

I came here earlier and posted my progress with this tool in the totally wrong board, and in there I said I was thinking of writing a compression algorithm... well, here it is.

To recap: I reversed the decompression function for Okage: Shadow King, and made a tool in Python for extracting the assets in the game's .XPF archives.
Now there's an options to pack the assets back into .XPF archives, because I wrote a compression function based on the decompressor.
It recompresses decompressed files just as well (tends to actually produce identical compression streams), and even better (even smaller files), than the original tool the developers used.

Attached is the tool written in Python. When fed an .XPF file, it will extract and decompress all contained files. When fed a folder, it will compress all files and pack them into an .XPF file.
Also included are C implementations of the same decompression and compression functions standalone, because I simply don't know to make an equivalent tool in this language - but I made these out of necessity because speed is an issue with Python; it may very well take hours to compress 1 file, while CPP takes mere seconds to do the same. See the recompression table under EDIT 2.

EDIT: Fixed a blunder that would confound anyone reading the compression algorithm. Also attempted to speed up the Python version a bit, and added a readme to help you use it.

EDIT 2: Added a compression levels option, which includes medium compression in a reasonable timeframe, and an option to completely match the compression in the original game files.
I selected this big file below to demonstrate the different levels. It's probably in the extreme end of time differences, but that also serves to clearly show what the levels are capable of.

Code:

Recompression table for DATA\BTLDATA\ALPHA\FIELD\BEILY41.XPF using OSK Tool.py
Level   Elapsed   Compressed size
0       0.02s     346,764 (No compression.)
1       9.6s      111,640 (Low compression; short writes, short seeks.)
2       9.6s      111,585 (Shallow compression; long writes, short seeks.)
3       128.14s   94,211  (Identical game compression; short writes, long seeks.)
4       118.21s   93,716  (Maximum compression; long writes and long seeks.)

EDIT 3: I just found out that you can completely bypass Python's slowness by running the script using PyPy instead of official Python - it matches the C implementations in speed, taking 1 second where it would otherwise take 1 minute... Wish I knew that earlier.

OSK Tool and code.zip

aluigi, posted Sat Oct 10, 2020 11:59 am (59207)

---

Well done and thanks for the code.
I will support this compression in quickbms 0.11, I just added it to the beta today and I will update the public beta next week.
In quickbms I will not handle the 32bit field at the beginning because I prefer to make the algorithm usable in the comtype_scan2 scanning.