This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=503

aluigi, posted Tue Jan 06, 2015 12:32 pm (2480)

---

Feel free to link here any compression algorithm that is not currently available in quickbms.

Doesn't matter if you have the source code or only the binary dump of the function, everything is welcome

Ekey, posted Tue Jan 06, 2015 9:32 pm (2490)

---

Ok! Here one - for Zen Studios games. Masm32

ZenLib-src.rar

RetroHelix, posted Thu Jan 08, 2015 7:13 pm (2578)

---

Would be nice if these algorithms could be added: http://www.romhacking.net/utilities/826/ The download comes with source files.

aluigi, posted Sun Jan 11, 2015 10:29 am (2647)

---

@ RetroHelix
I think that some of them are already implemented in quickbms, but without samples for each algorithm I'm sure that for me is more simple to add all of them directly to the next quickbms rather than checking each one

RetroHelix, posted Sun Jan 11, 2015 4:06 pm (2656)

---

aluigi wrote:

@ RetroHelix
I think that some of them are already implemented in quickbms, but without samples for each algorithm I'm sure that for me is more simple to add all of them directly to the next quickbms rather than checking each one

I encoded the history.txt that came with the tool to make some sample files in case you want to test your implementation in quickbms.

aluigi, posted Wed Jan 14, 2015 7:23 am (2670)

---

Thanks a lot, I have checked it and the following are already implemented:

Code:

LZSS history (LZ10).txt = LZ77WII and NITROSDK
LZX-evb history (LZ11).txt = LZ77WII
RLE history.txt = LZ77WII and NITROSDK

So those to implement are:

Code:

BLZ history.txt
LZE history.txt
LZX-ewl history (LZ40).txt

RetroHelix, posted Thu Jan 15, 2015 4:40 pm (2693)

---

aluigi wrote:

Thanks a lot, I have checked it and the following are already implemented:

Code:

LZSS history (LZ10).txt = LZ77WII and NITROSDK
LZX-evb history (LZ11).txt = LZ77WII
RLE history.txt = LZ77WII and NITROSDK

So those to implement are:

Code:

BLZ history.txt
LZE history.txt
LZX-ewl history (LZ40).txt

Very nice! Thank you very much.

puggsoy, posted Mon Feb 02, 2015 8:08 am (3028)

---

Can you implement LZEXE? The code for both compression and decompression can be found here: http://bellard.org/lzexe.html

aluigi, posted Mon Feb 02, 2015 9:32 am (3031)

---

If you mean the algorithm with the magic "FAB", yes it will be implemented in the next version.
It's used in scummvm so it's very easy for me to implement it.
I don't know if the one in lzexe is the same but I will implement also the code of unlzexe5.c

Ekey, posted Sat Feb 07, 2015 2:48 pm (3093)

---

STALKER compression open source'd (code) from OpenXRay Engine - https://github.com/OpenXRay/xray-16/blo ... /LzHuf.cpp

aluigi, posted Sun Feb 08, 2015 11:18 am (3104)

---

Great, added to the TODO.

tsanford01, posted Thu May 28, 2015 2:24 pm (5479)

---

Hello again, Please add the updated LZ4 r129 as we discussed in a previous post. Just wanted to tack it here for the official to do list.

Thank you again,
Tom

aluigi, posted Fri May 29, 2015 2:23 pm (5487)

---

I just released the new quickbms.

eri619, posted Sun Nov 29, 2015 11:19 pm (10205)

---

Could you add the compression algorithm for yukes.bms(BPE/Byte Pair Encoding) please?

yuke.rar

aluigi, posted Mon Nov 30, 2015 8:53 am (10211)

---

If you have the source code of the recompression algorithm yes, otherwise no.

eri619, posted Mon Nov 30, 2015 5:37 pm (10220)

---

please take a look at this
viewtopic.php?f=13&t=1882
i will soon post the BPE Code.

aluigi, posted Wed Dec 02, 2015 9:26 am (10244)

---

That error is caused by the missing algorithm.

ThatTrueStruggle, posted Tue Dec 15, 2015 8:48 pm (10465)

---

Could you add a new decompression algorithm? This one is a variant of LZSS, from Donkey Kong: Tropical Freeze. http://www.metroid2002.com/retromodding ... ompression
Basically all this an updated algorithm but it adds some extra features, including different modes.

I also have an example, if needed: https://www.dropbox.com/s/qem0thhn4q7vm ... e.bin?dl=0
Compressed Size: 0x13E6
Uncompressed Size: 0x145E

Thank you so much!

aluigi, posted Mon Dec 28, 2015 4:56 pm (10581)

---

Added in my TODO list

Ekey, posted Wed Jan 27, 2016 10:36 pm (11070)

---

https://github.com/thatfool/SCAN/blob/master/CLZF2.cs

aluigi, posted Wed Jan 27, 2016 11:57 pm (11073)

---

quickbms already has lzf implemented and it seems the same algorithm, have you checked it?
There is also fastlzah that should be similar.

Racial, posted Fri May 06, 2016 9:13 pm (13193)

---

I found this python codefor BPE
Can be used for something? I will try to translate to other language

Code:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

# checks whether a character is usable for a definition
def char_usable(char, text):
   good = [9, 10, 13]; # "safe" characters with ordinals < 32
   if (char <= 255 and chr(char) in text) or (char < 32 and char not in good):
      return False # not usable, try another one
   else: return True # usable
   
# Byte-pair compression
# Works by replacing common pairs of letters with a single letter (a "definition") and placing that definition in a dictionary
def compress(text):
   dictionary = [] # dictionary of pair definitions - very ironic
   dict_char = 9
   while not char_usable(dict_char, text): dict_char  = 1 # find dictionary-delimiting character
   if dict_char > 255:
      print "Document uses all ASCII characters and is uncompressable."
      exit(2)

   while True: # infinite loop
      character = 9
      while not char_usable(character, text) or character == dict_char: character  = 1 # find character to define
      p = 0 # initialize pair pointer
      dict = {} # temp dictionary of pair frequencies
      while p < len(text): # loop through all possible pairs
         x = text[p:p 2]
         dict[x] = dict[x]   1 if dict.get(x, False) else 1 # increment pair's frequency counter
         p  = 1
      large = 0
      large_pair = ""
      for pair,amount in dict.items(): # check through dictionary for most frequent pair
         if amount > large:
            large_pair = pair
            large = amount # use it
      if large == 1 or character > 255: break # if there are no common pairs or we are out of characters to define, exit the loop
      text = text.replace(large_pair, chr(character)) # perform the replacement with the definition
      dictionary.append([large_pair, chr(character)]) # add the definition to the dictionary
   dict_str = chr(dict_char) # get the dictionary-delimiting character
   for item in dictionary: # convert the dictionary into a string
      dict_str  = item[0]   item[1]
   return chr(dict_char)   text   dict_str # construct the final compressed string with all information necessary

# Byte-pair decompression
# reconstructs the dictionary, then applies it
def decompress(text):
   dictionary = [] # init dictionary
   dict_char = text[0] # find dictionary-delimiting character
   text = text[1:] # remove dict-delim char from text
   dict_start = text.index(dict_char) 1 # remove dicionary character from text
   dict_str = text[dict_start:] # separate out dictionary and text 
   text = text[:dict_start-1] # remove dictionary from text
   i = 0
   while i < len(dict_str): # loop and parse dictionary
      dictionary.append([dict_str[i:i 2], dict_str[i 2]])
      i  = 3
   dictionary.reverse() # reverse it so items are read in right order
   for item in dictionary:
      text = text.replace(item[1], item[0]) # perform replacements
   return text

# Checks whether a file has been compressed using this script
def is_compressed(text):
   # first character in document (dictionary character) only appears elsewhere once: probably compressed using this algorithm 
   return True if text[1:].count(text[0]) == 1 else False  

choice = raw_input("[c]ompress or [d]ecompress: ")
input = raw_input("Enter file to process: ")

# Opens the file
try:
   file = open(input, "rb")
   text = file.read()
except IOError:
   print "Can't read input file."
   exit(2)

# Runs selected option
if choice == "c":
   result = compress(text)
   print "----------"
   print "Original length:\t"   str(len(text))   " bytes"
   print "Compressed length:\t"   str(len(result))   " bytes ("   str(round((float(len(result)) / float(len(text))) * 100, 2))   "% of original size)"
   print "----------"
elif is_compressed(text): result = decompress(text)
else:
   print "Input file not compressed!"
   exit()
   
new_file = raw_input("Enter output file (or leave blank to display): ")

# Writes to output file
if len(new_file):
   new = open(new_file, "wb")
   new.write(result)
   new.close()
else: print result

aluigi, posted Sat May 07, 2016 9:14 am (13196)

---

Do you know that quickbms already implements various BPE compression algorithms?
Is this one different than those already available?

TheUkrainianBard, posted Sat Jun 11, 2016 9:18 pm (14313)

---

Is there a PS3/PSVita LZ77 implementation?

Sorry, I'm not fluent in C, so here are some QuickBMS implementations and game files.

Be careful, "debug" scripts give an enormous logfile with what got copied and where.

ps3_lz77.zip