aluigi, posted Sun Nov 28, 2021 11:19 am (67732)

---

Ok the problem with -F was that the pak archive was loaded as list of files (first tentative as file, second as string) and there was a "for(i = 0; ret[i]; i )" invoked on a NULL ret.

I can add a check on NUL bytes in the filter in case the -F FILE is not a proper text file.

I found an apk for that DragonRaja game but I get no errors on its PNG archives.

aluigi, posted Sun Nov 28, 2021 12:03 pm (67734)

---

@z4ruz
Regarding "2) Print array elements as hex issues" where you get "1" and 31 instead of 1 and 01.
The reason is that the variables arrays (subvar or whatever we call them) only store a string representation of the variable.

@johnz1
The reason why in 0.10.1 reimport.bat worked while it fails on 0.11 is that I slightly changed the behavior of the zopfli compression.
In short in 0.10.1 it was a more extreme and slower implementation.
Anyway I'm going to "fix" it by reenabling the extreme mode with reimport1

@spiritovod
Ah, remember to use ever {} instead of * in the filters because Windows is going to interpret that char before quickbms and doing a mess

aluigi, posted Sun Nov 28, 2021 1:04 pm (67737)

---

@spiritovod
The access violation you noticed is probably the same bug reported in ttgames few posts above.
I can't replicate these issues but I guess it's related to the impossibility of allocating the memory, even if I don't get why 0.10 worked.
Did you use the -9 option maybe?
We need a simple way to replicate the issue.

aluigi, posted Sun Nov 28, 2021 6:12 pm (67744)

---

@z4ruz
Regarding "string var2 r var" use "string var2 0r var" and it will correctly handle the NUL byte

aluigi, posted Mon Nov 29, 2021 2:14 pm (67755)

---

The following is a beta that should fix most of the reported issues:
http://aluigi.org/beta/quickbms_exe.zip

(If interested in the source code, the diff from 0.11 is on http://aluigi.org/beta/quickbms.diff, plus I updated some compressions like lzma, zstd and blosc)

gamelandresearch, posted Tue Nov 30, 2021 8:36 pm (67772)

---

i don't know what to say, words not enough...
i can't decrypt .pak still i got error after updating quickbms to latest
that's problem decrypting .pak:

-------------------
*EXCEPTION HANDLER*
-------------------
An error or crash occurred:

*EH* ExceptionCode c0000005 access violation
*EH* ExceptionFlags 00000000
*EH* ExceptionAddress 0AA15B90
*EH* NumberParameters 00000002
*EH* 00000000
*EH* 381A0416

Last script line before the error or that produced the error:
46 calldll MEMORY_FILE10 decryptPackage tcc RET MEMORY_FILE SIZE
coverage file 0 100% 2201972 2201972 . offset 00000000
coverage file -1 0% 0 2201972 . offset 00000000
coverage file -10 0% 0 1018 . offset 00000000

i used script: https://zenhax.com/viewtopic.php?f=11&t=15965

aluigi, posted Wed Dec 01, 2021 1:27 pm (67777)

---

@gamelandresearch
You are talking about a different topic.
I can't help with that because that function (that comes from an action script here https://github.com/LolHacksRule/AngryBi ... lLoader.as) doesn't have any sense to me (mRandom declared as 32bit signed integer but with unsigned right shift of 35 and division of 31 bits at the end).
The crash is caused by the second cycle, even if you limit it the result will still be garbage.

spiritovod, posted Mon Dec 13, 2021 8:07 pm (68224)

---

aluigi, sorry for the late reply, I was quite busy recently. I've checked all 5 games with azure package format (the one which is used in Dragon Raja as well) and couldn't reproduce that memory allocation error, maybe it was an unfortunate coincidence for some particular version of one of them.

And I'm aware about {} usage, but it was more about difference between 0.10.1 and 0.11 behavior - if the same parameters didn't work for both of them, it would be totally understandable, but in 0.10.1 it works just fine. Maybe you need to adjust documentation on this matter, because currently * is used as wildcard in most samples in documentation.

aluigi, posted Mon Dec 13, 2021 9:53 pm (68230)

---

Do you mean the bug with -F ?
As far as I know I fixed it in the beta, the * got expanded by windows on both 0.10.1 and 0.11 but on the latter it had hit a new bug.
Definitely going to replace * with {} in most of the examples.

spiritovod, posted Mon Dec 13, 2021 10:30 pm (68232)

---

Yes, I mean -F usage in batch scripts. Also, sample from this post now produces "the input folder is empty" error with latest beta (as said above, it works fine with either 0.10.1 or {} as wildcard in the script).

tbmq008, posted Tue Dec 14, 2021 4:58 pm (68241)

---

i have a Windows 10 PC with 6GB of RAM and it uses more than half of that and i cannot decompress a big 1.53GB file.
memory errors and all that.

Code:

E:\quickbms>quickbms "E:\[REDACTED]\xcompress_file.bms" "E:\[REDACTED]\Gen_Common.lin.bf" "E:\[REDACTED]\Gen_Common.lin.bf"

QuickBMS generic files extractor and reimporter 0.11.0
by Luigi Auriemma
e-mail: [email protected]
web:    aluigi.org
        (Apr  5 2021 - 13:56:34)

                          quickbms.com  Homepage
                            zenhax.com  ZenHAX Forum
                     @zenhax @quickbms  Twitter & Scripts

- open input file E:\Microsoft - Xbox 360\MotionSports [RF][Kinect]_2\Data\CBin\Gen_Common.lin.bf
- open script E:\Microsoft - Xbox 360\xcompress_file.bms
- set output folder E:\Microsoft - Xbox 360\MotionSports [RF][Kinect]_3\Data\CBin\Gen_Common.lin.bf
- the folder doesn't exist, do you want to create it (y/N)?:
  y

  offset   filesize   filename
--------------------------------------
  00000000 1649442550 Gen_Common.lin_unpack.bf

- error in src\extra\xalloc.c line 618: xdbg_malloc()

Error: memory allocation problem
       No h recursos de memria disponveis suficientes para processar este comando.


press ENTER to quit

and yes this does kinda affect quickbms in a way since i had no idea xmemlzx "allocates" bigger sizes than necessary given how LZX compression algorithm actually works.
i'll be uploading a small sampler of that file upon request.

aluigi, posted Tue Dec 14, 2021 7:23 pm (68246)

---

@tbmq008
Unfortunately I have no solution for that problem at the moment.
Quickbms is compiled as 32bit application for compatibility reasons (in short it must work on any Windows version on any hardware) and therefore the memory allocation is limited to 2 or 3 Gb.
Since all the compression algorithms are buffer-to-buffer, it must allocate memory for the source and for the destination, so if the input is 1.5 Gb it probably needs to allocated 3.5 or more Gb.

No plans for releasing 64bit versions. Maybe I can think about doing that for quickbms_4gb_files.exe but I would like to get more feedback about this possible choice.

tbmq008, posted Tue Dec 14, 2021 7:53 pm (68247)

---

same error with quickbms_4gb_files, unfortunately. but whatever.

is "dynamic memory allocation" an OK thing to suggest? this would be particularly useful for decompression algorithms like xmemzlx given what i found.

is there any way to simply work around this issue? like having xmemlzx decompress raw compressed data only instead of reading everything.

i had to pull off massive hacks into one of my scripts which covered a file format that used this compression algorithm by having the script feed fake header data plus the compressed chunk data into memory so it can be decompressed into memory.

aluigi, posted Wed Dec 15, 2021 8:32 am (68249)

---

@spiritovod
The reason of that error is that quickbms accepts a text file or a string with the -F option, because somtimes it may be useful to have a list of files in a text file rather than having a long command-line option.
My mistake is that probably I should limit the file input to the .txt extension (so -F list.txt is ok while -F list.dat is considered a string).
Let me know if that may be better and I will add it.
*edit* anyway there is something that I have to check in the beta because there was currently a NUL-byte check on the filter file that seems to fail
*edit2* yeah it's a bug, long story short the input files have the local folder in them so I have to find a simple solution: .\EnderLilies-WindowsNoEditor.pak

@tbmq008
quickbms_4gb_files is 32bit too, what I mentioned is a "possible" idea for the future.
Anyway I released a new beta of quickbms with a possible solution and a work-around.
The solution was already available in quickbms but only activated when using the -9 option, while now in the beta it's ever working. It consists in freeing the current buffer if reallocation fails (because realloc takes double memory).
The work-around instead is allocating only one buffer containing both input and output, obviously this is WRONG but it may work often since the input buffer is located at the end of the output one and the decompression should advance both input and output without overlapping.
A warning message is displayed when the work-around is activated.
beta exe:
http://aluigi.org/beta/quickbms_exe.zip

(source code diff is ever at http://aluigi.org/beta/quickbms.diff)

aluigi, posted Wed Dec 15, 2021 10:42 am (68252)

---

@spiritovod
ok I just updated the beta and now it's all fixed and I even added support for input list encoded with UTF16 (UTF8 BOM, UCS2 BE/LE BOM)

tbmq008, posted Wed Dec 15, 2021 2:10 pm (68259)

---

alright, i'll try using quickbms 0.11.0 with the -9 option first then the beta.
i'll edit the post when i'm done.

---

tried the format and it immediately told me that there wasn't enough memory to decompress that one file.
tried the latter and... honestly, it wasn't too bad (works as intended) but i haven't seen quickbms "dump" the decompressed file yet so i used HxD to see what was going on there (HxD can open the "main memory" where there are various "processes" to choose from) and it seems that quickbms is still decompressing the whole thing in memory?

again, i used the xcompress_file.bms script with a 1.5GB compressed file. this is the first quickbms instance i've opened.
so i opened up another quickbms instance and used that same script with a 31MB compressed file and the file was decompressed just as expected. the output file was dumped just as quickly, which is within a second on my PC.
of course, how much is the file actually "dumped" into a decompressed one depends on how big the compressed file is in size to begin with i didn't expect the former to hang for so long.
i'll keep this updated whenever possible.

aluigi, posted Wed Dec 15, 2021 5:12 pm (68265)

---

I use the following script for generating a 1.5 Gb file in memory and decompressing it as an 1.8Gb with lzss and it "works".
I mean that there is no endless loop and it takes 6 seconds to do the job:

Code:

math ZSIZE = 1500
xmath SIZE "ZSIZE   300"
xmath ZSIZE "ZSIZE * 1024 * 1024"
xmath SIZE  "SIZE * 1024 * 1024"
log MEMORY_FILE 0 0
putvarchr MEMORY_FILE ZSIZE 0
goto 0 MEMORY_FILE
comtype lzss0
clog "dump.dat" 0 ZSIZE SIZE MEMORY_FILE

The endless loop happening there may be caused by overlapped input and output data.
The pointer in the destination may reach the pointer in the source since it's the same buffer, that may happen as written in the Alert message.
Or maybe it's just the decompression which is slow or the xcompress library which is unable to uncompress a so big amount of data, there are better chances that this is the real explanation.

I like this work-around and it may help in those very rare cases where such big files need to be decompressed.

In your specific case I suppose that on x360 the library handles the file as a compressed filesystem or similar, being able to seek and read parts of it without performing any decompression of the whole file.

tbmq008, posted Wed Dec 15, 2021 7:16 pm (68270)

---

interesting...
so, that script you've just made... it took like 2-3 minutes to "do the job" and even then you actually got the file. and a decompressed one at that, given that the compressed input is all generated into memory and all that.
i couldn't get anything similar with that one file despite being compressed into an LZX container.

aluigi, posted Wed Dec 15, 2021 8:13 pm (68271)

---

Do you get 2-3 minutes with the testing script above or with xcompress_file.bms?

tbmq008, posted Wed Dec 15, 2021 8:30 pm (68272)

---

testing the script above.
i had to use it with a text file.

aluigi, posted Wed Dec 15, 2021 8:32 pm (68273)

---

It's strange, here only 6 seconds and there all that time?

It's not really important since the goal of the test was only the memory allocation, but still yours is 25x slower.

tbmq008, posted Wed Dec 15, 2021 8:39 pm (68274)

---

heh. must be my current PC hardware.
anyway, i'll just test the "empty lzss" script again and after that i'll let you know how it REALLY went this time.

---

OK, the whole thing took 1 minute and 12 seconds to run. the output file was dumped after that point.

spiritovod, posted Wed Dec 15, 2021 11:01 pm (68276)

---

aluigi, thanks for the fix, new beta works fine with all batch scripts I had for tests. As for suggestion with using txt files for filters, I'm not sure there will not be certain misunderstandings - like, you expect -F "*.txt" to work with all txt files in the folder (as input for script), but they will be loaded as filters instead(?)

aluigi, posted Thu Dec 16, 2021 9:33 am (68286)

---

It simply tries to load the specified file if it ends with a .txt extension, if it doesn't exist it will consider it a string, so:
-F "{}.txt" tries to load {}.txt which doesn't exist and will use it as a string
-F file.txt tries to load it as a file and use its line-by-line content as filter, if it doesn't exist "file.txt" is the filter string
-F asdf.asdf is a string

spiritovod, posted Thu Dec 16, 2021 2:49 pm (68293)

---

It sounds good. But what about cases like -F "file.txt; file2.txt", when you point to particular input files (not lists with filters). I suppose if it will be more than 1 argument for -F, it can be considered as string (i.e. input) regardless of extension, right?