aluigi, posted Mon Jun 22, 2020 4:56 pm (57327)

---

Shokoniraya wrote:

when i want to run a exe with a none-standard cstring name like ( or ) that should used " in beging and end of name, in execute command in bms script, command not working fine

"zlib(v2.4).exe -c #INPUT# #OUTPUT#"
or even this one: "\"zlib(v2.4).exe\" -c #INPUT# #OUTPUT#"
or this: ""zlib(v2.4).exe" -c #INPUT# #OUTPUT#"

The problem is not in the string, I guess it's quickbms that uses parenthesis as escape so using " or \( doesn't solve it.

I'm quite sure there is a reason behind that behavior but I will check it when I will work on the next release

(I will slowly try to answer to all the posts, just wait)

aluigi, posted Mon Jun 22, 2020 5:00 pm (57328)

---

Shokoniraya wrote:

sir aluigi

there is a problem in QuickBMS that always making some problem in text importing
there is some string like \n or \f or \e
but can you skip none cstring error? like \:

Error: cstring() failure, your input string has some wrong escape sequences or
it's not a valid escaped string

i know i should use \\:
but \: not used in cstring, right? becuase there is not any \: or \k too, so error and stop the process can't helping, then simply ignore unavailable escapes

and i know if i export that text again, it will turned to \\:
but there is no problem with that, so can you please ignore wrong escapes in next version?

thank you

Are you using strings with invalid escapes? And why?
Are you talking about the Slog command?

I probably need a complete example for understanding if it's a problem or not.

The error is necessary because people need to know if something failed.

aluigi, posted Mon Jun 22, 2020 5:02 pm (57329)

---

Shokoniraya wrote:

sir aluigi, about cstring errors, i know there is some problems for that if you ignore it as default, but it's better to make a command option to ignore incorrect \* cstring

and about unicode converting that skips at zero, normaly if we want to read a 64 bytes that stored a name in it, it will stop at zero (null). so there must be a option to ignore zero bytes too
because of such cases

Code:

63 6F 6D 6D 6F 6E 5C 00 00 00 00 00 75 69 5C 00 common\.....ui\.
00 00 00 00 00 00 00 74 65 78 74 00 00 00 75 72 .......text...ur
00 00 65 00 00 00 5C 00 00 00 66 6E 74 5F 00 00 ..e...\...fnt_..
00 00 00 30 2E 00 00 00 00 00 66 6E 74 00 00 00 ...0......fnt...

I don't know what you mean.
And that's not unicode.
Clearly a format like splitted strings for manual reassembling in different ways and not a string.

aluigi, posted Mon Jun 22, 2020 5:08 pm (57330)

---

Shokoniraya wrote:

another problem, but i don't know that if it's a QuickBMS feature or not

append -1 not working as well, i think it should add binary to start of file but has problem

please test it on a 40 byte file (you can see it better in a small file)

Code:

get SIZER asize

for i = 0 append -1
log FILE_APPEND 0 SIZER
append -1
next i

Ok, here I guess it's a bug because I tried 10 Log of 3 bytes each but the output is 6 bytes instead of 30.
Added to the list of things to check.

Ah, it works perfectly with MEMORY_FILE so it's only related to log to file.

Likely "append -1" is never used

aluigi, posted Mon Jun 22, 2020 5:20 pm (57331)

---

spiritovod wrote:

Now I see what you mean, "append -1" only works for a single iteration, because it's allocating additional memory only once - on the 2nd iteration and the rest it stop allocating and start to rewrite file (like in "append 1" mode). Not sure, but it may be actually related to position checks for memfile. If you allocate required memory manually, it works fine though.

I hope aluigi will take a look at that when he has time. Also, some basic example would be appreciated, since I'm also confused with how "append 1" works - in documentation it's stated that you can choose position where to write stuff in rewrite mode with goto, but you expect that goto will work on current file. You may expect that you should open result file as 1, then something like "goto OFFSET 1" and then "log ..." into that file from memfile, but that doesn't work as expected.

Append 1 is meant to be used on MEMORY_FILEs because you can't move on output files on disk.

The following scripts use "append 1" if interested:
http://aluigi.org/bms/bsc.bms
http://aluigi.org/bms/clutch.bms
http://aluigi.org/bms/gh_pak_chnk.bms
http://aluigi.org/bms/marvel_ultimate_alliance_2.bms (there is also a comment explaining why it gets used)
http://aluigi.org/bms/ravioli_reimporter.bms
http://aluigi.org/bms/spiderman_pcpack_nch.bms
http://aluigi.org/bms/the_order_1886.bms
http://aluigi.org/bms/zombiu.bms

Regarding the script of Shokoniraya, it's necessary to use "append" without any argument for resetting the append mode, so "append -1" log "append".

aluigi, posted Mon Jun 22, 2020 5:34 pm (57333)

---

Shokoniraya wrote:

there is two problem that i've found

1) codepage problem
when codepage used in a script, some empty folders will be create with none-codepaged names, but it just a junk folder and codepage works correctly

2) unicode names for input file
it's clear that cmd can't support unicode as well, but for %%x can be used in any file, no matter file names has space character or unicode names and quickbms can't support unicode in input name

I will check what causes that "Ianei" folder to be created.

Regarding the second problem I can check why the name isn't correctly handled as utf8 when quickbms points to the folder, probably a bug in the function that scans the folders.

spiritovod, posted Mon Jun 22, 2020 6:23 pm (57337)

---

^ Thanks for those examples, now I see how "append 1" should be used in different situations. Also not sure, if "append 1 -- -- append 1" is correct construction in bsc script, can you please check it?
As for the "append -1" problem, if you'll use it just once (append -1 / log to file / append) it will work as expected, so it looks more like positioning/allocating issue for multiple iterations.

aluigi, posted Fri Jun 26, 2020 9:06 pm (57373)

---

"append 1 -- append" is the correct one.

Shokoniraya, posted Wed Jul 29, 2020 11:22 pm (57808)

---

hello sir aluigi
can you check these two things?

1) compression library
QuickBMS zlib compression (in reimporting) is really slower than any other zlib compressor tools
can you add a heavy optimized zlib in QuickBMS?

and some compressed lzo1x files are bigger than original files, (i mean recompressing original files without any change), i think you should change it with a newer version
about LZX compression, i can see QuickBMS already has decompressor for that, can you add LZX compressor too?

2) search content of MEMORY_FILE in findloc
i want to read a few byte (14 kb) and use it in findloc, is this possible?
after reading QuickBMS txt file, i could not find any useful thing to do it
can you tell me its possible or not, or how can i do it? becuase its really needed

spiritovod, posted Thu Jul 30, 2020 4:39 pm (57817)

---

^
1) How can you compare compression tools with quickbms if reimport function is doing more work than any generic compressor by default? It's not about zlib, it's about reimport feature in general.

2) You can do it with something like:
getdstring SEARCH BYTES_LENGTH
# not sure if "set SEARCH binary SEARCH" conversion is required
findloc RESULT binary SEARCH MEMORY_FILE ""

Shokoniraya, posted Thu Jul 30, 2020 10:42 pm (57819)

---

thank you spiritovod for your answer
but QuickBMS zlib and lzo1x is not optimized, you can compress files even without using reimport option and just with adding _compress to your comtype, if QuickBMS have compressor for that comtype

Code:

comtype zlib_compress
get FILE_SIZE asize
clog compressed_file.zlib 0 FILE_SIZE FILE_SIZE

and getdstring is not a good option for reading a few byte, its for null terminated string and that means output result will stop on first null byte (zero \x00"), so can't use it to read \x00\x01\x00\x04 and search it, getdstring is works just like get TEXT string, but just can adjust number of bytes to read it
and i tried MEMORY_FILE and not working

spiritovod, posted Fri Jul 31, 2020 2:22 pm (57828)

---

The problem is actually not that. I remember that aluigi said somewhere that it's not possible to directly use a variable to search in binary file like that, but it's rather because you should understand how quickbms is working with strings / cstrings / numbers and convert them (for example, I don't). And it seems there is a problem with leading zeroes, which works as delimiters as well. So, for example, this code will work as intended:

Code:

set SEARCH binary "\x11\x00\x22"
set MEMORY_FILE binary "\x01\x01\x02\x02\x00\x11\x00\x22"
findloc RESULT binary SEARCH MEMORY_FILE ""
print "Result is: %RESULT%"

but the following ones doesn't (considering that you change memfile accordingly):
set SEARCH binary "\x00\x11\x00\x22"
set SEARCH binary "\x01\x00\x02"

The example above works only because it was created this way. Actually it's searching only "\x11" in the memfile, because SEARCH is getting trimmed due to the delimiter (which is obviously non-intented in quickbms).

----------------------------------------------

And getdstring works in a different way, it's about representation as well. Take a look at the example:

Code:

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
getdstring SEARCH 4 MEMORY_FILE
string SEARCH b SEARCH
string SEARCH - "0000"
string SEARCH >> 2
print "Search: %SEARCH%"

I believe if leading zeroes problem will be fixed somehow, you can find anything with getdstring and some workaround regardless of zero bytes delimiters.

Shokoniraya, posted Fri Jul 31, 2020 4:24 pm (57832)

---

I know how it works and thank you so much to spending your time to explain it

so one of things that we need in next version is getBinary

Code:

getbinary MY_BINARY 4

spiritovod, posted Fri Jul 31, 2020 5:33 pm (57833)

---

Shokoniraya wrote:

I know how it works

Sorry, but considering that you didn't even try to understand my previous post, I hardly doubt.

Let me put it in more simple way: "getbinary MY_BINARY 4" = "getdstring MY_BINARY 4" with current implementation of "binary" term in quickbms. It's just "binary" is being resolved as null-terminated string, so every leading \x0 (it's not zero byte) is considered as delimiter. Otherwise first example from my previous post would work in all possible cases. And findloc is working with strings in that case (it's ever written in the documentation that quickbms handles both binary and strings in the same way), not with byte array. For some functions there is already additional option to ignore delimiter, so you can use null bytes with them as well (for example, putct).

I wonder though, why nobody didn't notice that till now. Null delimiters and leading zeroes are common questions when you're starting to work with binary files (especially binary <-> hex strings representations) in almost all common languages.

-----------------------------------------------

I just came up with one funny example. If you can understand how it works, you can understand the whole representation / conversion thing.

Code:

math TMP = 4
set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
getdstring SEARCH TMP MEMORY_FILE
set SEARCH1 string SEARCH
set SEARCH string ""
math TMP * 2
for i = 0 < TMP
   getvarchr TMP2 SEARCH1 i
   string SEARCH   TMP2
next i
print "Search: %SEARCH%"
string SEARCH h SEARCH
set MEMORY_FILE5 binary "\x01\x01\x02\x02\x00\x01\x00\x02"
findloc RESULT binary SEARCH MEMORY_FILE5 ""
print "Result: %RESULT%"

Shokoniraya, posted Sat Aug 01, 2020 12:50 am (57841)

---

i understand, but can you tell me about math TMP * 2 at line 6? was it necessary? because TMP is 4 and with * 2, it reading 8 times and other 4 times is just zero

but after take a look at line 9, you did it in this way: string SEARCH TMP2
actually, you just attaching decimal numbers together, (you can set all \x00 to \xFF in line 2 and add a print "%TMP2%" after line 8 to see it) like below example

Code:

math TMP = 4
set MEMORY_FILE binary "\xFF\x01\x00\x02\x00\x03"
getdstring SEARCH TMP MEMORY_FILE
set SEARCH1 string SEARCH
set SEARCH string ""
math TMP * 2
for i = 0 < TMP
   getvarchr TMP2 SEARCH1 i
   print "%TMP2%"
   string SEARCH   TMP2
next i

and about line 12: string SEARCH h SEARCH
you want to alterant 01020000 to hex, but we want to find \x00\x01\x00\x02 in MEMORY_FILE5, right? so it ended to \x01\x02?! i think reason is getvarchr, because it just reading decimal numbers and string SEARCH h SEARCH reading string two character by two character and set them as hex, so for example, \xFF\x01\x00\x02 is 4 decimal number and getvarchar and string will make 255102

here a clear examlpe that tells its not working with \x00

Code:

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
get MEM_SIZE asize MEMORY_FILE

string HEX_STRING = ""
for i = 0 < MEM_SIZE
get BYTE_NUM byte MEMORY_FILE #same job as getvarchr to get every VAR in for-next
string HEX_BYTE p "x" BYTE_NUM #so it will make a real hex-string
string HEX_STRING   HEX_BYTE
next i
print "hex string: %HEX_STRING%"
string HEX_STRING h HEX_STRING #and it will make it byte, but its string! so \x00 will be a problem!

set MEMORY_FILE5 binary "\x00\x01\x00\x02\x00\x03"
findloc RESULT binary HEX_STRING MEMORY_FILE5 ""
print "Result: %RESULT%"

null terminate: means reading string until \x00
but getbinary is needed

Shokoniraya, posted Sat Aug 01, 2020 1:03 am (57842)

---

sir aluigi, after working on some files with lzo1x, im sure QuickBMS lzo is a buggy version (maybe old and corrupted) and some files will be corrupt in decompressing, some equal bytes in decompressed files always messed up or changed with some uncorrect bytes

aluigi, posted Fri Aug 21, 2020 9:38 am (58237)

---

quickbms already uses the latest versions of the compression libraries.
It also already uses the best compression levels, example for lzo1x:
lzo1x_999_compress(in, insz, out, &len, wrkmem);

Using a better compression level doesn't mean that the output will be ever smaller, sometimes it can be bigger with some samples.

zlib is slow for various reasons.
Differently than other packing tools, QuickBMS is a reimporter and so its priority is the final size of the data which means using maximum compression for fitting the original size.
It even uses 3 different types of zlib if the final data is bigger than the original which means a huge amount of time.

I guess that you have that problem only with reimport.bat since reimport2.bat should just use the normal compression.

Searching binary data with quickbms is difficult because the tool is not meant for that.
I can try to improve findloc binary but I doubt in any better result.

What is that "getbinary"???
getdstring already does the job

Shokoniraya, posted Tue Aug 25, 2020 10:37 am (58338)

---

getdstring can't do that job

here a clear example

Code:

set MEMORY_FILE binary "\x00\x01\x00\x02\x00\x03"
get MEM_SIZE asize MEMORY_FILE

string HEX_STRING = ""
for i = 0 < MEM_SIZE
get BYTE_NUM byte MEMORY_FILE #same job as getvarchr to get every VAR in for-next
string HEX_BYTE p "x" BYTE_NUM #so it will make a real hex-string
string HEX_STRING   HEX_BYTE
next i
print "hex string: %HEX_STRING%"
string HEX_STRING h HEX_STRING #and it will make it byte, but its string! so \x00 will be a problem!

set MEMORY_FILE5 binary "\x00\x01\x00\x02\x00\x03"
findloc RESULT binary HEX_STRING MEMORY_FILE5 ""
print "Result: %RESULT%"

and can you do something to have some options to choose temp files for exe executing? some codec tools don't have [output] (example: program.exe -c [input] [output]), so output will have a random name

aluigi, posted Tue Aug 25, 2020 11:18 am (58339)

---

That's exactly the job of getdstring.
What you are showing is something different related to strings (strings are NUL delimited) and findloc which is still related to strings.
I can do something for Findloc and maybe a work-around for String when used with some operators.

This is a limitation that affects me too and I want to find a solution, but it's NOT related to getdstring

Regarding the other request I can't do anything because quickbms gets the result from the output file generated by the program.
quickbms doesn't act as hooker or hypervisor of the program, it just creates the input file, pass it to the program and read the generated output file.

Shokoniraya, posted Tue Aug 25, 2020 5:20 pm (58348)

---

and about temp names, temp files created in quickbms temp folder, if there was a way to have last temp file adress that created in folder, then problem will solve

Shokoniraya, posted Wed Aug 26, 2020 1:02 pm (58362)

---

aluigi wrote:

That's exactly the job of getdstring.
What you are showing is something different related to strings (strings are NUL delimited) and findloc which is still related to strings.
I can do something for Findloc and maybe a work-around for String when used with some operators.

This is a limitation that affects me too and I want to find a solution, but it's NOT related to getdstring

Regarding the other request I can't do anything because quickbms gets the result from the output file generated by the program.
quickbms doesn't act as hooker or hypervisor of the program, it just creates the input file, pass it to the program and read the generated output file.

and about getdstirng, that's right, that little example wasn't related to getdstring, but i'm saying that string will skip in \x00 (null delimited)
so if we use getdstring TEXT 15 on \x68\x65\x72\x65\x00\x61\x6E\x00\x65\x78\x61\x6D\x70\x6C\x65, then it will stop on \x00, but we need whole string, that's why im saying that there is a getbinary needed

kotn3l, posted Sun Aug 30, 2020 4:03 pm (58441)

---

Trying to extract PSARC files using the latest quickBMS and brink.bms versions from a new-ish PS4 game. I had to use the 4GB exe version, because the PSARC file I wanted to unpack is over 7GB.

Code:

- error in src\extra\xalloc.c line 618: xdbg_malloc()
Error: memory allocation problem
       Not enough memory resources are available to process this command.

It can extract some psarc files pretty easily. I guess the problem here is that the file it wants to unpack is way too big. Any workaround? Or do I just wait for a fix? Is there a way to continue unpacking after that point?

Shokoniraya, posted Tue Sep 01, 2020 4:29 pm (58482)

---

there is two problem that i have found in QuickBMS

1) bms_folder
BMS_FOLDER can't get script folder as well
my_script.bms is located in "G:\my_scripts\works\pak_format\my_script.bms"

and this is my command
"quickbms.exe" -o "G:\my_scripts\works\pak_format\my_script.bms" "my_file.bin" "output_folder"
result is "G:\my_scripts\works\pak_format" and it will work

but if i use it without a full name and drive and file name, result is "G:\my_scripts\works" for this command
"quickbms.exe" -o "my_script.bms" "my_file.bin" "output_folder"

i think QuickBMS removing last sub-folder or sub-name from bms adress
problem is that i can't use it if i write script name without full path


2) EXECUTE command
EXECUTE in comtype command has a problem, and maybe it can't works well with Temp folder on a windows user name that has space character in it (like my system user name)
because i get this error
@calc\calc.exe INPUT_NUMBER / 32 >C:\Users\************\AppData\Local\Temp\quickbms_00000da80000000181adcaa4289aa2b5.tmp
The system cannot find the path specified.

here a example: http://www.mediafire.com/file/ew70myc2t ... m.zip/file

aluigi, posted Wed Sep 02, 2020 3:03 pm (58500)

---

Quote:

and about temp names, temp files created in quickbms temp folder, if there was a way to have last temp file adress that created in folder, then problem will solve

There are tons of processes creating files there

Quote:

Trying to extract PSARC files using the latest quickBMS and brink.bms versions from a new-ish PS4 game. I had to use the 4GB exe version, because the PSARC file I wanted to unpack is over 7GB.

That's off-topic here.

Quote:

so if we use getdstring TEXT 15 on \x68\x65\x72\x65\x00\x61\x6E\x00\x65\x78\x61\x6D\x70\x6C\x65, then it will stop on \x00, but we need whole string, that's why im saying that there is a getbinary needed

I'm working on the next version of quickbms just in these days and the results are interesting because the String command is now no longer limited to NUL strings.
Obviously there will be some incompatibilities with some scripts and some headaches in some situations, I will upload a beta quickbms.exe soon so you can try the new features.

Quote:

1) bms_folder
BMS_FOLDER can't get script folder as well

What's the current folder from where are you launching quickbms?
If I'm in g:\my_scripts\works\pak_format both the examples create output_folder where expected.

Quote:

2) EXECUTE command
EXECUTE in comtype command has a problem, and maybe it can't works well with Temp folder on a windows user name that has space character in it (like my system user name)
because i get this error

Good, this is definitely something to fix (and apparently related to the name(blah).exe bug)

Shokoniraya, posted Thu Sep 03, 2020 1:52 am (58514)

---

about temp folder, yes, there was so many files on there, i'm not talking about last file that created in temp folder (based on time stamp), i mean last file that EXECUTE created it
quickbms dumping that file on temp folder, right? so it will be useful if get dumped file path from dumper function

and about bms_folder, i upload a example, please use it under two sub folder, like C:\folder_1\folder_2\bms_folder
http://www.mediafire.com/file/2zf956e2j ... r.zip/file

in just_script_name.bat, last sub folder is missing from result

and can you add LZX compressor too?

using -n in QuickBMS, is there any way to get file name in a download link that does'ny have any name?
becuase some links does not have any name or not a normal name like php file link
for examles, these links
"https://zenhax.com/download/file.php_avatar=2_1542023899.png"
"https://zenhax.com/files/file.php_id=9002"
"https://avatars1.githubusercontent.com/u/5502283?s=460&u=861e923455e5411cd3bfb07a8c533db777ea9276&v=4"

maybe they are using some java script method or redirected links
and i thing downloading a file from a server with QuickBMS is slow (downloading speed)