tbmq008, posted Sat Nov 28, 2020 1:30 pm (60707)

---

checked with current beta, same problem.

Code:

-------------------
*EXCEPTION HANDLER*
-------------------
An error or crash occurred:

*EH* ExceptionCode      c00000fd stack overflow
*EH* ExceptionFlags     00000000
*EH* ExceptionAddress   75727FC7
                        756C0000   00067fc7 msvcrt.dll
*EH* NumberParameters   00000002
*EH*                    00000000
*EH*                    03D62000

Last script line before the error or that produced the error:
  329 getarray next_padded_sub_dir_number 8 current_directory_number
  coverage file 0     0%   192085     3490467840 . offset 000063e4

as for the script being long... well, thanks.
i've only made it long because i wanted to find a way to parse that specific structure without using "hacks" or anything similar. so i went all out with it.

aluigi wrote:

BTW, I guess it's a problem with the script because that "stack overflow" error is clearly referred to a function called many times recursively till it saturates the (huge) stack assigned to the process.

i guess that's why the script keeps breaking quickbms.

aluigi, posted Sat Nov 28, 2020 2:55 pm (60712)

---

Ok so it's definitely that one the problem.
Even if callfunction is invoked without saving variables, it still calls two internal functions recursively (start_bms and CMD_Function_func).

I will think if there is something I can implement to "reduce" this issue.

aluigi, posted Sun Nov 29, 2020 4:32 pm (60758)

---

The following is an example script for testing the problems of callfunction (without preserving variables which is memory consuming without the -9 option):

Code:

math COUNTER = 0
callfunction FUNCTION 1
startfunction FUNCTION
    print "%COUNTER%"
    math COUNTER   1
    callfunction FUNCTION 1
endfunction

It can do over 7000 iterations before consuming the stack.

Some years ago I played a lot with the "-Wl,--stack,BYTES" option for the gcc linker and at the end I opted for -Wl,--stack,4194304.
There is a reason behind that number (4Mb) based on all my tests but I don't remember if it was related to a limitation of the compiler or an incompatibility on older Windows platforms.

There is for sure a problem in getarray because it may consume about 600 bytes for nothing (easy to fix), but it would not increase much the number of cycles.

*edit*: I even tried to remove all those "goto" in start_bms() in bms.c but the result is the same.

aluigi, posted Sun Nov 29, 2020 5:41 pm (60761)

---

Anyway I just updated quickbms_exe.zip.
The cycles are the same as before, you may probably get few more due to the less space used for variable_copy invoked by getarray/putarray... but it would not solve the problem.

I have no other ideas at the moment.

tbmq008, posted Sun Nov 29, 2020 6:05 pm (60764)

---

alright
what i do need to tell you though is that this part

Code:

   goto 0
   math vol_ver = 0
   math c_01 = 0
   do
      savepos temp_02
      get vol_check_01 long
      get vol_check_02 long
      if vol_check_01 == 0xacb990ad
         math c_01   1
         putarray c_01 0 temp_02
         if vol_check_02 == 0x00020002
            math vol_ver = 1
         endif
         if vol_check_02 == 0x00030001
            math vol_ver = 2
         endif
      endif
      math temp_02  = 0x800
      goto temp_02
   while temp_02 != 0x2000000

is somehow causing a bug in the new beta.
like this.

Code:

  offset   filesize   filename
--------------------------------------
. 00000000 get     is_vol     "vol" -1002
.
*cut by aluigi*
.

-------------------
*EXCEPTION HANDLER*
-------------------
An error or crash occurred:

*EH* ExceptionCode      c00000fd stack overflow
*EH* ExceptionFlags     00000000
*EH* ExceptionAddress   75C97FC7
                        75C30000   00067fc7 msvcrt.dll
*EH* NumberParameters   00000002
*EH*                    00000000
*EH*                    03862000

Last script line before the error or that produced the error:
  17  do
  coverage file 0     0%   0          4261412864 . offset 00000000
.

aluigi, posted Sun Nov 29, 2020 7:51 pm (60766)

---

Yeah sorry and thanks for noticing it.
Removing "goto" from a function ever leads to problems
The exe is now fixed.

tbmq008, posted Sun Nov 29, 2020 10:03 pm (60770)

---

alright, with that being fixed i can now focus on trying to fix my script.
as i try to brave myself through as many stack overflow errors as possible.

aluigi, posted Mon Nov 30, 2020 7:24 am (60773)

---

Two funny things I noticed:

- quickbms_4gb_files uses less stack despite adopting 64bit integers since it can go over 10k cycles instead of 7k of quickbms.exe

- the new beta without "goto" performs worsen than the previous one (still in quickbms_beta.zip) that used the old goto solution in start_bms, the difference is about 200 cycles more in the old one

aluigi, posted Mon Nov 30, 2020 8:54 pm (60804)

---

spiritovod wrote:

Not sure if you read that topic already, but there is possible bug with quickbms_hash or escapes in general: viewtopic.php?f=9&t=14485

The problem was in the script.
This is a simple test to understand the behavior of "binary":

Code:

set VAR string "asdf\x22asdf"
print "%VAR%"
set KEY binary VAR
print "%KEY%"

I know, handling non-string variables and content is ever a headache with quickbms due to the zeroes (even if this is not the case) but the beta solves these problems with String:

Code:

set VAR binary "asdf\x00asdf"
print "%VAR%"
String KEY  = VAR
print "%KEY|dump%"
String KEY 0= VAR
print "%KEY|dump%"

spiritovod, posted Mon Nov 30, 2020 10:41 pm (60808)

---

So it's old good zeroes. While the base artemis_engine script now works fine with the game, I expect it will fail if some \x00 will slip in (though it should not, since it's a hash). Sorry, but did you consider to add some basic explanation about how zeroes works in quickbms to the documentation? Because such examples, when a script works fine with 2 out of 3 files with exactly the same structure, are... you know... confusing. For example, in UE4 there are guids instead of hashes, which can easily contains "\x00\x00" - and thus you can't use VAR h/b VAR in expected way for hex representation and you should use something like this instead (or any other workaround). It's like all related examples in the documentation doesn't consider that 00 will exist in input/output variables.

aluigi, posted Tue Dec 01, 2020 12:00 pm (60817)

---

The only problem with zeroes is related to the visualization of binary data and strings operations, and maybe copying binary variables to others.

In the example of the HASH you mentioned, there is absolutely no problem using "getdstring HASH 16" to read the data and it's stored and handled with the zeroes, so "putdstring HASH 16" perfectly works.
All the variables in quickbms are in binary form and include zeroes.

The problem is how to visualize that data to something readable for the user.
This one works: print "%HASH|dump%" and print "%HASH|hex 16%"
Also this one works: slog "file.txt" -1 16 (the output is with \ escapes)
This one does NOT work or not work well: string OUTPUT b HASH or string OUTPUT B HASH

In quickbms 0.11 this one works: string OUTPUT 0b HASH ; print "%OUTPUT%"

So, yeah, there is not really any problem.
All the constant strings in the bms scripts are handled as strings without any escape, those with the \ escape are tagged as "cstring". It's something inherithed from the original BMS language.
There is an option for both command-line and quickbmsver that forces all the constant strings to be cstring like any other programming language.

spiritovod, posted Tue Dec 01, 2020 1:10 pm (60820)

---

Yes, and my point was to put those kind of examples somewhere in the documentation around the header. Because null-terminated strings only exist in ?, so if you're switching from java/python to quickbms, those things are not obvious. Btw, I don't quite understand why you've mentioned cstring - if you mean -J option, does it works like usual string, but with escapes? I mean, how escapes works has nothing to do with such kind of examples in other languages, because the problem is caused by null-termination of strings which doesn't exist in vanilla version of most other programming languages.

Again, I'm sorry, but don't you think that you shouldn't explain how quickbms works internally to end users?
Let's take a random user - he want to convert a binary string to hex string with quickbms. He go to documentation and see that strings and binary are handled almost equally. And there is option to convert string to hex via b/B option. But like you've said by yourself, it will not work well in certain cases (and there are a lot of them if you're dealing with binary sequences). Do you mention that there are exceptions? No. Is that somehow explained in documentation? No. Should that user start to learn ? to understand things? Um... probably not.

aluigi, posted Tue Dec 01, 2020 3:09 pm (60822)

---

The original BMS language was very limited.
There was nothing like backslash escaped strings or similar, so what I did was simply improving the language by keeping most of its syntax.
And this process took many years since it was based on my personal needs with new formats.

That's why "string VAR VAR" where VAR has 0x00 bytes in it, has a different result than what you get by doing the same on languages like java.

Each version I'm trying to improve these mechanisms and, indeed, now (beta) "string VAR 0 VAR" works with binary data too.

The documentation already contains all the references to these limitations.
I improved it in the beta so maybe it's a bit more clear there.

quickbms started as a simple open source tool for the BMS language for my personal usage to avoid to make a standalone tool for each format, I didn't expect it to become now so diffused among users and modders for writing scripts.
If I knew that in advance maybe I would have taken difference choices

Ah, yes, with -J any string you use in the script can be escaped:
set VAR string "hello\x20\x22world\x22"

Shokoniraya, posted Tue Dec 01, 2020 3:18 pm (60823)

---

dear spiritovod
both string and binary works fine, it depends to user
and if a user wants a dufferent result, then it's better to do it by hand, QuickBMS already working as a binary tool as well to do things by hand, even converting unicode strings by hand

spiritovod, posted Tue Dec 01, 2020 7:14 pm (60834)

---

aluigi, you're right, next 0.11.x version should be a big improvement in handling binary sequences in more "streaming" way, like in common languages. And it's not like I'm complaining, quickbms is a great tool to deal with some daily tasks and especially all kind of encryption algorithms
I've checked papers for latest beta and couldn't find anything related to zeroes usage and limitations in strings. Maybe it's worth to add something like "if you expect null bytes in the string, it's better to use binary mode with 0 prefix" to related functions of String command or String description. And maybe it's just me, but old descriptions and examples for base string functions ( - ^ $ etc) were much more clearer than current ones.

Shokoniraya, that's not actually true. "By hand" doesn't mean you should guess how exactly a function works, compared to its description, and the only one who can explain such misunderstandings would be aluigi. But a simple example in the documentation can resolve a lot of those questions.

Shokoniraya, posted Tue Dec 01, 2020 9:48 pm (60844)

---

sir aluigi, there is a problem with creating a MEMORY_FILE by string

Code:

get FILE_NAMER filename
set MEMORY_FILE string FILE_NAMER
get MEM_SIZE asize MEMORY_FILE
log file_name 0 MEM_SIZE MEMORY_FILE

output file has many zero bytes after main string

aluigi, posted Thu Dec 03, 2020 10:23 am (60878)

---

@Shokoniraya
That's one of the weird things that rarely happen in quickbms due to some internal work-arounds.
It's a situation that should never happen since put/putdstring/putct are the best way for handling the writes to MEMORY_FILE (which is a file indeed).

Many years ago I decided to add a security layer to quickbms due to the various external libraries and code I used (and for my buggy code too obviously).
The source code is stored in extra\xalloc.c and it's a complete and transparent replacement of malloc/calloc/realloc/free.
In short every time some memory is allocated, quickbms puts it between two read-only pages that will trigger an exception if you try to read/write before/after your memory.
The first page also contains important information about the allocated memory.

Every page is 4kb, so if you allocate 1 byte quickbms will take 12kb (4 4 4).
It's a good thing for security and usually it doesn't hit much on the performances, obviously it takes more memory but it's still "ok" during normal usage.
If you use many recursive functions it's going to take tons of memory.

This feature was part of an experiment that I decided to implement in quickbms since it was perfect for it.

Long story short, the variables shorter than 271 characters are directly stored in the global array of variables without doing any allocation.
Longer variables require memory allocation.

That's the reason why you get that result but it's also a bug that I'm going to check since MEMORY_FILE should get the real length and not that "internal" length.

Ah, that allocation protection can be disabled with the -9 option.
It's usually suggested in some specific cases and requires user's acknowledgement if set via quickbmsver.

aluigi, posted Tue Dec 08, 2020 5:26 pm (60985)

---

spiritovod wrote:

And maybe it's just me, but old descriptions and examples for base string functions ( - ^ $ etc) were much more clearer than current ones.

I will check what I can do for the description of the String examples, I hope the following script can be helpful for testings all the various outputs:

Code:

# String command test

set VAR1 string "MyStringExampleString!"
set VAR2 string "STRING"
math NUM = 3
math NEG = -3

strlen TMP VAR1
print "VAR1 is %VAR1%"
print "VAR1 is %TMP% bytes"
print "VAR2 is %VAR2%"
print "NUM  is %NUM%"
print "NEG  is %NEG%"
print "."

print "    append"
set VAR string VAR1
string VAR   VAR2
print "      %VAR%"
set VAR string VAR1
string VAR   NUM
print "      %VAR%"
set VAR string VAR1
string VAR   NEG
print "      %VAR%"

print "-   remove, truncate"
set VAR string VAR1
string VAR - VAR2
print "      %VAR%"
set VAR string VAR1
string VAR - NUM
print "      %VAR%"
set VAR string VAR1
string VAR - NEG
print "      %VAR%"

print "^   xor"
set VAR string VAR1
string VAR ^ VAR2
print "      %VAR%"
set VAR string VAR1
string VAR ^ NUM
print "      %VAR%"
set VAR string VAR1
string VAR ^ NEG
print "      %VAR%"

print "<   strrstr   var2 (before)"
set VAR string VAR1
string VAR < VAR2
print "      %VAR%"
set VAR string VAR1
string VAR < NUM
print "      %VAR%"
set VAR string VAR1
string VAR < NEG
print "      %VAR%"

print "*   replicate"
set VAR string VAR1
string VAR * VAR2
print "      %VAR%"
set VAR string VAR1
string VAR * NUM
print "      %VAR%"
set VAR string VAR1
string VAR * NEG
print "      %VAR%"

print "%%   strstr (before), truncate, mod"
set VAR string VAR1
string VAR % VAR2
print "      %VAR%"
set VAR string VAR1
string VAR % NUM
print "      %VAR%"
set VAR string VAR1
string VAR % NEG
print "      %VAR%"

print "&   strstr"
set VAR string VAR1
string VAR & VAR2
print "      %VAR%"
set VAR string VAR1
string VAR & NUM
print "      %VAR%"
set VAR string VAR1
string VAR & NEG
print "      %VAR%"

print "|   strstr   var2"
set VAR string VAR1
string VAR | VAR2
print "      %VAR%"
set VAR string VAR1
string VAR | NUM
print "      %VAR%"
set VAR string VAR1
string VAR | NEG
print "      %VAR%"

print "$   strrstr"
set VAR string VAR1
string VAR $ VAR2
print "      %VAR%"
set VAR string VAR1
string VAR $ NUM
print "      %VAR%"
set VAR string VAR1
string VAR $ NEG
print "      %VAR%"

print "!   strrstr   var2"
set VAR string VAR1
string VAR ! VAR2
print "      %VAR%"
set VAR string VAR1
string VAR ! NUM
print "      %VAR%"
set VAR string VAR1
string VAR ! NEG
print "      %VAR%"

print ">   strrstr (before)"
set VAR string VAR1
string VAR > VAR2
print "      %VAR%"
set VAR string VAR1
string VAR > NUM
print "      %VAR%"
set VAR string VAR1
string VAR > NEG
print "      %VAR%"

aluigi, posted Sun Dec 13, 2020 6:40 pm (61091)

---

I just post it here, it's not an error but rather a "feature".
Long story short some instructions like calldll load the MEMORY_FILE from the current position rather than from its beginning.

The reason is that in some situations it's very useful to seek to a specific offset of the memory file and using it from that position.

I now realized that also the NameCRC instruction and probably others do the same.

For example I built a MEMORY_FILE with the list of hashes and filenames, and it didn't work.
The reason is that I had to use "goto 0 MEMORY_FILE" first.

Keep it in mind!

gameside, posted Tue Dec 15, 2020 8:55 pm (61131)

---

Hi, its possible to remove some byte from a file?
I mean like put or putdstring, but instead of replacing bytes, remove them

aluigi, posted Wed Dec 16, 2020 4:55 pm (61142)

---

As far as I remember there is no easy way to do that.
Something like that is automatically done when using reimport3 and your reimported file is smaller.
Your best alternative is just dumping a new file with the desired size.

Shokoniraya, posted Thu Dec 17, 2020 2:47 pm (61159)

---

sir aluigi, i found two problem in QuickBMS


1) switching encryption key
it's a script to encrypt a MEMORY_FILE and looks like there a problem with switching encryption key

Code:

set MEMORY_FILE1 binary "\x16"

set MEMORY_FILE3 binary "\x00"
for i = 0 < 255
goto 0 MEMORY_FILE3
get KEY_CODE byte MEMORY_FILE3
encryption xor KEY_CODE
goto 0 MEMORY_FILE3
math KEY_CODE   1
put KEY_CODE byte MEMORY_FILE3
log MEMORY_FILE2 0 1 MEMORY_FILE1
encryption xor ""
goto 0 MEMORY_FILE2
get BLOCK byte MEMORY_FILE2
print "%BLOCK%"
next i

2) label and continue
a attached a zip file, in _export_table.bms, in line 18 there is a continue SECTION_PARSER
and also there is a label SECTION_PARSER in line 26 too, but not working at all

continue_problem.zip

spiritovod, posted Thu Dec 17, 2020 3:51 pm (61163)

---

@Shokoniraya: It's just the parser script logic is invalid, because you can't jump into cycle, only outside of it, due to its variables may not be initialized at the moment (and other reasons as well).

Shokoniraya, posted Thu Dec 17, 2020 4:41 pm (61166)

---

i found the real problem
continue not working under if-endif command

Code:

math ALLOW_MODE = 1

if ALLOW_MODE = 1
continue PASSING
else
print "not allowed"
cleanexit
endif


label PASSING
if ALLOW_MODE = 1
print "allowed!"
endif

there is same problem with do-while

Code:

math COUNT = 2

do
print "same number detected"
continue next_for
while COUNT != 2
cleanexit


for k = 0 < 4
label next_for
print "%k%"
next k

spiritovod, it's possible to jump into a cycle

Code:

math COUNT = 0
for i = 0 < 4
math COUNT   1
print "%COUNT%"
next i
continue next_for
cleanexit


for k = 0 < 4
label next_for
print "next for!"
print "%k%"
next k

problem is jumping out of a cycle, if-endif or do-while

Code:

math COUNT = 0
for i = 0 < 4
math COUNT   1
print "%COUNT%"
continue next_for
next i
cleanexit

label next_for
print "passed!"

spiritovod, posted Thu Dec 17, 2020 8:19 pm (61170)

---

@Shokoniraya: That's how quickbms is dealing with initializing unknown variables, but it doesn't work properly that way. You can run the script with -v option to understand it more clearly. Take a look at it's output - with normal approach "for k = 0 < 4" should get you "0 1 2 3", but in your case it will be "k 1 2 3". So you expect that K variable will be integer in the first iteration, but it's actually a string. The best advice would be not to use such expressions in any script.

But problem with jumping with "continue" from if-endif / do-while cycles is indeed exist in latest beta version (in 0.10.1 it works fine).