aluigi, posted Wed Mar 08, 2017 12:14 pm (21282)

---

Compression and custom encryption algorithm should not implemented in bms language.
The correct way is dumping the function/dll (maybe in a MEMORY_FILE to avoid external files) and using it with CallDLL.

Labels for the cycles?
The language is meant to be simple and as close as possible to the original one.

Nameless, posted Wed Mar 08, 2017 12:36 pm (21287)

---

Then how to fix that very annoying bug in the other way? Sometimes break is pretty much useless in its current state, either it doesn't work as it should or gives unpredictable result, but many decompression algorithms rely on break in the cycles, there is no other solution for such cases.

aluigi, posted Wed Mar 08, 2017 12:41 pm (21288)

---

The work-around I use when I need it (rarely to be honest) is an OK variable set to 1 by default, and set to 0 when I have to break or I set the initial value of the cycle to a value that causes its termination (for example i = 9999 if for i = 0 < 100).

Just curious, what compression algorithms are you implementing in bms language?
I bet it's graphic stuff, right?

Nameless, posted Wed Mar 08, 2017 12:44 pm (21289)

---

aluigi wrote:

Compression and custom encryption algorithm should not implemented in bms language.
The correct way is dumping the function/dll (maybe in a MEMORY_FILE to avoid external files) and using it with CallDLL.

Dumping function from where to where? From MIPS code to where? Or from PowerPC? Or from ARM? Games exist not only for PC, don't you understand it? And the only way is to disassemble game's native code, find decompression routine and translate it manually into human readable code. If quickbms can do so many things then it should handle the most basic functions flawlessly. Otherwise why you implemented all those functions at all?

aluigi, posted Wed Mar 08, 2017 12:47 pm (21290)

---

"break" wasn't available in the original language.
So can't you just translate the reverse engineerd function into C code and use it (yeah "dumping" since there is no need of dll if it's simple) with calldll?

aluigi, posted Wed Mar 08, 2017 12:48 pm (21291)

---

And in any case I prefer to have a "break" that works in most of cases rather than not having it at all.

Nameless, posted Wed Mar 08, 2017 12:59 pm (21292)

---

aluigi wrote:

"break" wasn't available in the original language.
So can't you just translate the reverse engineerd function into C code and use it (yeah "dumping" since there is no need of dll if it's simple) with calldll?

I can, but otherwise what for we have quickbms? For me personally quickbms is very handy because of its simplicity, it works out of the box and the most important - it doesn't take too much time to write script which will do the job.

aluigi, posted Wed Mar 08, 2017 1:12 pm (21293)

---

I know.
Let's say the "break" problem gets fixed, it will remain the huge slowness issue affecting the bms code used for compression and encryption stuff (input->operations->output). Even 10 megabytes of input data can take minutes to be elaborated.
Regarding the readability, it's impossible to compare the length and readability of a function written in bms (long and chaotic) with the same written in C.

So the easiest and most elegant solution is just a precompiled function or dll used with CallDLL.
It's fast, you can embed it directly in the script and if you want to keep the source code you can just put it in a comment.
There is even the possibility to use it in the Log/CLog command through the Comtype and Encryption instructions, the input/input_size/output/output_size will be automatically handled.
The only negative point is that the user will be prompted to acknowledge the usage of the code, just that.

aluigi, posted Wed Mar 08, 2017 1:15 pm (21294)

---

There are various examples on my website about calldll, the following covers both calldll and the commented original function:
http://aluigi.org/bms/tales_of_berseria.bms

While this is the most known example related to calldll used directly in log/clog:
http://aluigi.org/bms/rfactor2.bms

Nameless, posted Thu Mar 09, 2017 8:28 am (21314)

---

I still think that quickbms must have ability to do everything on its own, in other words it must become self-contained tool which doesn't require external bells and whistles.

And I really whish that in one day labels in conjunction with jump operator will be added as a new feature. Is it really that hard to add such feature?

aluigi, posted Thu Mar 09, 2017 12:34 pm (21316)

---

Quote:

I still think that quickbms must have ability to do everything on its own, in other words it must become self-contained tool which doesn't require external bells and whistles.

In my opinion here there is a misunderstanding because quickbms is not an interpreter of a programming language, we have just a limited set of instructions with the final goal of extracting data from an input (file, network, memory and so on).
The problems raise when people start to use quickbms instead of a programming language.
I'm glad of that but making (for example) a 3d model converter in bms instead of python is not a good option.

For example this is a script I made years ago to implement the lzss algorithm in bms language: http://aluigi.altervista.org/bms/pure_l ... t_only.bms
It's huge and slow, much better to implement it as one-line pre-compiled binary to use with calldll.
Or better to use the native one, if people have custom lossless compression algorithms they want to see natively in quickbms then it's enough to contact me and I will add them immediately in the next release.

Quote:

And I really whish that in one day labels in conjunction with jump operator will be added as a new feature. Is it really that hard to add such feature?

My goal is to fix the "break" command, but I don't want to have something that will start to no longer work due to this change, and being "break" a sort of experimental instruction it's priority is really very low.

Currently the development of quickbms is focused on new algorithms to implement and bugfixes, so everything that doesn't require time and that will not affect the current features (fixing "break" is a bugfix with possible effects on the stability).
A feature like the one you proposed (which is basically the label goto of C) is of no use to users, as you said that "may" be useful only if you reverse engineer a compression/encryption algorithm and want to implement it as bms language. But in that case the solution already exists: calldll.

Could you please tell me what's your problem with calldll?
It's easy and elegant, I really don't get what's the problem.

Nameless, posted Thu Mar 09, 2017 1:53 pm (21318)

---

Quote:

Could you please tell me what's your problem with calldll?
It's easy and elegant, I really don't get what's the problem.

The problem is that not always you can have access to all needed toolchain for that. And when you want to do something only once, is speed really important? Absolutely no. When job is done, the used script becomes pretty much useless and the only thing you can borrow further is your experience. But if I really need to use programming language, then I can use it from start to finish, right? Why do I need quickbms then? But no, I need it because it's much easier to deal with, no need to bother about dependencies, compiling and debugging. Convenience is the most important factor here, the less dependencies the better.

aluigi, posted Thu Mar 09, 2017 2:06 pm (21319)

---

Can you provide some context?
I mean what you are doing, what programming language do you know, how big is the function you are reverse engineering, why quickbms would be better than python or C, and so on.

I'm thinking to what I can do, maybe there are easy solutions that I can implement in the next version.
If a 100% working "break" solves all the problems then I will put it at highest priority and finally closing this bug.
"Label" "goto" is probably possible (I mean that it's probably not a huge work) if "break" works, obviously "goto" would not be used because already exists that instruction, maybe it can be an additional feature of "break" or "continue" (yeah there is even an experimental continue instruction), like "break label1" or "continue label2".
Just thinking...

Nameless, posted Thu Mar 09, 2017 2:43 pm (21321)

---

aluigi wrote:

Can you provide some context?
I mean what you are doing, what programming language do you know, how big is the function you are reverse engineering, why quickbms would be better than python or C, and so on.

My main hobby is modding, I never concentrate my doings on one language because of that. Different games have different gameplay core, some use their own script language, some use common ones, some use C, so when I deal with a new game, I need to use its language and because most of programming languages have lots of similarities it's not that hard to learn rules of the new one.

And why I really asked for labels is because it can help to make quickbms more versatile. Of all decompression algorithms supported to this day I had only two very lucky cases when quickbms detected used compression of the files and those cases were pretty laughable, one was packbits-RLE and the second was deflate, lol, I could figure it out by my own if I wasn't so lazy in that day... So I thought that It would be great to have some sort of backdoor in quickbms where it will be possible to create custom decompression functions by converting disassembled code into script. Actually RISC processors, for example, have very simplistic instruction set, so the only thing left for such possibility is labels and jump.

aluigi, posted Thu Mar 09, 2017 2:55 pm (21322)

---

Ok, I'm doing some tests with "break" after a quick fix that includes also the dormant "continue" instruction.
Everything seems ok but obviously I need more tests cases because I'm sure there is something missing or wrong, it would be too good to have fixed this issue in less than 5 minutes

If you can produce some scripts that I can test, it would be great.

aluigi, posted Thu Mar 09, 2017 4:51 pm (21323)

---

Situation at the moment:
Currently it looks like "break" and "continue" work correctly.
Labels work only if they are below the current goto ("continue"), so label loc_1, continue loc_1 will not work.
I have the possibility to embed a runtime C compiler in quickbms, I have already tried it and works perfectly

aluigi, posted Thu Mar 09, 2017 10:14 pm (21331)

---

aluigi wrote:

The hanging of put unicode with the default codepage is for sure a bug... I released quickbms 0.7.7 just few days ago so who knows when I will check this thing.

Using codepage 1251 before the put command worked.
You have also to use "set str unicode str" after "get str unicode".

Anyway are you sure you really want to use quickbms for string related stuff like this?

I'm checking this bug of the endless loop with codepage 1200 but it's not a bug of quickbms.
Basically MultiByteToWideChar is unable to return a valid character with this codepage and so quickbms tries to give it an input which increases its size till the maximum length of the string, and the result is a super-slow function (not an endless loop).
In short there is nothing I can do for real because the characters can't be decoded.
I can only avoid the super-slow code.

aluigi, posted Thu Mar 09, 2017 10:31 pm (21332)

---

I don't remember who reported the following bug:

Code:

log MEMORY_FILE 0 0
set NAME string "XXXinputXXX"
strlen NAMELEN NAME
filexor 0x3a
putdstring NAME NAMELEN MEMORY_FILE
print "%NAME%"

where the content of the original variable gets modified (xored) after using putdstring.
Anyway I have decided to not fix it. filexor is a core component that works just in that way and this is a very rare event that never happens during normal use (filexor put* reusage of the variable).
This post is just to keep the problem logged.

aluigi, posted Fri Mar 10, 2017 11:43 am (21347)

---

This is the awesomeness you will see in quickbms 0.7.8

Code:

set MEMORY_FILE10 binary "
    int mycrypt(int a, int b) {
        return a ^ b;
    }
    int foo(char *data, int size, int key) {
        int     i;
        for(i = 0; i             data[i] = mycrypt(data[i], key);
        }
        return i;
    }
"

get SIZE asize
log MEMORY_FILE 0 SIZE
calldll MEMORY_FILE10 "foo" "tcc" RET MEMORY_FILE SIZE 0xff
log "dump.dat" 0 SIZE MEMORY_FILE

And this is the second awesomeness:

Code:

set MEMORY_FILE10 tcc "
    int mycrypt(int a, int b) {
        return a ^ b;
    }
    int foo(char *data, int size, int key) {
        int     i;
        for(i = 0; i             data[i] = mycrypt(data[i], key);
        }
        return i;
    }
"

get SIZE asize MEMORY_FILE10
log "dump.dat" 0 SIZE MEMORY_FILE10

This is possible thanks to the TinyCC compiler http://bellard.org/tcc/

Probably the second example "may" have a possible security impact because I'm not sure if it's worth to insert a warning for the user, the risk is related to the security bugs in the compiler. It's boring but I think I have to add the same warning used for CallDll.

aluigi, posted Fri Mar 10, 2017 6:10 pm (21362)

---

Unfortuantely I was wrong about the fixing of break/continue.
And I have almost finished the available time for this stuff, so it looks like everything will remain as-is.

aluigi, posted Fri Mar 10, 2017 7:10 pm (21366)

---

Good news and just in time: after some tests with some real scripts (asura.bms was very useful), few scripts created on the fly with nested if/for/break and a few provided by some users, it looks like the new code works correctly.
I have not much time for beta testing unfortunately so if someone wants to create more testing scripts with for/do/while/if/break/continue I will test them on the fly.
Even the experimental labels work.

aluigi, posted Sat Mar 11, 2017 10:44 am (21381)

---

And the following are the labels:

Code:

print "000"

test:
    print "AAA"

continue test2

print "BBB"

label test2     # "test2:" or "label test2" is the same
    print "CCC"

continue test

with the result:

Code:

000
AAA
CCC
AAA
CCC
AAA
CCC
AAA
CCC
AAA
CCC
AAA
CCC
AAA
CCC
AAA
CCC
...

Just a note: inside some cycles (for/do/while) may be necessary to use Break instead of Continue, anyway it's easy to notice this situation since you have an endless loop (in that case use the other command).

And yes, this feature is experimental, I don't officially support it, I don't use it, it costed me nothing to implement it, blah blah blah

aluigi, posted Sun Mar 12, 2017 2:24 pm (21422)

---

Ok, 0.8.0 is out.
This new version includes some changes to core components (like how the "if" statements are handled) so, even if I tested some scripts and conditions, it's impossible to excluded new bugs.
Please report any problem and negative difference you notice compared to the previous 0.7.7 version.
Thanks

aluigi, posted Sun Mar 12, 2017 6:48 pm (21434)

---

chrrox wrote:

the chunks are from the same archive.
quickbms decompressed 19gb from the file fine then failed on everything for the remaining 12gb starting at this sample here.
this is ps4 sample.
looks like it might be the 2.40 version they used.
Oodle 2.4.0 is up--New Hydra automatically selects Kraken/Mermaid/Selkie and new Mermaid compressor with slightly higher compression!

Can you check if quickbms 0.8.0 gives the same result?
It uses oodle 2.4.1

chrrox, posted Sun Mar 12, 2017 7:51 pm (21436)

---

The game is Horizon Zero Dawn.
Still get the same error on new quickbms .
I tried
COMP_OODLE,
COMP_OODLE_LZH,
COMP_OODLE_LZHLW,
COMP_OODLE_LZNIB,
COMP_OODLE_LZB16,
COMP_OODLE_LZBLW,
COMP_OODLE_LZNA,
COMP_OODLE_BitKnit,
COMP_OODLE_LZA,
COMP_OODLE_LZQ1,
COMP_OODLE_LZNIB2,
COMP_OODLE_Selkie,
COMP_OODLE_Akkorokamui,

Old chunks that extracted still extract.