aluigi, posted Mon Apr 05, 2021 9:37 pm (63240)

---

It's a bit difficult to implement it because pickle is a mix of various types of basic and complex serializers, that's why it's great as bms script but difficult to "embed" in quickbms (for various technical and design reasons).
pickle is also rarely used.
Isn't the script-library good enough in doing the job?

Shokoniraya, posted Mon Apr 05, 2021 10:00 pm (63241)

---

aluigi wrote:

It's a bit difficult to implement it because pickle is a mix of various types of basic and complex serializers, that's why it's great as bms script but difficult to "embed" in quickbms (for various technical and design reasons).
pickle is also rarely used.
Isn't the script-library good enough in doing the job?

i know that, and also it can't used in put or some other things
but useful to passing serialized bytes or reading blocks and reverse it back in reimport-mode
scripting is good enough, but a var type could be handy

Ekey, posted Tue Apr 06, 2021 8:33 pm (63255)

---

RAD Game Tools got bought by Epic Games, and they've made Oodle at least available on GitHub: https://www.unrealengine.com/en-US/blog ... via-github

Oodle source is included in three zip files, located in

Code:

Engine/Plugins/Compression/OodleData/Sdks/2.9.0/src/oodle2_src_for_unreal_data_2.9.0.zip
Engine/Plugins/Developer/TextureFormatOodle/Sdks/2.9.0/src/oodle2_src_for_unreal_texture_2.9.0.zip
Engine/Plugins/Compression/OodleNetwork/Sdks/2.9.0/src/oodle2_src_for_unreal_network_2.9.0.zip

maybe useful

aluigi, posted Tue Apr 06, 2021 11:01 pm (63256)

---

License is still a limitation, indeed the source code is not public.
But at least we will see many more oo2*.dll in the future games that we can embed in our tools.
The old oodle dll currently used in quickbms will still be mandatory since some algorithms are not compatible (or no longer available) in newer versions of oodle.
Definitely a good news btw

Shokoniraya, posted Fri Apr 09, 2021 8:34 pm (63301)

---

hello sir alugi, just want to say that append put* not working (tested on last release)

(same example by you)

Code:

set MEMORY_FILE binary "0123456789"
goto 4 MEMORY_FILE
append
putdstring "hello" 3 MEMORY_FILE
append
get SIZE asize MEMORY_FILE
log "output.dat" 0 SIZE MEMORY_FILE

aluigi, posted Sun Apr 11, 2021 10:40 am (63324)

---

It depends by what you want to do:
append: put it at the end
append 1: replace the data at offset 4
append 2: put it at offset 4 making space in the file (like reimport3)
append -1: at the beginning

Everything seems to work as intended

aluigi, posted Sun Apr 11, 2021 7:45 pm (63328)

---

A note for who uses the String command in quickbms 0.11 with the following operators (lower case): b, e, c.
I had to update some of my scripts because the way quickbms 0.11 works will return different results than 0.10.1.

It's a sort of bug because in theory it should be forced to use the length of the variable but instead it handles it as a NUL delimited string.
I guess I made the mistake while working on the String command and I used the length of the variable in a first moment and later switched to NUL delimited forgetting to fix b/c/e.
In short b/e/c give the same result of B/C/E and you must use the '0' prefix to make it work properly.

I will definitely fix it in the next version but who knows when it will ever be released.
Solution: String VAR1 0b VAR2

The solution is compatible with old versions of quickbms too

tbmq008, posted Tue Feb 01, 2022 5:56 pm (69547)

---

here is a feature request from me.
generate an entire tar file instead of extracting every file.
if quickbms can generate an entire ISO and zip file why not tar? all the files inside a tar are uncompressed, and a user may have to deal with millions of files (including duplicate ones, or files that look like such judging by their names or even files with weird names) anyway.

aluigi, posted Tue Feb 01, 2022 6:38 pm (69549)

---

tar is much more simple to implement than ISO and ZIP, it doesn't even have a header or footer.
It's so simple that it can be done just with some bms code around the log/clog command in the script used to dump the files.
(The only limitation would be with output filenames longer than 99 bytes since it requires more code for "././@LongLink")

I'm not sure how many users may need support for a tar output, I don't even know if anyone uses the ISO and ZIP outputs

Just curious, why did you think to tar?
Isn't ISO better due to the mounting on any operating system?
And why do you have millions of output files? That sounds exagerated.

tbmq008, posted Tue Feb 01, 2022 11:06 pm (69559)

---

aluigi wrote:

Just curious, why did you think to tar?
Isn't ISO better due to the mounting on any operating system?
And why do you have millions of output files? That sounds exagerated.

because it's easy to see whatever's in there in a hex editor, but also so i don't have to deal with compressed stuff (zip).
not sure about mounting though.
so i can do something with them? honestly i don't have that many files lying around but... yeah.

just saying it would be nice to have a "tar" feature.

aluigi, posted Tue Feb 01, 2022 11:40 pm (69561)

---

Yeah mounting works perfectly.
I just tried it now for confirming that it works correctly (for being 100% sure about the current version) by using zip.bms on the zip of quickbms source code with the -i option, I mounted the generated ISO and I had full access to all the "extracted" files.

Adding an additional output like tar would be like opening a pandora's box for me, because if I can add a new output format I would like to add N new formats maybe even with possible external non-builtin formats via user's dll or some definition/script code.

Currently adding only tar doesn't give any benefits to users compared to ISO and ZIP.

If you need to check a large amount of files with hex editor, one option may be to use a binary grep for searching a specific string (I wrote a tool called mygrep), or maybe a preview of the first bytes of each file (like "quickbms hexdump_scanner.bms folder"), or using the -O FILE option of quickbms that dumps all the files in one unique file called FILE.
Just some ideas.

aluigi, posted Wed Feb 02, 2022 2:42 pm (69576)

---

I just released a new beta of quickbms with experimental support for TAR when using the -O option, example:
quickbms -O output.tar script.bms archive.dat

There is no support for long filenames since it was just an experimental feature consisting of a few additional lines of code.
Basically I can easily add support to any format that uses metadata and sequential content.

Please remember that -O appends data to the existing output.tar because it's an option meant for concatenating the files even when launching quickbms on a folder with multiple input files.

This beta also fixes a bug with callfunction when using multidimensional arrays (VAR[x]), a bug when using encryption calldll with append mode (quickbms asked to overwrite the file for no reason), a bug in the size field of encryption incremental, plus other stuff fixed in the previous betas.

The link is ever the same of all the previous beta versions:
https://aluigi.altervista.org/beta/quickbms_exe.zip

spiritovod, posted Fri Feb 04, 2022 12:46 pm (69620)

---

aluigi wrote:

a bug when using encryption calldll with append mode (quickbms asked to overwrite the file for no reason)

Oh, so it was a bug after all. I wonder how you've found out that issue.

Anyway, could you please add some additional info about using RSA encryption to documentation. I failed to understand how it works in quickbms context. At least, I couldn't get it somehow working with keys in binary format (like from this topic) or PEM format (with public/private key headers) or converted versions of it (PEM format consists of modulus and exponent as last 4 symbols, if I got it right). I mean, quickbms simply crashed or produced zero output. Comparing to some known functions like openssl_public_decrypt or respective analogs from wincrypt / CryptoPP would be also appreciated.
Thanks in advance.

aluigi, posted Fri Feb 04, 2022 6:22 pm (69632)

---

I spotted that bug while fixing the size field of the "incremental xor" encryption, due to a bug in the size (32bit was wrongly interpreted) I had to rely on a tcc function, since only the first part of the file was encrypted I had to use "encryption calldll log encryption "" append log" and that's how I noticed something wrong

Regarding the private key encryption... it's a mess.
There are various different implementations and libraries so there is definitely a way in quickbms but I don't know how since it depends by what implementation you want to emulate.

I leave you some topics that may be useful:
viewtopic.php?t=13947
viewtopic.php?t=14660

This is a brief explanation of what API are called for each Encryption command that may be related to private key operations:

rsa tomcrypt:
only the key is used

rsa:
the key is given to PEM_read_bio_* for all the supported methods available in openssl (RSA_PUBKEY, RSAPublicKey, RSAPrivateKey, X509, SSL_SESSION and so on) plus BN_bin2bn (for 'n' a sort of raw public key encryption like modpow) if everything else fails
the ivec is only used with BN_bin2bn for the 'e' field so it's ignored in normal conditions.

modpow:
as above, key for 'n' and ivec for 'e'.
It's also possilble to set the 'zed' field of modpow if specified (example: Encryption modpow_zed)

PKCS5_PBKDF2_HMAC or PKCS5_PBKDF2_HMAC_SHA1 and so on:
it calls "PKCS5_PBKDF2_HMAC(key, keysz, ivec, ivecsz, cycles" where cycles is 1000 by default or any number you prefer if you set the encryption mode, example: Encryption PKCS5_PBKDF2_HMAC key ivec 1234

Rfc2898DeriveBytes:
the first part works just like PKCS5_PBKDF2_HMAC plus it performs an AES 256 CBC encryption.
It uses sha1 by default like for the previous one, if you need other algorithms try: Encryption Rfc2898DeriveBytes_MD5 ...

BytesToKey:
Example:
Encryption BytesToKey KEY IVEC 1000
Encryption "BytesToKey aes_256_cbc sha1" KEY IVEC 1234
It's a raw call to EVP_BytesToKey using aes 256 cbc, sha1, and 1000 cycles by default

PBKDF1 / PBKDF2
It's a wrapper for tomcrypt, also in this case KEY, IVEC, CYCLES.
It's also possible to specify the hashing algorithm and if we want to use openssl for PBKDF1:
Encryption "PBKDF1 SHA1 openssl" KEY IVEC 1000


It's a mess, yeah I fully agree!
Tons of libraries, different ways for doing the same thing in other API and libraries.

spiritovod, posted Fri Feb 04, 2022 8:27 pm (69642)

---

Sorry, it seems it's not quite clear about what I'm doing. Please open an online RSA decryption/encryption and key generator tool and try to decrypt any random data with any given valid public key (or private key, not sure). I suppose RSA is data-independent, so you should be able to somehow "transform" any data as long as you're using valid key format. Also you can take a look at the topic, mentioned in my previous post - binary key there looks like ASN.1 type of RSAPublicKey.
The problem is that I couldn't use or reproduce any of this in quickbms, or at least get any result from encryption rsa/rsa_tomcrypt, except for crashes or zero output.

aluigi, posted Sat Feb 05, 2022 12:06 pm (69653)

---

It's openssl that refuses to load the key, there is nothing I can do.

The code for loading the key is simple and it's also confirmed on all the examples on the Internet:

Code:

keybio = BIO_new_mem_buf(key, keysz);
if(keybio) {
    rsa = PEM_read_bio_##X(keybio, NULL, NULL, NULL);

where ##X is a cycle of the following methods:

Code:

DHparams
DSAPrivateKey
DSA_PUBKEY
DSAparams
ECPKParameters
ECPrivateKey
EC_PUBKEY
NETSCAPE_CERT_SEQUENCE
PKCS7
PKCS8
PKCS8_PRIV_KEY_INFO
PUBKEY
PrivateKey
RSAPrivateKey
RSAPublicKey
RSA_PUBKEY
SSL_SESSION
X509
X509_AUX
X509_CERT_PAIR
X509_CRL
X509_REQ

Doesn't matter if the key is binary, base64, between "-----BEGIN ... -----END...", everytime that function returns NULL.
I even tried passing rsa as second argument but it's still the same.

*edit* even replacing BIO_new_mem_buf with BIO_new(BIO_s_mem()) BIO_write doesn't work (just checked since the former may need a const key valid till the end of the decryption, which is not this case btw)

If you find a way to load the key with openssl I will immediately implement it.

aluigi, posted Sat Feb 05, 2022 12:53 pm (69654)

---

Ok it works.
It's openssl which is a disaster in handling the input key.

Long story short openssl wants the key in base64 encapsulated between "-----BEGIN PUBLIC KEY-----" and "-----END PUBLIC KEY-----".
ADDITIONALLY it also wants the key to be truncated at the 64th character so you can't copy&paste a long key but you must ensure that each line is not "too long".

So if you use an RSA encryptor online you can decrypt it with quickbms.
I will work on the beta for making all those "manual" changes automatically made by quickbms.

spiritovod, posted Sat Feb 05, 2022 1:16 pm (69655)

---

Thank you for looking into the issue. I've tried before with passing "set KEY string "public key with proper tabs and header" " as key, but it didn't work, that's why I've asked about particular format. Do input should be base64 encoded as well for "encryption rsa KEY" to work as expected in this case?

aluigi, posted Sat Feb 05, 2022 2:50 pm (69663)

---

KEY:

Code:

-----BEGIN PUBLIC KEY-----
line1_64_chars
line2_64_chars
line3_64_chars
...
lineN_64_chars
-----END PUBLIC KEY-----

Input file must be binary.

Example script:

Code:

set KEY string "
-----BEGIN PUBLIC KEY-----
...
-----END PUBLIC KEY-----
"

get SIZE asize
encryption "rsa" KEY
log "output.dat" 0 SIZE

Probably you will notice that the output has the same size of the input, that's because quickbms ignores any returned size from the encryption commands since they gave some problems in the past with some algorithms.

aluigi, posted Sat Feb 05, 2022 5:48 pm (69676)

---

Since you linked that topic of atom0s I made a script for using that key in quickbms:

Quote:

set KEY string "
30 81 89 02 81 81 00 E2 72 5E F9 BB 16 88 71 C2 38 D9 1B 64 CF B8 B1 33 2F 1B BC F1 05 F4 0F 25 2F B9 3F 3A 60 9D 52 4C F8 F5 EE 09 BC 55 4F D9 18 DB 8B B3 53 1D 6F 88 BE FE A4 BF BD F5 1C B1 E1 DF 5E 5D FA 83 FD 65 84 D3 7E 27 99 24 22 4F C4 F8 BB 6C 98 ED 50 D2 70 02 E8 BA 21 F3 5F 01 55 A0 8D 9E D2 76 71 40 32 AE EC DA 06 6C 17 FA 54 F1 C3 3E 5D AF 8B 33 2B 3C C0 77 14 90 A1 52 61 B2 DD 90 8F 53 F1 02 03 01 00 01
"
string KEY h KEY
comtype base64_compress
string TMP_KEY 0c KEY

strlen TMP_KEYSZ TMP_KEY
set KEY ""
math KEYSZ = 0
for x = 0 < TMP_KEYSZ
getvarchr TMP TMP_KEY x
if x % 64
elif x != 0
putvarchr KEY KEYSZ '\n'
math KEYSZ 1
endif
putvarchr KEY KEYSZ TMP
math KEYSZ 1
next x

# the following don't work?
#string KEY p "%s\n%s\n%s\n" "-----BEGIN PUBLIC KEY-----" KEY "-----END PUBLIC KEY-----"
#string KEY P "-----BEGIN PUBLIC KEY-----\n%KEY%\n-----END PUBLIC KEY-----\n"

string KEY p "%s%c%s%c%s%c" "-----BEGIN PUBLIC KEY-----" '\n' KEY '\n' "-----END PUBLIC KEY-----" '\n'

print "%KEY%"

get SIZE asize
encryption "rsa" KEY
log "output.dat" 0 SIZE

Now my doubt is about those two commented "string KEY p/P" because I don't understand why the \n escape doesn't work.
I will check it later to know if it's something that needs to be fixed.
The Print command handles the escape, while String P doesn't.

*update*
Regarding String VAR p the problem is that the format argument is given as-is to sprintf without handling the escapes, I can fix it but I hope it will not make some scripts incompatible, for example: string NAME p "%s\%s PATH NAME
I ever use / so I'm worried about the scripts of other users.

spiritovod, posted Sat Feb 05, 2022 6:16 pm (69677)

---

Great, thanks for the script, it will allow to handle binary rsa keys as well. Alternatively, you can also convert them manually with any hex->base64 converter, add header and tabulate string to 64 char parts. Also a small note to myself, this is block encryption and data should be aligned to 16 bytes and key size in general, otherwise function will fail.

Update: Nice, after undestanding general key format and block thing with aligning, I managed to decrypt actual data with private key by manually converting it like explained above and using it on properly managed blocks. For private keys "BEGIN/END RSA PRIVATE KEY" header should be used. Thanks again for the help.

Update 2: Though I'm curious why do you need an empty string at the beginning and at the end of "string" variable. It's the same with C code for tcc, if there is no empty string at the start, code compilation will fail.

aluigi, posted Sat Feb 05, 2022 8:05 pm (69679)

---

I guess there was a discussion years ago about the "format" of the multi-line strings but no idea when and where it was.

Instead regarding that supposed bug in "String p/P" I think I reached a compromise:

- "p": if I use printf I expect that the format string contains escaped characters like in programming, but at the same time in quickbms that argument is a string... so there is no good or wrong.
A work-around may be to handle it as an escaped string if it's used directly in the command and as a string if it's a variable, example which should have the same result:
escaped: string VAR p "%s\n%s\n" VAR1 VAR2
normal string: set FMT binary "%s\n%s\n" ; string VAR p FMT VAR1 VAR2

- "P": same as above, when I use it I expect that the output of String P is identical to Print, also in this case the work-around would be like the same.

I have already implemented these work-arounds in the current beta on the website (https://aluigi.altervista.org/beta/quickbms_exe.zip) and it finally works as expected.
I look forward for any comments about it, hopefully it would have limited impact on existing scripts except for those using \ for example as path delimiters in building filenames.

The beta also contains all the automated code for building a valid key for openssl, no matters if the BEGIN/END ar missing or if it's directly the binary code because quickbms will do the job itself.

Additionally there is also a new feature that I was testing: ImpType.
This was a command of the original MexScript that was totally useless in quickbms due to the automatic work of the reimport mode.
With the new modes of ImpType is now possible to write an updated variable in the original field in any reimport mode.
From quickbms.txt:

Code:

        imptype var ORIGINAL_VAR NEW_VALUE
            put NEW_VALUE at the location of ORIGINAL_VAR field

        imptype var ORIGINAL_VAR
            update the field where ORIGINAL_VAR was read with its
            current value

        imptype off -16 NEW_VALUE
            put NEW_VALUE at the offset 16 bytes at the end of the
            file, its size is the same of the field that has been
            read at that location or a 32bit LONG if never read

        imptype off 16 NEW_VALUE
            as above but it's at offset 16

        imptype crc ORIGINAL_VAR
            replace the field of ORIGINAL_VAR with the current
            content of QUICKBMS_CRC or QUICKBMS_HASH, it requires
            the usage of Encryption with the relative crc/hash
            algorithm used in the *Log operation

Obviously I'm open to comments and improvements since this is just a test I tried and it worked well (needs more testings obviously).
I think it has a big potential for finally solving the problems of the crc and hash fields used in some archives.

*edit* I fixed the examples of imptype above because the "mode" was missing

aluigi, posted Tue Feb 08, 2022 12:39 pm (69744)

---

Another new beta is out.

I fully implemented the ImpType CRC method:

Code:

        imptype crc ORIGINAL_VAR
            replace the field of ORIGINAL_VAR with the current
            content of QUICKBMS_CRC or QUICKBMS_HASH, it requires
            the usage of Encryption with the relative crc/hash
            algorithm used in the *Log operation
            this command must be used AFTER *Log

        imptype crc ORIGINAL_VAR ALGO ...
            replace the field of ORIGINAL_VAR with the crc or hash
            calculated during a *Log operation and based on the
            chosen ALGO and its parameters, SAME as Encryption.
            this command must be used BEFORE *Log.
            the crc/hash is calculated on the data before
            encryption and before compression
            Example:
                get MYCRC long          # read the field
                imptype crc MYCRC md5 ""
                comtype zlib
                encryption xor "mykey"  # compatible with imptype
                clog NAME OFFSET ZSIZE SIZE
                imptype crc ""          # optional disable

I did some tests and it worked very well.
I also fixed few other things and I had to make some changes to core functions like perform_encryption for fully implementing ImpType CRC, so I expect some bugs.

With this feature it's now possible to bypass all the limitations caused by crc and hash in archives but we need more tests.

aluigi, posted Thu Feb 10, 2022 1:15 pm (69791)

---

New beta and new feature plus various bug fixes.

The new feature is related to calldll tcc and the file I/O, since it wasn't supported I replaced it with memory I/O.
Long story short any call to API like fgetc/fputc/fread/fwrite will use the input and output provided in calldll, it's enough to pass the argument as &#INPUT# and &#OUTPUT#.

Example:

Code:

set MEMORY_FILE10 string "

int func1(FILE *fd) {
    int     i, c;
    for(i = 0;; i  ) {
        c = fgetc(fd);
        if(c < 0) break;
    }
    return i;
}

int func2(FILE *fd, FILE *fdo) {
    int     i, c;
    for(i = 0;; i  ) {
        c = fgetc(fd);
        if(c < 0) break;
        c ^= ' ';
        fputc(c, fdo);
    }
    return i;
}
"

get SIZE asize

encryption calldll "MEMORY_FILE10 func1 tcc RET &#INPUT#"
log MEMORY_FILE 0 SIZE
encryption "" ""
print "%RET%"

comtype calldll "MEMORY_FILE10 func2 tcc RET &#INPUT# &#OUTPUT#"
clog MEMORY_FILE 0 SIZE SIZE
print "%RET%"

And yes, now any MEMORY_FILE, #INPUT#, #INPUT_SIZE#, #OUTPUT#, #OUTPUT_SIZE# can be passed as pointers which may be useful in some rare situations.

Ekey, posted Sun Apr 10, 2022 1:04 pm (71011)

---

https://github.com/widberg/fmtk/wiki/As ... ompression