Shokoniraya, posted Sun Jan 31, 2021 4:30 pm (61947)

---

sir alugi, i have a idea for next version of QuickBMS
can you make a repdstring command to replace a number of bytes in file? at first look, it's not useful and importing will does the job, but this command is really helpful in MEMORY_FILE file if we want to replace a few bytes in MEMORY_FILE
repdstring (replace dstring), just like reimport3, for increasing MEMORY_FILE without creating other MEMORY_FILE

for example:

Code:

repdstring MY_STRING 0 MEMORY_FILE2

aluigi, posted Sun Feb 07, 2021 10:16 pm (62063)

---

Are you talking about "putdstring"?

aluigi, posted Fri Feb 12, 2021 6:34 pm (62152)

---

A new beta is available, mainly focused on compatibility with non-Windows platforms and other compilers (clang) and versions of gcc.
For example now it compiles natively on Termux for Android and supports oodle on Linux x86.

I don't have other things to fix or features to add.

My plan is to find some time and desire for answering all the topics in the Game Archive section because it forces me to test quickbms on many file formats and spotting possible errors.
After that I will release the new version.

Shokoniraya, posted Sat Feb 13, 2021 2:42 am (62160)

---

im talking a about a command that can increase/inlarge MEMORY_FILEs by replacing a binary variable (for example, a new command with name: repdstring) that can works on both MEMORY_FILE and real files

for example, MEMOEY_FILE1 is "\x0A\x0B\x0C\x0D\x0E"
and we are going to replace some bytes,
MY_BINARY is "\x0F"

Code:

goto 0 MEMOEY_FILE1
repdstring MY_BINARY 3 MEMOEY_FILE1

now, result of MEMOEY_FILE1 is "\x0F\x0D\x0E"
that means we replaced first three byte of MEMOEY_FILE1 with MY_BINARY

it's possible to do this by creating another MEMORY_FILE and append command
but it will saves alot of time and it's faster

and about putdstring, something that really needed is putdstring MY_BINARY ""
set "" instead of a number should act like default size of variable and no need to set length of variable

aluigi, posted Sun Feb 14, 2021 10:42 am (62196)

---

I had in mind to change the behavior of the put* commands if append is enabled.
For example just like with a hex/text editor where the current mode is "overwrite" and in append mode it will become "insert".

Example:

Code:

set MEMORY_FILE binary "0123456789"
goto 4 MEMORY_FILE
append
putdstring "hello" 3 MEMORY_FILE
append
get SIZE asize MEMORY_FILE
log "output.dat" 0 SIZE MEMORY_FILE

I will work on it soon (also because I found a bug with set MEMORY_FILE string in the beta...)

Shokoniraya, posted Sun Feb 14, 2021 4:42 pm (62205)

---

sir aluigi, is it possible to implemente single line mode in bms files?

Code:

print "message_1" ; print "message_2"

Shokoniraya, posted Sun Feb 14, 2021 6:29 pm (62208)

---

i think QuickBMS can be more optimizer and faster

i did a test for file exporting speed in my system

dump_1.bat (normal dumping) = 24 sec
dump_2.bat (with reserved memory, i mean memory file) = 28 sec

also, there was a lot of operation in dump_2.bms, but there is no any of them in dump_1.bms, i mean storing a few byte in memory and then dumping them at once is faster and better


for example, i remember some ps2 games (talking about old days), there was some games that no loading page included in them or loading was very very faster than other games, such as god of war or ben 10 the protector of earth
data files was readed 2048 by 2048 (or 4096 by 4096?)
maybe i am wrong just about theory of this (not about mechanism and result), but most of games or programs that has padded size or padded offset are fast

maybe QuickBMS can have such option as a new feature

dump_test.7z

aluigi, posted Sun Feb 14, 2021 11:40 pm (62219)

---

The new beta fully implements the Append Put* idea.

The one-line scripts work only with the -s option.
I can't support it in the script file.

dump_test.7z is password protected.
Anyway I don't think I can improve much the performance of quickbms without rewriting most of its core (which I'm not going to do)

Shokoniraya, posted Mon Feb 15, 2021 6:16 am (62228)

---

sorry, password removed. you can check it now

aluigi, posted Mon Feb 15, 2021 12:59 pm (62238)

---

Interesting, the problem affects quickbms_4gb_files only because quickbms.exe takes just 5 seconds.
This is going to be a nightmare to figure out...
Good finding btw, it needs to be fixed for sure.

aluigi, posted Mon Feb 15, 2021 4:45 pm (62249)

---

I think it was only related to disk caching because now the time is the same.
So no problem I suppose.

chrrox, posted Mon Feb 15, 2021 6:28 pm (62262)

---

can you expose PKCS #5 from tomcrypt_pkcs.h

aluigi, posted Mon Feb 15, 2021 11:54 pm (62271)

---

If only I would know how to implement it in the Encryption command
I think there is no problem to add pkcs_5_alg* but that pkcs_1* is a mess.

I still have to check the other old topic you opened, let me know what else I can add and I will check it

tbmq008, posted Fri Feb 26, 2021 2:22 pm (62504)

---

first of all, i would like to know if there is a similar feature for the getdstring feature except for when it accepts various character encodings other than just ASCII.
i've stumbled upon a unreal pak file that uses various utf16 strings and there are "name size" values placed before them.

if there isn't, what kind of workaround do i have to implement to get around something of that sort.

aluigi, posted Sat Feb 27, 2021 10:05 am (62523)

---

I guess you refer to the codepage, if it's a japanese game it'e enough to use the "codepage 932" command at the beginning of the script (or as command-line option -P 932).

tbmq008, posted Sat Feb 27, 2021 12:35 pm (62525)

---

OK

all i want is for quickbms to cover up all available utf16 chars, not by invoking this "get NAME unicode" command, but by something like this

Code:

codepage 1200 # utf16
get name_size long
getdstring base_path name_size

and yet, for whatever reason, i get results like this

Code:

. 0000000000000058 getdstr base_path  "." 10
    2e 00 2e 00 2f 00 2e 00 2e 00                     ..../.....

i should be getting something like this

Code:

. 0000000000000058 getdstr base_path  "../../../" 20
    2e 00 2e 00 2f 00 2e 00 2e 00 2f 00 2e 00 2e 00   ..../...../.....
    2f 00 00 00                                       /...

aluigi, posted Sat Feb 27, 2021 12:50 pm (62526)

---

math name_size * 2

tbmq008, posted Sat Feb 27, 2021 1:10 pm (62528)

---

yeah... that
i added that line of code before the getdstring line and still, this is what i get

Code:

. 0000000000000058 getdstr base_path  "." 20
    2e 00 2e 00 2f 00 2e 00 2e 00 2f 00 2e 00 2e 00   ..../...../.....
    2f 00 00 00        

aluigi, posted Sat Feb 27, 2021 1:39 pm (62529)

---

That's correct.
I don't think I understand what other output you need, read the raw data and convert it with "set base_path unicode base_path".
That's how it works and in fact it perfectly works in my UE4 script.

tbmq008, posted Sat Feb 27, 2021 1:40 pm (62530)

---

yeah, i got this.

chrrox, posted Wed Mar 24, 2021 5:37 pm (63010)

---

add these hash functions?
I did not see them listed in quickbms txt.
https://sites.google.com/site/murmurhash/
https://github.com/PeterScott/murmur3

aluigi, posted Wed Mar 24, 2021 7:50 pm (63012)

---

the beta already has them.
if they are not listed in Encryption hash, take a look at crc.c:

Code:

         if(ctx->type == 0)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ (MYCRC >> 8))
    else if(ctx->type == 1)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ (MYCRC >> (ctx->bits - 8))) & 0xff] ^ (MYCRC << 8))
    else if(ctx->type == 2)  CRC_CALC_CYCLE(    ((MYCRC << 8) | MYBYTE) ^ ctx->table[(MYCRC >> (ctx->bits - 8)) & 0xff])
    else if(ctx->type == 3)  CRC_CALC_CYCLE(    ((MYCRC >> 1)   ((MYCRC & 1) << (ctx->bits - 1)))   MYBYTE)
    else if(ctx->type == 4)  crc =              crc_in_cksum(MYCRC, data, datalen);
    else if(ctx->type == 5)  CRC_CALC_CYCLE(    MYCRC ^ MYBYTE)
    else if(ctx->type == 6)  CRC_CALC_CYCLE(    MYCRC   MYBYTE)    // lose lose
    else if(ctx->type == 7)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ MYCRC)
    else if(ctx->type == 8)  CRC_CALC_CYCLE(    ctx->table[(MYBYTE ^ MYCRC) & 0xff] ^ (MYCRC >> (ctx->bits - 8)))
    else if(ctx->type == 9)  CRC_CALC_CYCLE(    (MYCRC << 1)  ^ MYBYTE)
    else if(ctx->type == 10) CRC_CALC_CYCLE(    (MYCRC << 1)    MYBYTE)
    else if(ctx->type == 11) CRC_CALC_CYCLE(    rol(MYCRC, 1, 0) ^ MYBYTE)
    else if(ctx->type == 12) CRC_CALC_CYCLE(    rol(MYCRC, 1, 0)   MYBYTE)
    else if(ctx->type == 13) CRC_CALC_CYCLE(    ror(MYCRC, 1, 0) ^ MYBYTE)
    else if(ctx->type == 14) CRC_CALC_CYCLE(    ror(MYCRC, 1, 0)   MYBYTE)
    else if(ctx->type == 15) CRC_CALC_CYCLE(    (MYCRC << 5)   MYCRC   MYBYTE) // djb2 5381
    else if(ctx->type == 16) CRC_CALC_CYCLE(    (MYCRC * ctx->poly)   MYBYTE) // djb2 and sdbm
    else if(ctx->type == 17) CRC_CALC_CYCLE(    (MYCRC * ctx->poly) ^ MYBYTE) // djb2 and FNV-1
    else if(ctx->type == 18) CRC_CALC_CYCLE(    (MYCRC ^ MYBYTE) * ctx->poly) // FNV-1a
    else if(ctx->type == 19) CRC_CALC_CYCLE(    MYBYTE   (MYCRC << 6)   (MYCRC << 16) - MYCRC) // sdbm 65599
    else if(ctx->type == 20) CRC_CALC_CYCLE(    ctx->poly * (MYCRC   MYBYTE * (i   1)))
    else if(ctx->type == 21) crc =              qhashmurmur3_32(data, datalen);
    else if(ctx->type == 22) crc =              qhashfnv1_32(data, datalen);
    else if(ctx->type == 23) crc =              qhashfnv1_64(data, datalen);
    else if(ctx->type == 24) crc =              XXH32(data, datalen, ctx->poly);
    else if(ctx->type == 25) crc =              XXH64(data, datalen, ctx->poly);
    else if(ctx->type == 26) crc =              jenkins_one_at_a_time_hash(data, datalen);
    else if(ctx->type == 27) crc =              xPear16(data, datalen);
    else if(ctx->type == 28) crc =              CityHash32(data, datalen);
    else if(ctx->type == 29) crc =              CityHash64(data, datalen);
    else if(ctx->type == 30) crc =              CityHash64WithSeed(data, datalen, ctx->poly);
    else if(ctx->type == 31) crc =              StormHash(data, datalen, MPQ_HASH_TABLE_INDEX);
    else if(ctx->type == 32) crc =              StormHash(data, datalen, MPQ_HASH_NAME_A);
    else if(ctx->type == 33) crc =              StormHash(data, datalen, MPQ_HASH_NAME_B);
    else if(ctx->type == 34) crc =              StormHash(data, datalen, MPQ_HASH_FILE_KEY);
    else if(ctx->type == 35) crc =              StormHash(data, datalen, MPQ_HASH_KEY2_MIX);
    else if(ctx->type == 36) crc =              jenkins_hashlittle(data, datalen, ctx->poly);
    else if(ctx->type == 37) crc =              adler32(0, data, datalen);
    else if(ctx->type == 38) crc =              fnv32(data, datalen, crc ? crc : 0x811c9dc5);
    else if(ctx->type == 39) crc =              UHash(data, datalen, crc, ctx->poly, 0x7fffffff);
    else if(ctx->type == 40) crc =              spookyhash_32(data, datalen, ctx->poly);
    else if(ctx->type == 41) crc =              spookyhash_64(data, datalen, ctx->poly);
    else if(ctx->type == 42) crc =              XXH3_64bits(data, datalen);
    else if(ctx->type == 43) crc =              XXH3_64bits_withSeed(data, datalen, ctx->poly);

aluigi, posted Mon Apr 05, 2021 12:56 pm (63229)

---

QuickBMS 0.11 is finally out
http://quickbms.com

shekofte, posted Mon Apr 05, 2021 1:00 pm (63230)

---

aluigi wrote:

QuickBMS 0.11 is finally out

It is gold

Shokoniraya, posted Mon Apr 05, 2021 8:19 pm (63238)

---

really out! thank you, sir aluigi!
i wish many good times for you, sir aluigi

can i give you a suggestion? can you do something about pickle as a data-type?
for example:

Code:

get MY_VAR pickle

i know it's stupid and blineded for users as a var type (based on what i found in pickle.bms), but useful

maybe a guessvar needed for that

Code:

get MY_VAR pickle
guessvar MY_VAR_TYPE MY_VAR
print "%MY_VAR_TYP%"