Shokoniraya, posted Sat Sep 19, 2020 4:46 am (58765)

---

thank you, but still i get this error in with a input file that has unicode name in batch file

- error in src\file.c line 602: fdnum_open()
Error: Invalid argument

exmaple file: download/file.php?id=9198

aluigi, posted Sat Sep 19, 2020 8:22 am (58767)

---

My fault, I didn't move the beta zip to the folder ...
Now it's fixed.

Shokoniraya, posted Sun Sep 20, 2020 12:35 am (58774)

---

thank you, works fine

aluigi, posted Tue Sep 22, 2020 8:41 pm (58821)

---

I have updated the beta with the latest version which I hope is also the one that will be released on the website.
It's already compiled with optimizations and with both the executables just like a normal release.
It includes the full source code, the changelog and also a revisioned quickbms.txt (on which I may work more).

Please test it as much as you can and tell me if you spot any bug or have any questions.

The link is the same (let's use the direct https one btw, copy the URL in the browser if you want to skip the redirect page) and the size is 30Mb:
https://aluigi.altervista.org/beta/quickbms_beta.zip

I plan to release quickbms 0.11 at the beginning of October.

Shokoniraya, posted Wed Sep 23, 2020 9:05 am (58830)

---

it's really better to set names for MEMORY_FILE too

Code:

set MEMORY_FILE_sewew string "1"
set MEMORY_FILE_wcfow string "2"

log memory 0 1 MEMORY_FILE_wcfow

and this one too

Code:

set MEMORY_FILE1 string "1"
set MEMORY_FILE2 string "2"

math COUNT = 2
string MEMORY_SLOT p "MEMORY_FILE%s" COUNT
print "%MEMORY_SLOT%"
log memory 0 1 MEMORY_SLOT
#or even log memory 0 1 MEMORY_FILE%COUNT%

Shokoniraya, posted Thu Sep 24, 2020 5:58 am (58851)

---

a problem in downloading file with -n option

most of links using / (slash) for passing directories
but if i use \ (back-slash), url not working and give me (404) as a remote error

exmaple
woring: http://web.com/image.png
not working: http://web.com\image.png

aluigi, posted Sat Sep 26, 2020 7:33 am (58878)

---

That thing of the memory files you suggested is complicated and without any real advantage.

URI/URL must use / and not \, that's part of the HTTP protocol.

Shokoniraya, posted Sat Sep 26, 2020 8:52 am (58882)

---

anyway, it's really helpful to have such option in QuickBMS
(sorry, i wanted to change my previous comment but i write a clear one)
without any real advantage? names are more explicit than memory numbers

Code:

set MEMORY_FILE1 string "1"
set MEMORY_FILE2 string "2"

math COUNT = 2
string MEMORY_SLOT p "MEMORY_FILE%s" COUNT
print "%MEMORY_SLOT%"
log memory_test1 0 1 MEMORY_SLOT #MEMORY_SLOT is MEMORY_FILE2, but it just acting as 0 file number, like this:(log memory_test1 0 1 0)

log memory_test2 0 1 MEMORY_FILE%COUNT% #not working, result must be 2, but output is 1, %COUNT% not working as MEMORY_FILE file number at all

and about names for MEMORY_FILE
result should be "take", but it does not using names for MEMORY_FILE
that means it's acting as unnumbered MEMORY_FILE (it's just a suggestion, using names for MEMORY_FILE could be useful)

Code:

set MEMORY_FILE_take string "take"
set MEMORY_FILE_last string "last"

log memory_name 0 4 MEMORY_FILE_take #result should be "take"

ExtractResponseUnit, posted Wed Sep 30, 2020 2:42 am (58960)

---

Thought about File Ripper

I catched up a few lines about to add such feature in QuickBMS. It's time expansive for sure and the solution would be a community-based
task. Unfortunately I'm not blessed with coding skills so I offer support in alternative ways. The wheel was already invented so I think it
would be less time consuming if you could implement a existing open source based file ripper and customize it instead making a neat one.

aluigi, posted Wed Sep 30, 2020 9:34 pm (58969)

---

@Shokoniraya
It would be difficult to implement that thing because all the files are just numbers in the bms language.
And the benefits are basically none.
You can just add a comment after each MEMORY_FILE to know what's meant for.

@ExtractResponseUnit
That would be nice if it's a separate tool than quickbms because it needs a different logic.
File rippers are difficult because sometimes there are no ways to calculate the size of the file, and scanning the magic values after the found one would produce many false positive.
I stay away from writing rippers

Instead, regarding the binary String feature implemented in the beta, I just noticed that it's not compatible with those situation where we have something like:

Code:

getdstring PATH 0x20
getdstring NAME 0x20
string PATH   NAME

The result is just the original PATH and not PATHNAME because in binary mode the whole variables are concatenated, so the result is 0x20 bytes of PATH followed by 0x20 bytes of NAME.
Not sure what solution I'm going to implement but probably a parameter specified by the user for setting the binary mode.
Currently String has '0' as special prefix for the operator, what about using it also for the binary mode?
Like: string PATH 0 NAME
Obviously the mode using NUL-delimited strings will remain the default as it is now.

ExtractResponseUnit, posted Wed Sep 30, 2020 10:24 pm (58970)

---

aluigi wrote:

@ExtractResponseUnit
That would be nice if it's a separate tool than quickbms because it needs a different logic.
File rippers are difficult because sometimes there are no ways to calculate the size of the file, and scanning the magic values after the found one would produce many false positive.
I stay away from writing rippers

I don't know how to compile pascal source code but I think someone on this forum does so here we go . . .

chrrox, posted Fri Oct 02, 2020 1:29 am (58982)

---

Does quickbms support these hash functions?
they are used in unity.
https://github.com/facebook/folly/tree/ ... folly/hash

referenced here
https://github.com/Unity-Technologies/U ... okyHash.cs

aluigi, posted Fri Oct 02, 2020 3:10 pm (58989)

---

Ok I just added spookyhash, murmurhash and xxhash in Encryption (not available in the online build yet).
Some of them are also available in the crc engine

Shokoniraya, posted Sat Oct 03, 2020 12:31 am (58996)

---

sir aluigi, i found out that current beta version of QuickBMS has a problem

Code:

log MEMORY_FILE6 0 0
#or even set MEMORY_FILE6 binary ""
put MY_VAR long MEMORY_FILE6

but it says: Error: MEMORY_FILE6 has not been used/declared yet
can't make a empty MEMORY_FILE?
there is no problem in 0.10.1 version

and one other problem, i've attached a bms file in this comment, QuickBMS can't download this file, i can download it from any downloader (it's a direct link), can you check it?
thank you

link_bms.zip

aluigi, posted Sun Oct 04, 2020 12:08 am (59016)

---

Good catch on that MEMORY_FILE problem, I'm checking it right now.

I tried passing download/file.php?id=9258 to quickbms 0.11 as input file and it correctly downloads it, are you talking to some different problem?

Shokoniraya, posted Sun Oct 04, 2020 12:16 am (59017)

---

yes, i'm takling about some direct links that can't download it with QuickBMS

please download attached file in my previous comment (link_bms.zip) and open zip file, then use test_link.bms
it's a bms file to download a direct link, but can't download it with QuickBMS by using that script

i'm talking about test_link.bms file (bms open "." ""), not attached zip link

aluigi, posted Sun Oct 04, 2020 5:57 am (59022)

---

Just checked and the behavrious of quickbms is correct.
In short your link is a huge file of 2147483648 bytes, so quickbms tries to allocate such amount in memory for downloading it and fails.

The maximum amount you can allocate is 2147221471 bytes (Content-Length field) which is crazy because quickbms is NOT a downloader.
It will not tell you if the file exists but it will first download it completely.

aluigi, posted Sun Oct 04, 2020 5:55 pm (59032)

---

I just updated the beta on the website.
I did some changes to the multidimensional arrays (VAR[i] and VAR[i][j]) so please check them too.

Shokoniraya, posted Sun Oct 04, 2020 11:25 pm (59040)

---

about huge files, what about 4gb version? has same problem since it should handle unit

i tested array in some different ways, looks like works fine, can you tell some detail about those changes?

aluigi, posted Mon Oct 05, 2020 5:20 pm (59049)

---

Not sure if 4gb bypasses it but for sure you should not use quickbms for downloading 2Gb files

Regarding the changes, they are mainly programming stuff without additional feature.

One change was that now VAR[] is considered a stand-alone variable itself while before it was a multidimensional array with a "" index that basically is just the same as now.
My plan is that maybe I can use it as a sort of way to receive the total number of entries or just leave it as-is.

I also moved a function that allocates memory for the arrays (add_multi_dimensional) because it was leaking lot of RAM for nothing.
Basically just this but I'm sure it may be possible to optimize it even more.

Shokoniraya, posted Mon Oct 05, 2020 11:33 pm (59055)

---

i think you should leave it be same as is (sorry for my limit grammar)
and just a question, can you check if it's possible to optimize read/write function, making it faster?

and one more thing, some of files are read-only, i know it can simply disabled, but can you do something about Error: Permission denied? (passing read-only, i mean if input file was read-only, then quickbms disable read-only for that file)

aluigi, posted Tue Oct 06, 2020 6:34 pm (59070)

---

What you mean with making faster read/write?
Do you have an example?

Good idea the "read-only to write" thing, going to add it soon.

aluigi, posted Tue Oct 06, 2020 7:13 pm (59073)

---

Ah that permission denied is NOT related to read-only files.
It's related to the permissions of the folder, long story short I will never touch that.
But I did the read-only thing when using -w, in that case quickbms showed the message: Do you want to create it from scratch (y/N)?

Shokoniraya, posted Tue Oct 06, 2020 11:01 pm (59077)

---

sir aluigi, i found a problem in savepos, i know it can be done with a little more coding in bms file. but can you check it? i think savepos has problem in asize (reimport mode)

i've attached a zip file in this comment
there is a savepos.bms in zip file, put quickbms.exe in savepos_problem folder, and goto file_content folder and open a file with name sec_3 in hex editor, and add a few byte (or remove a few byte) from that file
and click on _import.bat, and you will see that END_OFFSET (in bms file) not changed in reimporting (asize problem in savepos)

also, there is same problem with xmath and math too, becuase they will not change in reimporting

can you check other commands that related to such cases too?

savepos_problem.zip

aluigi, posted Wed Oct 07, 2020 12:57 am (59081)

---

There was for sure a problem with asize because I fixed it few hours ago while checking another topic in which clog added a 32bit field into the compressed data (but I don't think it's the problem you refer).

Regarding reimport3 and savepos... it's a mess.
quickbms already resets the variables containing the asize value without considering any math operation on them.
For savepos it would be complicated to adjust them.