michalss, posted Fri Oct 05, 2018 7:44 pm (39147)

---

Shokoniraya wrote:

i totally get it, aluigi! thank you for explain!
and i just want to tell you: can you add force mode in reimport2?
because in reimport1 force, file size cant update...
half of problem solved in force mode, but if you add force mode in reimport2 too, i can solve that!

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

Shokoniraya, posted Sat Oct 06, 2018 5:18 am (39163)

---

michalss wrote:

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

dear michalss: reimport2 means; set new offset and size and i totaly know that
and force mode means replace raw file

but we cant do it on single files white reimport2!
need a command option to replace raw files like force mode, not write, just like text files! and just adding a tag for do this thing...

go to end of the archive and in data table rewrite is size and offset: and set old data as "00 00 00 00", i know that.

aluigi, posted Sat Oct 06, 2018 8:24 am (39166)

---

What's probably not clear to many users is that the "force" option in reimport.bat is BAD and should be never used, NEVER.

It's something that corrupts the data and the only reason why it exists is that I preferred to give people a choice in case they really know what they are doing and that the edited archive will not work the 99.9% of times.
Please don't force me to remove that useless option from quickbms

Long story short: if you need to use the "force" option it means you are doing something completely wrong and you will receive no support.

That's the clear position about quickbms, but in your case, as already explained, you don't need quickbms to do something simple like that
Use a hex editor and do the job in one minute.

aluigi, posted Mon Oct 08, 2018 9:20 pm (39311)

---

I would like to introduce a better way to use the reimport and reimport2 modes by performing some additional checks.

1)
If you use reimport.bat for reimporting 1000 files and one of them is bigger than the original then the reimporting will be interrupted.
Possible solution:
- ask the user if he wants to retry (y/N/r/force) so he can edit the file and retry without running the process again

2)
If you have extracted 100000 files and have edited only one of them, quickbms will try to reimport all the 100000 files... it may be a problem for modders because deleting the non-edited files before reimporting may be a difficult (boring) process in some games.
Possible solution:
- calculate an hash (sha1 or sha256) on the archived file and an hash of the one on disk, compare them and reimport the file only if it's different. obviously if the size is different there is no need to make the hash.
While this solution seems nice, it has the disadvantage that basically quickbms needs to read (and decompress, just a full extraction without disk writing) the input file and compute the hash spending time and resources, multiply it by two since the same is done on the file on disk too.
sha1 is faster than sha256 and in this non-security context it's probably the best choice since no collisions are expected while editing a file.
I may implement this feature as an option for quickbms but I don't know if I have to add it to reimport.bat and reimport2.bat

GHOST DEAD, posted Tue Oct 09, 2018 3:31 am (39318)

---

Shokoniraya wrote:

michalss wrote:

I dont think you understand the reimport2 method. There is no such a force needed neither necessary as aluigi told you, when file is bigger its automatically go to end of the archive and in data table rewrite is size and offset, etc or other props needed. This method can't be used in same cases of segmented archives i guess..

dear michalss: reimport2 means; set new offset and size and i totaly know that
and force mode means replace raw file

but we cant do it on single files white reimport2!
need a command option to replace raw files like force mode, not write, just like text files! and just adding a tag for do this thing...

go to end of the archive and in data table rewrite is size and offset: and set old data as "00 00 00 00", i know that.

yeah, right. but you can't do it with QuickBMS
Try to learn a language for making your own Tools

QuickBMS can't help in such thing: Text Files, Single Files, converting an File to another File Like PNG to tex. replace raw_data

aluigi, posted Tue Oct 30, 2018 7:07 am (39971)

---

quickbms 0.9.1 will use the Capstone library as disassembler and will support every architecture supported by the library, so not just x86.

aluigi, posted Wed Oct 31, 2018 4:27 pm (39997)

---

reimport2 will work also with archives having the TOC at the end, maybe not 100% of times but most of them

aluigi, posted Thu Nov 01, 2018 9:31 pm (40035)

---

I still have no release date for quickbms 0.9.1 but at least I have finished to write the (temporary) changelog (the list of things fixed and added).
Even today I have fixed some bugs that have been casually found during a test.
I have also updated some libraries.

The only priority at the moment is beta testing for spotting new issues.
Hopefully next week I will have good news

michalss, posted Fri Nov 02, 2018 6:12 am (40040)

---

aluigi wrote:

I still have no release date for quickbms 0.9.1 but at least I have finished to write the (temporary) changelog (the list of things fixed and added).
Even today I have fixed some bugs that have been casually found during a test.
I have also updated some libraries.

The only priority at the moment is beta testing for spotting new issues.
Hopefully next week I will have good news

Are you also planning release fixed DDL with compress suport pls ? Waiting for this . thx

aluigi, posted Fri Nov 02, 2018 8:47 am (40041)

---

Sure

Shokoniraya, posted Sun Nov 04, 2018 6:37 am (40088)

---

GHOST DEAD wrote:

yeah, right. but you can't do it with QuickBMS
Try to learn a language for making your own Tools

QuickBMS can't help in such thing: Text Files, Single Files, converting an File to another File Like PNG to tex. replace raw_data

i saw your posts. you had same problem, right? how did you solve it then? i cant do it

aluigi wrote:

Sure

really? gonna be Great! even with text files? gonna possible?
cant wait to make new scripts with Good Stuff!

aluigi, posted Sun Nov 04, 2018 1:56 pm (40098)

---

What you mean with text files?

The problem described by michalss with the DLL was about the memory allocation method which is custom in quickbms and created a conflict when the dll is used in other tools.

Shokoniraya, posted Sun Nov 04, 2018 2:49 pm (40100)

---

aluigi wrote:

What you mean with text files?

The problem described by michalss with the DLL was about the memory allocation method which is custom in quickbms and created a conflict when the dll is used in other tools.

mean?
i mean, can we write code for export and import text files from a binary file?
i know that i asked before. but i just really glad to see that quickBMS can turn to a better tool for modders or file editors (actually thanks to you for all of this stuff)

i just want to know can i write a script for export text in a .txt file in new version? from a file like decompressed localization file?
because i know some algorithm for some localization file and dont know any other programming skill

and one other think; i know, in this times, maybe i asked alot and sorry for that, really i am

aluigi, posted Sun Nov 04, 2018 3:13 pm (40102)

---

Ah, that's already possible with the Slog command.
It does magic things BUT reimporting is limited to the original string size if the strings are in sequential order (since you can't allocate new space in the middle of the file).
If the strings is referenced by offset, reimport2 can import a new string of any length.

You can find all the information about Slog in quickbms.txt:

Code:

SLog NAME OFFSET SIZE [TYPE] [FILENUM]

    This command allows to export strings to an output file and being
    able to reimport them later with reimport.bat.
    The reimporting feature of this command has the same limitations of
    the global one, so you cannot reimport strings that are longer than
    the original.
    Currently QuickBMS will simply tell you that the new string is longer
    without interrupting the importing so pay attention.
    The space between the end of the new and old string will be filled
    with zeroes but this behaviour "may" be changed in future to avoid
    situations in which there are sequential NULL delimited strings and
    using zeroes will causes problems in the software that reads the
    file. Anyway in these situations you can just insert the spaces by
    yourself in the new string.
    The dumped strings are handles as C strings, basically the '\' char
    (backslash) is an escape that allows you to specify any byte you
    desire, the \r and \n you will see are just the 0x0d and 0x0a bytes
    that allow to insert the whole multiline string in one line.
    Each line is an exported string.
    Notepad   highly suggested.
    Small tip: Use INS (the replace character) mode of your text editor
    for editing the string file to reimport so that you will have no
    problems with longer new strings.
    The SLog function is new so any feedback and suggestion is highly
    appreciated.

    Arguments:
        NAME    Name of the output file. It will be created from scratch
                the first time and then will be used in append mode from
                the second line on. Automatic, simple and error-proof.
                The output file is just in UTF-8 with Windows line feeds
                ("\r\n") and the conversion of unicode strings is performed
                by QuickBMS automatically with the codepage in use
        OFFSET  The offset where is located the string you want to dump.
                >=0 works just like the other *log commands and dumps
                    the string located at that OFFSET, it doesn't change
                    the current offset
                <0  dumps the string at the current offset and updates it,
                    so it advances in the file
        SIZE    >=0 it works just like the Getdstring command allowing
                    you to dump a string of a certain amount of bytes
                <0  just like the Get command
        TYPE    This is the type of data to read just like the Get
                command, if not specified it's considered String. You can
                dump most of the types and even the non-string ones so if
                you use the Long type you will have the decimal value easy
                to edit inside the output file
        FILENUM the input file

    Examples:
      # the test file is available here http://aluigi.org/bms/slog_test.dat
      set STRINGS_FILE string "strings.txt"
      slog STRINGS_FILE -1 0xb string
      slog STRINGS_FILE -1 -1
      slog STRINGS_FILE -1 -1 long
      endian big
      #slog STRINGS_FILE -1 -1 unicode
      slog STRINGS_FILE -1 0x18 unicode
      slog STRINGS_FILE 0x2e -1

You can set NAME as "" and quickbms will automatically use a txt output filename:
slog "" -1 -1

Shokoniraya, posted Wed Nov 07, 2018 6:17 am (40169)

---

yes, thats the problem, cant import in bigger size
you know... some files does not have any offset, any...
my question is: quickBMS will support this option in future?

i dont say change reimport2, as i know, reimpor2 already is done and perfect for do with offset in normal files
but i say add a option like reimport3 mode for break this limits, if you worry maybe file will turning to an corrupt, i must say when i editing a localization file in hex, file can work verry well in game and just has SIZE, not offset.
please do somthing or at least say quickBMS will support this option in future?

aluigi, posted Wed Nov 07, 2018 7:58 am (40171)

---

It's technically impossible.
Sequential data doesn't have an offset and it's impossible to inject bigger data without breaking the whole rest of the format.
Yeah sure some very very simple formats that are 100% sequential would have no technical problem but it's a very very limited case, by experience they are rare, and in that case would be easier to do it by hand with a hex editor.

Shokoniraya, posted Thu Nov 08, 2018 8:01 am (40190)

---

aluigi wrote:

It's technically impossible.
Sequential data doesn't have an offset and it's impossible to inject bigger data without breaking the whole rest of the format.
Yeah sure some very very simple formats that are 100% sequential would have no technical problem but it's a very very limited case, by experience they are rare, and in that case would be easier to do it by hand with a hex editor.

just add a command option for breaking the whole rest of the format (for very very simple files like text without offset!
just add a option! not change other option
please do something about that, it is really on my mind and there is a lot of file and i cant make it possible with reimport2! just add a command options to break it! a command like -texter
really on my mind !

every time and every second and every hour! on my mind!

aluigi, posted Thu Nov 08, 2018 12:29 pm (40191)

---

What I meant with my example is a file with sequential unreferenced (no offset and no size) that you can edit even with notepad .

ANY other type of file that uses references (offset or size) CANNOT be forced OR requires accurate manual hex editing or just rebuilding from scratch.

Hope it's clear now

Shokoniraya, posted Fri Nov 09, 2018 5:48 am (40203)

---

for example
look at this zip file (its possible to make good scripts for texts, if you accept it

should i do these with all of simple text whit hex ! one by one! my time! my head! or maybe i get a wrong number, then game will close! taking a loooooong! (that is why its really really really on my mind!!!)
but hex is OK for big archive or 20 or 30 files, because we dont want to edit all files in archive, just for modding a texture or replace a binary template, just if file dont have any offset... but what about text? or a lot of simple files? yeah, easy with hex but taking soo long, i am working on modding a game and has 153214 simple file without offset! take 15 week to set all of them or more! my mind! game has .png and its not dds and cant add same size!
please do something in new release!

add a option with name ( -reraw ) or anything you want (or a better idea! add in bms... rlog? like slog or clog? (rlog = raw log) )
and add this things to this options:
-add replace raw file (not add a new offset in end of file)
ikow, ikow... maybe some archives will be corrupt for that, but people must use reimport2 (-r -r) in archives, not this option (because maded for files without offset, right?)

and its possible, and i cant believe... and because i know its possible... (in real, not by believing, can do it)

michalss, posted Fri Nov 09, 2018 7:34 am (40204)

---

Shokoniraya wrote:

for example
look at this zip file (its possible to make good scripts for texts, if you accept it !)

should i do these with all of simple text whit hex ! one by one! my time! my head! or maybe i get a wrong number, then game will close! taking a loooooong! (that is why its really really really on my mind!!!)
but hex is OK for big archive or 20 or 30 files, because we dont want to edit all files in archive, just for modding a texture or replace a binary template, just if file dont have any offset... but what about text? or a lot of simple files? yeah, easy with hex but taking soo long, i am working on modding a game and has 153214 simple file without offset! take 15 week to set all of them or more! my mind! game has .png and its not dds and cant add same size!
please do something in new release!

add a option with name ( -reraw ) or anything you want (or a better idea! add in bms... rlog? like slog or clog? (rlog = raw log) )
and add this things to this options:
-add replace raw file (not add a new offset in end of file)
ikow, ikow... maybe some archives will be corrupt for that, but people must use reimport2 (-r -r) in archives, not this option (because maded for files without offset, right?)

and its possible, and i cant believe... and because i know its possible... (in real, not by believing, can do it)

You kidding us or what, to there is already option to replace raw file in archive for god sake, luigi told this many times, dont get what you want anymore and i highly doubt you know it yrselfe... qbms can do exatly you asking so what is the point to get this any futher.. Extract files where are no any offset, etc.. is only matter of the script not capability of qbms..!

Shokoniraya, posted Fri Nov 09, 2018 8:24 am (40205)

---

not kidding, i swear! then where is the option? where?
can you tell me how can i import more size with a file without OFFSET? then tell me...

i want to kill my self

aluigi, posted Fri Nov 09, 2018 10:02 am (40209)

---

In the 3 files you provided in text.zip only text.localer is a format-less sequence of strings, just the one I said in my last post.
It's the only file that you can edit (rebuilding is better) without screwing the format.
If you don't want to make the work by hand you should write a program by yourself.
Even a script for quickbms may do the job because the format is extremely simple, one for extraction and one for rebuilding (yeah "rebuilding" not "reimporting").

This script automatically does both:

Code:

endian big
get EXT extension
get ARCHIVE_SIZE asize
if EXT == "localer"

    for OFFSET = 0 != ARCHIVE_SIZE
        get SIZE short
        slog "" -1 SIZE
        savepos OFFSET
    next

else

    log MEMORY_FILE 0 0
    append
    for OFFSET = 0 != ARCHIVE_SIZE
        get TMP line    # removes the tabs/spaces at the end
        strlen SIZE TMP
        put SIZE short MEMORY_FILE
        putdstring TMP SIZE MEMORY_FILE
        savepos OFFSET
    next
    append
    get SIZE asize MEMORY_FILE
    get NAME basename   # removes .txt
    log NAME 0 SIZE MEMORY_FILE

endif

The reason why the rebuilt text.localer is different than the original is simply because "get TMP line" removes tabs and spaces at the end of the input, like the string "Successfully defended. "

Shokoniraya, posted Fri Nov 09, 2018 12:28 pm (40221)

---

then add such tag in new version for test, ok sir aluigi?
i want to say thank to you

just a test on new version,
in new version, add raw replacer tag to break this limit in reimport 2 (add a command tag, like -w and others), for test
its verry useful for text and single raw

aluigi, posted Fri Nov 09, 2018 12:31 pm (40223)

---

@Shokoniraya
I don't know how to tell you in other ways so I give up with you.

Write your own program for rebuilding and doing your custom editings.
Hopefully now it's all clear.

P.S.: I want to give you an answer anyway, that type of editing you have in mind (replacing 10 bytes with 100 by shifting the content of the file of 90 bytes) is not simple and extremely inefficient to do since quickbms works on the file on disk and putting new data in the middle means copying the portion of file from that position till its end byte per byte. Do it for N strings and you can imagine the operations performed, all for something that "may" work on the 0.01% of the situations.
That's why I suggested you to create your own rebuilder.

Shokoniraya, posted Fri Nov 09, 2018 1:56 pm (40225)

---

aluigi wrote:

@Shokoniraya
I don't know how to tell you in other ways so I give up with you.

Write your own program for rebuilding and doing your custom editings.
Hopefully now it's all clear.

P.S.: I want to give you an answer anyway, that type of editing you have in mind (replacing 10 bytes with 100 by shifting the content of the file of 90 bytes) is not simple and extremely inefficient to do since quickbms works on the file on disk and putting new data in the middle means copying the portion of file from that position till its end byte per byte. Do it for N strings and you can imagine the operations performed, all for something that "may" work on the 0.01% of the situations.
That's why I suggested you to create your own rebuilder.

ok, its end here... thanks you anyway