This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=8609

Mana, posted Sun Sep 23, 2018 7:14 am (38740)

---

Is there any way to force appending something in reimport 2 mode? I'm pretty sure I've figured out how to add new files to the end of the Shadow of War .arch06 archives, but I'd need to have quickbms append them so that they're compressed. I'd then manually edit the file to add the appropriate file name and file size values (just before each file in the archive theres a value for the size of that file). Reimport2 doesn't work with these files because they're sequential, and I can't force it with that mode.

Mana, posted Sun Sep 23, 2018 7:25 am (38743)

---

While I wait to hear back, I'm going to test my theory by attempting to add a new file to the end of an archive. I'll edit this comment with the results.

EDIT: Wasn't as easy as I'd hoped it would be, but initial results show promise. Haven't managed to append the file yet, as the format was more complicated than it first appeared. I am certain that appending a new file is possible with the right knowledge. Aluigi could almost certainly do it, I just don't know the format well enough yet. I still need to be able to force reimport2.bat to append a compressed file to the end of it.

aluigi, posted Sun Sep 23, 2018 9:34 am (38745)

---

No, you can't add new files

Mana, posted Sun Sep 23, 2018 9:41 am (38746)

---

Why not? Whats special about this format compared to most other archives that can have files added? I don't mean automatically using just quickbms and nothing else, but manually through hex editing. I just need to force the reimport to get the binary data I need, which I would then manually add into another copy of the archive with the appropriate adjustments to the header and file offsets. I've tried minor adjustments to the archives before, and provided the offsets are correctly altered the game loads them fine.

aluigi, posted Sun Sep 23, 2018 9:54 am (38747)

---

Mana wrote:

I don't mean automatically using just quickbms and nothing else

Why are you posting a game-specific question in a topic dedicated to a specific feature of quickbms?

You are off-topic, the topic of that game is viewtopic.php?f=9&t=5080

Mana, posted Sun Sep 23, 2018 9:59 am (38748)

---

Because it will require forcing the quickbms reimport2 mode to do it. At least for now. My question was is there any way to force reimport2 to attempt it. If not could I put in a feature request for that?

aluigi, posted Sun Sep 23, 2018 10:14 am (38749)

---

Once you correctly modify the TOC (table containing information for each file) adding the reference to the new file using offset and sizes equal than 0, quickbms will do everything automatically.

Mana, posted Sun Sep 23, 2018 10:22 am (38750)

---

Oh I see. Could you clarify what you mean by sizes equal to 0? Did you mean greater than 0? Thanks for the answer, I'll move to the games specific thread now. In order to get the correct sizes for the files (for example, a modified mesh or altered file), I'd need to reimport it into something wouldn't I? I feel like I'm misunderstanding something.

aluigi, posted Sun Sep 23, 2018 1:24 pm (38752)

---

When you run quickbms with the -V option it displays all the fields read from the format, it's a sort of debugging option which is extremely useful while writing scripts.

Depending by the script, a size equal than 0 may force the script to skip the file, so let's say (as universal rule) to use 1 for all the 2 1 fields involved:
OFFSET
SIZE
ZSIZE (the compressed size is sometimes optional, depends by the format)

So if all these fields for one file are equal than 1, then quickbms in reimport2 will be automatically forced to append the file at the end of the archive and rewrite the new 3 fields over the original.

Please note that I'm expecting you perfectly know things like TOC size (you can't add new files if you can't increase the TOC), endianess, size of the fields and everything else necessary to do what you are trying to do (and there are big chances that's not possible at all due to the TOC size).
If you have no experience with this stuff please stop, or feel free to continue but don't ask me support (eh limited time).

Now I would like to move these OT posts in a new topic in another section if that's ok for you.

Mana, posted Mon Sep 24, 2018 8:03 am (38769)

---

Of course. Thanks for being so accommodating. I've got experience with this stuff, but my knowledge is a bit iffy. I'm an artist initially, programming stuff takes me a bit to learn. I know most of the things you listed, though not necessarily how they work in this particular format just yet. My eventual plan is to add "new" files to these archives (or new archives completely) to allow for model swapping. Should be really cool if I can get it to work.

aluigi, posted Mon Sep 24, 2018 9:24 am (38770)

---

Adding new files with the limitations of the TOC size is something painful.
But, if the game accepts new archives (for example N archives with different names instead of one big monolytic archive, it was possible with Telltale games), you may opt for something like copying one existent archive and editing its entries replacing the existent ones with those of the new files you want to add.
But even in that case it's a quite big work to do manually through hex editing.

Mana, posted Mon Sep 24, 2018 11:00 am (38773)

---

Yeah, the archives the game loads are read from a plain text file. My current plan is to manually add in just the files I need to replace into a new archive and load that. Requires a bunch of manual work, but its looking promising. I wasn't actually expecting it to work out but so far its going well.

EDIT: Incidentally, I have a very odd "bug" resulting from doing this. The T in the LTAR of the first four bytes gets replaced with 00 for some reason. I just manually put the T back, but I can't imagine why that happens.

aluigi, posted Mon Sep 24, 2018 12:45 pm (38775)

---

Mana wrote:

EDIT: Incidentally, I have a very odd "bug" resulting from doing this. The T in the LTAR of the first four bytes gets replaced with 00 for some reason. I just manually put the T back, but I can't imagine why that happens.

Right, I forgot about that.
Basically reimport2 automatically zeroes the old location of the file for debugging while testing the edited archive and allowing the compression of the whole archive if the modder wants to distribute it.
If ZSIZE/SIZE was zero there was no problem but since you are forced to use a positive number (due to the script requiring it) then you will have this side-effect, just remember to manually restore the 'T'

aluigi, posted Mon Sep 24, 2018 12:59 pm (38778)

---

I have splitted the posts in a new topic so you can ask and provide information and updates without any OT problem

Mana, posted Mon Sep 24, 2018 1:05 pm (38780)

---

Thanks. This whole thing has been going so much better than I'd hoped. The folder structure is going to be one of the hardest bits I think. The game will be searching for these models in a certain folder structure. I'll need to create that in my new archive. I've looked at some existing archives to try and wrap my head around how the folders get extracted. Are they treated like files or something?

aluigi, posted Mon Sep 24, 2018 1:23 pm (38781)

---

If your archive is an LTAR one, then this is the part of the script you have to consider:

Code:

        get VER long
        get NAMES_SIZE long
        get MAIN_FLAGS long
        get FILES long
        getdstring DUMMY 0x1c

        savepos OFFSET
        log MEMORY_FILE OFFSET NAMES_SIZE
        math OFFSET   NAMES_SIZE
        goto OFFSET

        for i = 0 < FILES
            get NAME_OFF long
            get OFFSET longlong
            get ZSIZE longlong
            get SIZE longlong
            if VER >= 4
                get DUMMY byte  # 1
                get FLAGS byte  # usually 9 for compressed chunks
            else
                get FLAGS long  # usually 9 for compressed chunks
            endif

            goto NAME_OFF MEMORY_FILE
            get NAME string MEMORY_FILE
            string NAME R= "_" "/"

            if ZSIZE > 0
                if SIZE != ZSIZE    # or FLAGS & 1 (FLAGS is usually 9)
                    ...
                else
                    log NAME OFFSET SIZE
                endif
            endif
        next i

I don't remember if folder are archived anyway, maybe with ZSIZE zero, or not.
The script says that the path delimiter is the underscore and not the slash/backslash, so if you have a file with full name FOLDER1\FOLDER2\FILE.TXT you have to set it as FOLDER1_FOLDER2_FILE.TXT
Remember to set FLAGS to zero or 8 (you must try both) just in case the game enables compression when it's set otherwise.
The best suggestion I can give you is to make tons of tests with different settings.
The archive is probably not very complex and, apparently, there is only one field which is not parsed (DUMMY 0x1c) so you may opt for recreating it from scratch instead of editing existing ones.

Mana, posted Mon Sep 24, 2018 1:29 pm (38782)

---

Thanks. Doing lots of tests is my preferred method of learning these things. I'll try using the names to create the folder structure. The other option is that instead of mimicking the folders I instead alter the game to look for the files in a different place. If I can't make any headway with the folders I'll try that out.

Mana, posted Mon Sep 24, 2018 1:38 pm (38783)

---

Seems to have worked, I don't understand how the tool handles files that have an underscore in their name, but I don't need to worry about that for now. I'll keep testing things out, see if mimicking the file structure is enough to get the game to load it in place of the original or not.

EDIT: I've tried a bunch of different files, but haven't been able to replace anything yet. The game isn't crashing, so I'm pretty sure it just means I'm either naming the file wrong, putting it in the wrong file structure, or something similar to that. Tracking down the exact file I want to replace is proving very difficult. I've noticed that some files have underscores in their names, which will be problematic. I'll continue working on this tomorrow. Hopefully I can figure out what I'm doing wrong.

Mana, posted Tue Sep 25, 2018 4:28 am (38802)

---

I have a question, not sure if I should ask here or in the general SoW thread. The .BNDL files seem like they're supposed to be openable with the quickbms script, but every one I've tried has failed. Are they supposed to be opened?

aluigi, posted Tue Sep 25, 2018 5:51 am (38803)

---

Yes, the BNDL format is supported.
What error you get?
Please upload a sample or a the files generated by http://aluigi.org/bms/filecutter.bms

Mana, posted Tue Sep 25, 2018 6:42 am (38809)

---

By sample do you mean upload some bndl files for you to look at? I'll upload a folder with a bunch of them in it. I get different errors with different bndl files. Some of them give me an offset can't be reached error. Some manage to extract a file or two before failing. I've yet to have one actually work, but they don't all fail in the same way.

Mana, posted Tue Sep 25, 2018 9:55 am (38812)

---

Another thing. From what I can tell, some folders in the archives have an underscore in their name. I'm going to need some way to import that kind of folder for this to work. I think that might actually be why I haven't managed to pull off a replacement yet.

EDIT: Perhaps by renaming the folder once it's imported, like naming it "player-isildur" and then renaming it after importing to "player_isildur"

EDIT2: Tried that. Archive is functional but it didn't work either. game.gamedb is where I got the file path I tried out from. I'll have to keep trying things out till I find the right file path.

aluigi, posted Tue Sep 25, 2018 11:44 am (38813)

---

The bndl files you provided have no real data in them.
Any bigger bndl?

Mana, posted Thu Sep 27, 2018 11:04 am (38847)

---

Most of the bndls are fairly small. I was confused that they were considered an archive format. I'll post some larger ones. I don't think there are any much larger than this, unless I've missed some.

I've had to take a bit of a break from working on the format, as I'm at crunchtime on some projects. I believe the biggest obstacle I've currently got is the file structure. Theres some files that I think have underscores in their names, and folders with the same. Some archives I've looked at in binary have had some very odd looking ways of showing file paths. Currently the game loads my archives fine, so I know they're not broken, but the files within aren't ever being used. I believe thats because the filepath or file name is wrong.

I should be able to start testing things again next week. Let me know if you need anything and I'll dig around for it. I've gotten in touch with the game developers but sadly they haven't been able to share any information on the file structure.