This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=13719

zkdldk95, posted Sun Jun 07, 2020 5:51 pm (57093)

---

Hello, I'm trying to translate Zero Escape The Nonary Games but have some problems.

I extracted .sir files from ze1_data.bin, and making any change of data size cause errors in game.

Of course, I used reimport2.bat for reimporting the sir files and adjusted their offsets.

Also, I checked just that exchanging a word is fine. (door -> dddd is printed well)

I don't know what the problem is. Please give some help.
(And what is the 5-byte data at the end of offsets?)

haibaer, posted Tue Jun 09, 2020 10:21 pm (57126)

---

search old posts. i've explained why reimport2 is not working well.
For some kind of files, you also need to modify the last few bytes which contains a 7-bit encoded (check variable-length quantity on wikipedia) data size.
It has been a while since i finished the localization for my language so i need to check my codes for details if you need.
All questions are welcome

zkdldk95, posted Wed Jun 10, 2020 12:06 pm (57133)

---

haibaer wrote:

search old posts. i've explained why reimport2 is not working well.
For some kind of files, you also need to modify the last few bytes which contains a 7-bit encoded (check variable-length quantity on wikipedia) data size.
It has been a while since i finished the localization for my language so i need to check my codes for details if you need.
All questions are welcome

Thank you for the answer. I learned the variable-length quantitiy but still have some question.

A file name is 00009f13.sir and [04 08 84 A8 78] is on the last.

I thought [84 A8 78] designates the file size and changed it to binary number.

1000 0100
1010 1000
0111 1000

-> 100 010 1000 111 1000 = 70776(dec) = 0x14478

but the file size is 102KB (105,264 Bytes). I don't know what's going on.

And second, how did you change the fonts to your language?

Thanks again.

haibaer, posted Thu Jun 11, 2020 6:07 am (57139)

---

00009f13.sir contains script texts.
0x00 - 0x03: SIR1
0x04 - 0x07: start offset of text index table(0001147C)
0x08 - 0x0B: 0
0x0B - 0x0F: end offset of text index table
0x10 - 0x13: 0
0x14 - 0001147B: texts
0001147C : text index table
...

Each entry of the text index table is a 8-byte absolute offset of the file and each text is a 0-terminated string:
0x00000014: M10a_050_010_02_10
0x00000027: Talk
...
...

If your final script text won't exceed the original size, you can simply modify the index table, adjust the offset of each text item, and fill the rest part of texts with zero.(No need to use reimport2, and I believe aluigi has already provided a bms script to do this automatically)
If not, you need to modify the 7-bit encoded data.

04 -> the first 4-byte of the file
08 -> next 8 bytes
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
08 08 08 08 ...... -> Each '08' corresponds to an text index table entry.

00009ec1.sir is a font file, you need to modify these font files if your language contains non-ascii characters. They are stored as 8-bit grayscale bitmaps.

zkdldk95, posted Thu Jun 11, 2020 10:10 am (57140)

---

haibaer wrote:

00009f13.sir contains script texts.
0x00 - 0x03: SIR1
0x04 - 0x07: start offset of text index table(0001147C)
0x08 - 0x0B: 0
0x0B - 0x0F: end offset of text index table
0x10 - 0x13: 0
0x14 - 0001147B: texts
0001147C : text index table
...

Each entry of the text index table is a 8-byte absolute offset of the file and each text is a 0-terminated string:
0x00000014: M10a_050_010_02_10
0x00000027: Talk
...
...

If your final script text won't exceed the original size, you can simply modify the index table, adjust the offset of each text item, and fill the rest part of texts with zero.(No need to use reimport2, and I believe aluigi has already provided a bms script to do this automatically)
If not, you need to modify the 7-bit encoded data.

04 -> the first 4-byte of the file
08 -> next 8 bytes
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
08 08 08 08 ...... -> Each '08' corresponds to an text index table entry.

00009ec1.sir is a font file, you need to modify these font files if your language contains non-ascii characters. They are stored as 8-bit grayscale bitmaps.

I've tried it and still the error comes out.

Could you check my 00009f13.sir file? (I changed the last text 'door' to 'dooooor' so that the offset table still works.)

Or just give me your localization files, then i'll check it.

haibaer, posted Fri Jun 12, 2020 12:35 am (57148)

---

sorry but i did the whole work based on the japanese version so i didn't modify 00009f13.sir.

Here are the possible reasons why your file is not working:
1. AA AA AA at the end of the texts is used to keep the whole address things 4 bytes aligned. you can add one more 'o' to keep the alignment.
2. you've added 5 'o's so 84 a8 70 should now be 84 a8 75, not 84 a8 73.
3. maybe the offset and new file size are not corrected after reimort2.

I also uploaded my 0000119d.sir, a font file that I've modified both the contents and the file size. hope this could help.

I'll test your file on the english version on the weekend if you still have problem

zkdldk95, posted Fri Jun 12, 2020 3:12 am (57151)

---

haibaer wrote:

sorry but i did the whole work based on the japanese version so i didn't modify 00009f13.sir.

Here are the possible reasons why your file is not working:
1. AA AA AA at the end of the texts is used to keep the whole address things 4 bytes aligned. you can add one more 'o' to keep the alignment.
2. you've added 5 'o's so 84 a8 70 should now be 84 a8 75, not 84 a8 73.
3. maybe the offset and new file size are not corrected after reimort2.

I also uploaded my 0000119d.sir, a font file that I've modified both the contents and the file size. hope this could help.

I'll test your file on the english version on the weekend if you still have problem

Thank you!! I tried the first solution(adjust number of AA AA AA to keep the alignment) and it works!!

No more error occurs. Have a nice weekend!

zkdldk95, posted Sat Jun 13, 2020 10:54 am (57178)

---

Me again.
I reimported your font file and it did't work in japanese mode.

So I have two questions.
1. How did you change the characters? Did you use Python or crystaltiles2?(I'm trying to with crystaltile2 but it's difficult)

2. What are the offsets in the font files? They are pointing middle of a character(so I cant insert more characters)

sorry for many questions. I really want to localize this game. Thank you.

haibaer, posted Wed Jun 17, 2020 5:05 am (57230)

---

zkdldk95 wrote:

Me again.
I reimported your font file and it did't work in japanese mode.

So I have two questions.
1. How did you change the characters? Did you use Python or crystaltiles2?(I'm trying to with crystaltile2 but it's difficult)

2. What are the offsets in the font files? They are pointing middle of a character(so I cant insert more characters)

sorry for many questions. I really want to localize this game. Thank you.

I didn't use crystaltiles2.
The index table of the font files uses a relative offset (and divide by 2).
Here's my python code to modify the font file (000011a4.sir). hope this could help.
(The 7-bit encoded part is almost the same as other files If you need modify it.)

Code:

# @brief modify font files so that the game contains our own fonts.
def modify_fonts():
    fin = open('4.sir', 'rb')
    fout = open('000011a4.sir', 'wb')
    rel_poses = []
    old_file = fin.read()
    start = 0
    num = len(CN_ALL_CHAR)
    end = start   num * 4   20
    fout.write(b'\x53\x49\x52\x31')
    fout.write(start.to_bytes(8, byteorder='little'))
    fout.write(end.to_bytes(8, byteorder='little'))
    cur_pos = 20

    chs = list(ENCODE_TABLE.keys())
    chs.sort(key = sort_byte)

    for ch in chs:
        rel_poses.append(cur_pos)
        img = to_font(ch, 'fonts', 26)
        fout.write(img)
        cur_pos = cur_pos   len(img)
        if cur_pos % 4 != 0:
            pad = bytearray(cur_pos % 4)
            pad = pad.replace(b'\x00', b'\xaa')
            fout.write(pad)
            cur_pos = cur_pos   cur_pos % 4
    
    # update start and end position
    start = cur_pos
    end = start   num * 4   20

    fout.write(num.to_bytes(8, byteorder='little'))
    fout.write((0x0000001b).to_bytes(4, byteorder='little'))
    fout.write((0x00000014).to_bytes(4, byteorder='little'))
    fout.write((0x00000000).to_bytes(4, byteorder='little'))
    for rel in rel_poses:
        fout.write(int((rel - 0x14) / 2).to_bytes(4, byteorder='little'))           # <- the offset
    tab_pad = (16 - (start   20   4 * num) % 16) % 16
    pad = bytearray(tab_pad)
    pad = pad.replace(b'\x00', b'\xaa')
    fout.write(pad)

    fout.write(old_file[-16:])
    fin.close()
    fout.close()
    

zkdldk95, posted Fri Jun 19, 2020 7:19 pm (57276)

---

haibaer wrote:

Code:

# @brief modify font files so that the game contains our own fonts.
def modify_fonts():
    fin = open('4.sir', 'rb')
    fout = open('000011a4.sir', 'wb')
    rel_poses = []
    old_file = fin.read()
    start = 0
    num = len(CN_ALL_CHAR)
    end = start   num * 4   20
    fout.write(b'\x53\x49\x52\x31')
    fout.write(start.to_bytes(8, byteorder='little'))
    fout.write(end.to_bytes(8, byteorder='little'))
    cur_pos = 20

    chs = list(ENCODE_TABLE.keys())
    chs.sort(key = sort_byte)

    for ch in chs:
        rel_poses.append(cur_pos)
        img = to_font(ch, 'fonts', 26)
        fout.write(img)
        cur_pos = cur_pos   len(img)
        if cur_pos % 4 != 0:
            pad = bytearray(cur_pos % 4)
            pad = pad.replace(b'\x00', b'\xaa')
            fout.write(pad)
            cur_pos = cur_pos   cur_pos % 4
    
    # update start and end position
    start = cur_pos
    end = start   num * 4   20

    fout.write(num.to_bytes(8, byteorder='little'))
    fout.write((0x0000001b).to_bytes(4, byteorder='little'))
    fout.write((0x00000014).to_bytes(4, byteorder='little'))
    fout.write((0x00000000).to_bytes(4, byteorder='little'))
    for rel in rel_poses:
        fout.write(int((rel - 0x14) / 2).to_bytes(4, byteorder='little'))           # <- the offset
    tab_pad = (16 - (start   20   4 * num) % 16) % 16
    pad = bytearray(tab_pad)
    pad = pad.replace(b'\x00', b'\xaa')
    fout.write(pad)

    fout.write(old_file[-16:])
    fin.close()
    fout.close()
    

It is difficult to understand a little bit but it's ok.

I wonder how can I change a font to byte arrays. ("to_font" in your code might has that function)

I hope this is the last question...

Thanks a lot.

haibaer, posted Sat Jun 20, 2020 8:12 am (57288)

---

The font file uses 8-bit grayscale bitmaps.
You can generate grayscale bitmaps from ttf/otf fonts using any font engine. For me I use freetype. (You can use the Pillow library if you are using python).
Here is the structure of each glyph in the font file:

Shift-JIS encoding of that glyph (2 bytes) (Note: the game uses binary search to find the right glyph, so this encoding value should always be in increasing order.)
0x00 (4 bytes)
width of the glyph (1 byte)
height of the glyph (1 byte)
width of the border glyph (1 byte) (a larger bitmap used to draw borders, you can use the same glyph if you don't need border)
height of the border glyph (1 byte)
the glyph (a width * height grayscale bitmap)
the border glyph (a (width of the border glyph) * (height of the border glyph) grayscale bitmap)

Here's the source code of to_font, which may looks confusing too...

Code:

def to_font(ch, fontpath, height):
    enc = ENCODE_TABLE[ch]
    uni_idx = ord(ch)
    ret = bytearray()

    # read font bitmap
    fin = open(fontpath   '/'   str(uni_idx)   '.txt', 'rb')    # I generated the bitmaps and stored them to a text file.
    img = fin.read()
    mc = len(img)
    fin.close()

    metrics = FONT_METRIC

    width = metrics[uni_idx][4]

    if len(enc) == 1:
        L = [str(enc[0]), '0x0', '0x0', '0x0', '0x0', '0x0', str(width), str(height), str(width   2), str(height   2)]
    else:
        L = [str(enc[1]), str(enc[0]), '0x0', '0x0', '0x0', '0x0', str(width), str(height), str(width   2), str(height   2)]
    ret.extend(bytes(int(x, 0) for x in L))

    canvas = bytearray(width * height)
    # I removed some codes here because I did some tricks to draw the image onto the canvas, which may look confusing.
    # (I need to do so because my language has some special feature.)
    # You can use 'img' directly without using the canvas
    # e.g.
    # ret.extend(bytes(img))
    # ret.extend(bytes(img))
    # return ret
    
    ret.extend(bytes(canvas))
    ret.extend(bytes(to_border(img, mc, width, height, metrics, uni_idx)))
    return ret

zkdldk95, posted Fri Jun 26, 2020 6:15 pm (57370)

---

haibaer wrote:

The font file uses 8-bit grayscale bitmaps.
You can generate grayscale bitmaps from ttf/otf fonts using any font engine. For me I use freetype. (You can use the Pillow library if you are using python).
Here is the structure of each glyph in the font file:

Shift-JIS encoding of that glyph (2 bytes) (Note: the game uses binary search to find the right glyph, so this encoding value should always be in increasing order.)
0x00 (4 bytes)
width of the glyph (1 byte)
height of the glyph (1 byte)
width of the border glyph (1 byte) (a larger bitmap used to draw borders, you can use the same glyph if you don't need border)
height of the border glyph (1 byte)
the glyph (a width * height grayscale bitmap)
the border glyph (a (width of the border glyph) * (height of the border glyph) grayscale bitmap)

Could you see my modified font file and check what's wrong?

At the end of the file, i added the glyph '?' (at 0x2a834)

And then i changed offset tables (head and bottom both) and
number of characters (84 -> 85 at 0x2b18c) and
the data size(variable-length quantity).

But still the game is crashed.

it is English-font file, but you might understand it.

erito, posted Sat Jun 12, 2021 9:16 pm (64654)

---

Hey, I know this post have been inactive for quite some time, but I just wanted to ask a little thing:

haibaer wrote:

...
84 A8 70 -> 0x11470 next 0x11470 bytes (to the start offset of text index table)
...

I might be really dumb, but what is the relation between "84 A8 70" and "0x11470", how do you convert between the two?
Thanks!

EDIT: Forgot to check on variable-length quantity, my bad.
Quick brief for anyone passing by, using the example of `00009ed8.sir`:

• Take the offset of the TOC (`0x2A834`)
• Convert it to binary (`10 1010 1000 0011 0100`)
• Group them by 7, starting from the right. If the left group contains less than 7 character, add 0s : (`0001010`, `1010000`, `0110100`)
• Add a 1 at the start of each group, except the right one, where you must add a 0 : (`10001010`, `11010000`, `00110100`)
• Combine it all ! (`100010101101000000110100`)
• Convert it in hex. (`0x8AD034`)
• Put it after `04 08 80` at the end of the file
• Note: While most of the offset are written in little endian, this one seems to be written in big endian (which mean that the bytes aren't reversed)

erito, posted Thu Jun 17, 2021 7:07 pm (64754)

---

Hey, here I am again, always one year late...
So I ran into the same problem as zkdldk95: added a character for my language ("e", identified in Unicode by U 009E). So I tried, but each times it fails, even though I checked many times that I have the correct offsets, VLQ, etc...
I tried to go in a more in-depth experiment, trying to change each components of the file individually to understand what might cause crashes. Judging by my experiment, it seems like it could be:

• Beginning offset of table, however already checked that carefully
• The VLQ, but I also checked that

I'm really wondering on what could be wrong right now... At this point, I'm really wondering if reimport2, which was upgraded since your last posts, really does the job for this particular case? Do anyone who managed to actually add characters in the font files without crashes could explain how they did?
Looking forward to it, thanks!

erito, posted Sat Jun 19, 2021 12:48 pm (64790)

---

OK so I actually tried to rewrite the whole file, the characters are messy but that's no big deal...
The game crashed when clicking the start button; now, it just won't display the character I created (except the first one)... Why does only the first one works (?) and not the second (A) and the following? Do I have to, like, escape the other characters? I've noticed that " ' " are actually written as " ,r " in the text files, so is there this kind of manipulation for that too?
(of course, I changed `00009f0e.sir` in order to make those characters appear)

EDIT:
OK so now, just tried some new things, and it seems that when I try to print '84BE' or '84something', it only displays a big D on a white background, although the glyph entry for this wasn't that D at all...

erito, posted Tue Jun 22, 2021 9:55 am (64840)

---

Well, I think no one's watching this, but it may be useful for anyone trying to translate this, so here's what I found out:
The best is to rewrite the font file (`00009ed8.sir`) by yourself. It might take some time, but honestly that's the easiest way. Try to have a height of 35 px for each character. To create new characters (just as I want to add e, for instance), you have to modify the encoding of the glyph. e was 0xE900 (little endian), but in order to make it work, I had to transform it to 0xE981. Then, in your text file (for instance, `00009f0e.sir`, the beginning of the game) you just put `81E9` each time you want to use this character. Please, note that if characters are put in the wrong order in `00009ed8.sir`, the character won't appear.