[PS2] Tokimeki Memorial 3 - Getting Models from eeMemory?

[Natsuki Okusawa]

Would it be possible to get the models from eeMemory? It appears like that the game has compressed files for its assets, which are in those .atp files (I also made a separate thread about those files, but no reply so far...) so I'm thinking of the possibility of getting the models from eeMemory instead. Can anybody take a look?

eeMemory.rar

 

I suspect that the data that starts from 0x1700000 is a model data, but I'm not too sure.

I also pulled this data that I suspect to be a 3d model manually with a hex editor. Here is the file:

Untitled.rar

 

I can also find some interesting patterns, at around 0x155725a for example. These are probably textures.

 

Opening the data with Model Researcher doesn't seem to reveal anything that is too recognizable though, but I would seem to get the best results when using 'Short_sign'.

Edited April 15 by Natsuki Okusawa

[shak-otay]

There were some threads about eeMemory data on old Xentax forum.

You might search through the backups made/presented by id-daemon (see Archives forum).

Edited April 15 by shak-otay

[Natsuki Okusawa]

Thank you for the response. The threads about eeMemory on Xentax were for different games though. There wasn't even any mention of this particular game on Xentax. I'm also almost certain that the procedures on getting models from eeMemory depends on the game, especially considering that PS2 games tend to have different formats per game.

Edited April 15 by Natsuki Okusawa

[shak-otay]

Looks like there's structures. But atm they don't sum up to something useful when logging shorts using the counts before 0x6D..

 

[roocker666]

That model format(from Untitled.rar) is very similar to SH2/3/4 PS2 just with different file header:  0x00 header size(4 bytes), 0x04 pointer to textures(4 bytes), 0x08 pointer to textures2(4 bytes), 0x0C empty(4 bytes), 0x10 empty(16 bytes). Then at 0x20 is Main mesh header. I posted a topic about SH4 but for XBOX , Main mesh header is similar, you can check here: SH4 XBOX model

Sadly, XBOX and PS2 have different submeshes format, I never understood PS2 submeshes, those are a little bit complex(specifically submesh headers) for what I know, vertices and Uvs are shorts. I think the only person who understood PS2 submeshes was Murugo, he created a plugin for Blender so maybe you can examine his model script to find some answers: Murugo's sh2/3 model plugin

NOTE: Tokimeki Memorial 3 was released in 2001 so that means TM3 models could have more differences in Main mesh header and submeshes headers because this game is older that SH2/3/4..

Edited April 16 by roocker666

[Natsuki Okusawa]

Speaking of Silent Hill, a part of me was actually hoping that Tokimeki Memorial 3 uses a similar model format as the Silent Hill games from around the time of its release.

I was thinking of trying the Blender script for the PS2 SH2/3 models on the file (Untitled) that I provided to see if it will work, but I couldn't even get the script to work at all unfortunately, as I'm getting errors related to 'gsutil', which I think is some library necessary for the Blender script to work.

Well anyway, both Silent Hill 2 and Tokimeki Memorial 3 were released in 2001, with SH2 being released a few months prior to TM3.

Edited April 16 by Natsuki Okusawa

[shak-otay]

For the 32 MB eeMemory.bin after signature 04010001  (for addresses > 0x1484000).

 

Edited April 17 by shak-otay

[shak-otay]

Well, I ran out of ideas. No matter what I choose, ID 0x66 or 0x69 for example, all I get is $%&§ normals. Ok, last try will be 6D...

Edited April 16 by shak-otay

[roocker666]

OK!, I think I understand a little bit. Signature "04010001" is for vertex buffer, then we can find 3 more, these for vertices, Uvs and probably normals. SO it should be something like this:

 

 

[roocker666]

Third byte of  tag(in verts, Uvs, normals) is the count. In this case 19 HEX

Now this is an individual submesh but in other submeshes it seems like there are vertex groups.

NOTE: tags of verts, Uvs and normals change depending on the vertex buffer.  I think Uvs and normlas have padding but I am not really sure..

Edited April 16 by roocker666

[shak-otay]

10 minutes ago, roocker666 said:

OK!, I think I understand a little bit. Signature "04010001" is for vertex buffer, then we can find 3 more, these for vertices, Uvs and probably normals. SO it should be something like this:

3 More? Haha, 04010001 is contained 5180 times in the 32 MB eeMemory.bin.

Ah, you mean "the next 3 ones"?

So you say, red (ID 0x69) is for vertices?

Edited April 16 by shak-otay

[roocker666]

2 minutes ago, shak-otay said:

3 More? Haha, 04010001 is contained 5180 times in the 32 MB eeMemory.bin.

Ah, you mean "the next 3 ones"?

So you say, red (ID 0x69) is for vertices?

Sorry, by three more I mean 3 different more, one for vertices, other for Uvs and other for normals. SO it is like this:

04010001 --vertex buffer tag

     04001969 -- verts tag

          vertices (shorts)

     0500196D -- UVs tag

           Uvs(shorts)

     07001965 --Normals tag

           Normals(short)

 

04010001 -- next submesh

 

Don't use eeMemory.bin file, that has like 3 or 4 different models because is PCSX2 memory. Better use Untitled.rar that is only one model.

[shak-otay]

Maybe I should think about the creation of vertices? All I get is spheres...

0x5610: 04010001 0E00 0D69

0. 5618 13 (5680)
v -0.093750 1.589844 0.593750 
v 1.746094 0.593750 -0.093750 
...

 

edit: ooh, NO! I used a stride of 8 which was wrong. Totally.

Now it looks like so (could be "something"):

 

Edited April 16 by shak-otay

[shak-otay]

PS2 uses auto created face indices, iirc. But it looks like shelves then,

So I used a point cloud skinner (well, looks holey):

 

[roocker666]

1 hour ago, shak-otay said:

PS2 uses auto created face indices, iirc. But it looks like shelves then,

So I used a point cloud skinner (well, looks holey):

 

 

Yeah that looks better. So probably the model is not in T pose, it needs the bones. Well, at least we got something, lol.

[Natsuki Okusawa]

First of all, thank you very much guys for taking the time and effort to take a look at this, I highly appreciate it. I'm very glad to see that this at least have gotten somewhere.

 

5 hours ago, roocker666 said:

Don't use eeMemory.bin file, that has like 3 or 4 different models because is PCSX2 memory. Better use Untitled.rar that is only one model.

I think it would be ideal to create an extractor, like some QuickBMS script to extract or automatically pull out all of the model data present within the eeMemory file? Either that or find a way to decompress those .atp files, but that would probably be much harder and I think it would be easier to just pull them out of the eeMemory file. I also just found the file that I think is the 'Untitled' file in its .atp form, which I just attached in my other thread.

 

23 hours ago, roocker666 said:

That model format(from Untitled.rar) is very similar to SH2/3/4 PS2 just with different file header:  0x00 header size(4 bytes), 0x04 pointer to textures(4 bytes), 0x08 pointer to textures2(4 bytes), 0x0C empty(4 bytes), 0x10 empty(16 bytes). Then at 0x20 is Main mesh header. I posted a topic about SH4 but for XBOX , Main mesh header is similar, you can check here: SH4 XBOX model

I wonder if it would be possible to modify the SH4 Noesis script on the thread you linked to work with TM3 models, considering the possible similarity between the formats?

[roocker666]

5 hours ago, Natsuki Okusawa said:

I wonder if it would be possible to modify the SH4 Noesis script on the thread you linked to work with TM3 models, considering the possible similarity between the formats?

That will be a little hard because SH4 Noesis script is for XBOX meshes, PS2 meshes have very different format, only submeshes headers are similar. Maybe  modifying Murugo's script could be better(but Murugo's script is hard to understand for me..) because SH2 PS2 meshes format are almost identical, the only difference is in Normals; SH2 Normals don't have padding, in "Untitled" it seems like Normals have padding 2. Here is an update but I still don't know if it is correct at all(I found other tag at the end) of mesh:

 

Edited April 17 by roocker666

[Natsuki Okusawa]

2 hours ago, roocker666 said:

That will be a little hard because SH4 Noesis script is for XBOX meshes, PS2 meshes have very different format, only submeshes headers are similar. Maybe  modifying Murugo's script could be better(but Murugo's script is hard to understand for me..) because SH2 PS2 meshes format are almost identical, the only difference is in Normals; SH2 Normals don't have padding, in "Untitled" it seems like Normals have padding 2. Here is an update but I still don't know if it is correct at all(I found other tag at the end) of mesh:

Alright, I see. Did you at least got Murugo's script to work or installed in Blender? Unfortunately for me, as I mentioned earlier, I couldn't get the script to work at all. 😢

[roocker666]

8 hours ago, Natsuki Okusawa said:

Alright, I see. Did you at least got Murugo's script to work or installed in Blender? Unfortunately for me, as I mentioned earlier, I couldn't get the script to work at all. 😢

Yes but it loads the skeleton only, no submeshes. Maybe because there are some differences in submeshes(and probably their headers). The skeleton works, all bones are connected.

I added SH2 header in Untitled and saved it as .mdl

 

[Natsuki Okusawa]

11 hours ago, roocker666 said:

Yes but it loads the skeleton only, no submeshes. Maybe because there are some differences in submeshes(and probably their headers). The skeleton works, all bones are connected.

I added SH2 header in Untitled and saved it as .mdl

Nice! So the Blender script can at least import the 'Untitled' file. I guess that would mean taking a look at the script can give some hints on how to get the bones?

Anyway, how did you install the script? Did it involved compiling something? When I tried to install and use the script, I keep getting errors related to 'gsutil'.

Edited April 18 by Natsuki Okusawa

[roocker666]

1 hour ago, Natsuki Okusawa said:

Nice! So the Blender script can at least import the 'Untitled' file...

Yes but you need to edit the file in Hex editor, just delete the header from Untitled and paste the header of any SH2 model, then save it as .mdl

SH2 header includes an ID for model and pointers to mesh, textures and palettes. Pointers to textures and palettes are useless here because textures have a complete different format but are necessary to read the file(in Murugo's plugin).

About the plugin just download the zip from releases and install it on Blender 2.93.0, I got same problem with other Blender versions so I used that version.

I guess you can check the plugin to see how skeleton works(see import_model.py), you can export the skeleton from Blender if you need it but without meshes, lol. Here is Untitled with SH2 header and already saved as .mdl

Maybe meshes don't work because Normals in Untitled have 4 extra bytes or padding, in SH2 there is no padding. Maybe you can edit how to read Normals in "import_model.py" but I am not really sure if this could work, probably there are more differences in meshes format that I did not notice yet, who knows.. 

 

Untitled_SH2_hdr.rar

Edited April 18 by roocker666

[Natsuki Okusawa]

I just downloaded and installed the version of Blender you mentioned and installed the script on it, but I'm still getting the same 'gsutil' errors when I try to import anything.

May I see the files in your 'io_sh2_sh3' folder located on 'C:\Users\*\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\'? Thank you.

Nevermind, I finally got the script working, I was just downloading the wrong files in the Github page.

Edited April 18 by Natsuki Okusawa

[Natsuki Okusawa]

I noticed that there is an error when I import the modified Untitled file, and importing it would sometimes crash Blender for some reason.

Quote

Python: Traceback (most recent call last):
  File "C:\Users\ADMIN\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\io_sh2_sh3\__init__.py", line 50, in execute
    status, msg = import_mdl.load(context, **keywords)
  File "C:\Users\ADMIN\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\io_sh2_sh3\import_mdl.py", line 555, in load
    parser.parse(filepath)
  File "C:\Users\ADMIN\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\io_sh2_sh3\import_mdl.py", line 113, in parse
    self.parse_submeshes(f, model_header.submesh_count,
  File "C:\Users\ADMIN\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\io_sh2_sh3\import_mdl.py", line 298, in parse_submeshes
    vt for pair in [uv[loop.vertex_index] for loop in mesh_data.loops]
  File "C:\Users\ADMIN\AppData\Roaming\Blender Foundation\Blender\2.93\scripts\addons\io_sh2_sh3\import_mdl.py", line 298, in <listcomp>
    vt for pair in [uv[loop.vertex_index] for loop in mesh_data.loops]
IndexError: list index out of range

location: <unknown location>:-1

 

I went on and asked ChatGPT about the error but I kinda doubt what it said would be really helpful. (lol)

Edited April 19 by Natsuki Okusawa

[roocker666]

23 hours ago, Natsuki Okusawa said:

I went on and asked ChatGPT about the error but I kinda doubt what it said would be really helpful. (lol)

 

That helps, you can use "print(name of vertices, Normals or Uvs variable)" in import_model.py to print vertices, normals or Uvs into the console. I did that but console shows vertices of first mesh only(it should display vertices of all meshes), no Normals nor Uvs(with SH2 models it shows all fine). SO that means the problem are Normals, like I said in SH2 Normals are like this: N1(shortX, shortY, shortZ); N2(shortX, shortY, shortZ); N3(shortX, shortY, shortZ) etc.. In Untitled are: N1(shortX, shortY, shortZ, short); N2(shortX, shortY, shortZ, short); N3(shortX, shortY, shortZ, short) etc..

There is an extra short in Normals from Untitiled, So SH2 Normals have 3 shorts and Untitled has 4 shorts, fourth short is the problem. I think import_model.py reads the extra short as the start of the next Normal and that creates  a mess. If this is the problem, you need to fix how import_model.py reads Normals but the script is hard to understand.. I can't do more man, sorry.

 

Edited April 20 by roocker666

[Natsuki Okusawa]

Well I admit, I can't do that much either sadly. I would also often struggle in understanding this kind of stuff, though I do kind of get some of the logic but my understanding is very limited on this. From what I can see though, you at least appear to have a better understanding on this stuff than I do.

Anyway, I've been thinking of reaching out to Murugo by sending them a DM about this for them to at least take a look at this, seeing that they are on Discord, but then looks like that would be against the rules of the Xentax Discord server from what I saw in the server's rules. Should I still go on and send Murugo a DM?

Update: I just tried to send Murugo a DM but the message couldn't be sent. 🤷‍♂️ Oh well, I guess its understandable on why they may not want to get a DM.

Edited April 22 by Natsuki Okusawa