Shadow of the Colossus Remake (PS4) Models

[Shadowth117]

54 minutes ago, squall789 said:

Theres some awesome work here, I've been hoping to get the files from this game for ages, but similar to what was said earlier, nobody else managed to crack it, or wasn't really interested enough in the format to want to input all the time it might take to get the files out.

If I've understood the thread correctly, roughly 100% of models are supported, and the majority of textures (bar a few), which is seriously cool.

There's definitely a fair few models with problems, but I think it can be worked out. The harder part is probably getting the textures with all mips correctly, only because they tend to be packaged a bit funny. It's better in SOTC than Demon's Souls from a glance, but it's a whole thing.

[squall789]

I've been testing for a little while and you're right, there are some models that don't work, it seems to be more specific to the environment meshes, architecture etc, but skeletal meshes seem to work more or less perfectly.

Even then, some environment meshes work just fine. Things in the 'actor' folder have more issues.
I know this is all still a work in progress though, but honestly it's great so far.

[Dmnslyrx8]

Nice to see that a lot of progress has been made while I was out of the country on vacation, but I'll try the latest tool update when I'm back home Saturday evening.

[Shadowth117]

I went ahead and pushed out a version of the tools with support for Demon's Souls textures. At the very least, it doesn't seem to crash on the patched version's texture files and most seem to look correct. 

I'm going to go ahead with implementing the SOTC variation, but for now, don't expect that to work.

[squall789]

13 hours ago, Shadowth117 said:

I went ahead and pushed out a version of the tools with support for Demon's Souls textures. At the very least, it doesn't seem to crash on the patched version's texture files and most seem to look correct. 

I'm going to go ahead with implementing the SOTC variation, but for now, don't expect that to work.

would it be possible to select *all* textures, and depending on the file the tool knows whether it needs decompressing and converting, or just converting etc?

[Shadowth117]

5 hours ago, squall789 said:

would it be possible to select *all* textures, and depending on the file the tool knows whether it needs decompressing and converting, or just converting etc?

That's already a thing. I also made the decompression thing not die if it hits something that can't be decompressed, but idk if I put that in the currently released build yet.

Basically though, you're not going to have to think about the compression for supported formats and already don't for models and skeletons. The ability for users to decompress directly at all is more for research.

Edited April 28 by Shadowth117

[Shadowth117]

Okay, so a lot of fiddling around to figure out UI texture resolutions later (they work differently than power of 2 texture resolutions), the SOTC *.ctxr conversion is ready to go:

https://github.com/Shadowth117/Aqua-Toolset/releases

It ended up being very similar to what Demon's Souls had, with slight adjustments, and it ended up helping a lot to do those first. Demon's Souls has more and a wider variety of types overall.

There's some standouts, but if someone wants to shoot a PR out to fix them, all for it. 

I also found these in the gallery data, which was pretty funny.

[Dmnslyrx8]

Tried the updated tool, so far had no issues, dunno if there's a way to fix the broken vertex normals tho on static models (models with no bones/rigging)

And some1 that has messaged me over discord saying some models don't work, like the temple stuff, not sure which files those are tho, but I'll look around for them.

[Shadowth117]

35 minutes ago, Dmnslyrx8 said:

Tried the updated tool, so far had no issues, dunno if there's a way to fix the broken vertex normals tho on static models (models with no bones/rigging)

And some1 that has messaged me over discord saying some models don't work, like the temple stuff, not sure which files those are tho, but I'll look around for them.

I'm aware of the model issue, as I mentioned previously, but one thing at a time. I was focusing textures for a bit because I wanted to knock those out while my mind was on it. 

Models, I'll try to look into soonish again. 

Regarding model normals, I actually don't convert these at all right now for any model, which forces most things to recalculate them. BluePoint's normals are weird in Demon's Souls and this. Demon's Souls has some strange quaternion handling it does for that and SOTC has something similarly quizzical, though I believe distinct. It would be nice if they could be perfect, but given they can be recalculated very easily and typically aren't specially done for this game afaik, I may just leave them as is.

[Dmnslyrx8]

ah ok, that makes sense. Had no clue u didn't implement vert normals yet since most models look fine, and then there's some that aren't.


found 1 model belonging to the temple that doesn't work at least
models\shadow\levels\cathedral\_cmn\cathedral_garden.cmsh

edit: some more
cathedral_new_firstlevel.cmsh

Quote

************** Exception Text **************
System.OverflowException: Arithmetic operation resulted in an overflow.
   at Reloaded.Memory.Streams.BufferedStreamReader`1.ReFillBuffer()
   at AquaModelLibrary.Data.BluePoint.CMSH.CMSHVertexData..ctor(BufferedStreamReaderBE`1 sr, CMSHHeader header, Boolean hasExtraFlags) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\PSO2-Aqua-Library\AquaModelLibrary.Data\BluePoint\CMSH\CMSHVertexData.cs:line 81
   at AquaModelLibrary.Data.BluePoint.CMSH.CMSH.Read(BufferedStreamReaderBE`1 sr) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\PSO2-Aqua-Library\AquaModelLibrary.Data\BluePoint\CMSH\CMSH.cs:line 39
   at AquaModelLibrary.Data.BluePoint.CMSH.CMSH..ctor(Byte[] file) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\PSO2-Aqua-Library\AquaModelLibrary.Data\BluePoint\CMSH\CMSH.cs:line 35
   at AquaModelLibrary.Core.BluePoint.BluePointConvert.ReadCMSH(String filePath, Dictionary`2 materialDict, AquaNode& aqn) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\PSO2-Aqua-Library\AquaModelLibrary.Core\BluePoint\BluePointConvert.cs:line 150
   at AquaModelLibrary.Core.BluePoint.BluePointConvert.ReadCMDL(String filePath) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\PSO2-Aqua-Library\AquaModelLibrary.Core\BluePoint\BluePointConvert.cs:line 69
   at AquaModelTool.AquaModelTool.convertDemonsSoulsPS5CmdlToFbxToolStripMenuItem_Click(Object sender, EventArgs e) in C:\Users\Shadi\Downloads\GitHub\Aqua Toolset\AquaModelTool\AquaModelToolUI.cs:line 3761
   at System.Windows.Forms.ToolStripMenuItem.OnClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleClick(EventArgs e)
   at System.Windows.Forms.ToolStripItem.HandleMouseUp(MouseEventArgs e)
   at System.Windows.Forms.ToolStrip.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.ToolStripDropDown.OnMouseUp(MouseEventArgs mea)
   at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.ScrollableControl.WndProc(Message& m)
   at System.Windows.Forms.ToolStrip.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, WM msg, IntPtr wparam, IntPtr lparam)

 

Edited April 28 by Dmnslyrx8

[Shadowth117]

If someone wants a crack at the normals by the way, here's a cube from a test folder. The normals are in 4 byte chunks here, 0x60 total. Tangents are the same format and are directly after. I highlighted them in the screenshot provided for 1_to_1_ratiocube.

1_to_1_ratiocube is from SOTC, cube_1m is from Demon's Souls. The values aren't the same, but pretty similar. Could be just differences in the export, but either way, would love to see the normals understood properly. If not, probably not the hugest deal though like I said.

1_to_1_ratiocube.zip cube_1m.zip

[Shadowth117]

1 hour ago, Dmnslyrx8 said:

ah ok, that makes sense. Had no clue u didn't implement vert normals yet since most models look fine, and then there's some that aren't.


found 1 model belonging to the temple that doesn't work at least
models\shadow\levels\cathedral\_cmn\cathedral_garden.cmsh

edit: some more
cathedral_new_firstlevel.cmsh

 

As far as these files, it looks like they have a LOT of extra data after the material data. But they were breaking earlier than that because of an extra flag I wasn't expecting. I'll have to look at these a bit. 

Edit: Looks like there's 12 that fail entirely. I'm aware there are others that are breaking (would appreciate them pointed out as they're come across!), but only 12 that actually crash the program. To be fair, that's definitely a lot better than I was expecting.

Edited April 29 by Shadowth117

[Shadowth117]

Made a small update to fix a few texture mishaps and some problems with .cskl reading. This doesn't affect those crashing models above right now, but I'll get to it. 

I did also add a QOL feature to have any materials that get read with a cmdl output all referenced textures alongside the models in the output folder, so hopefully that saves some time. Should be especially nice for when some very random texture in an unrelated directory gets referenced.

[Shadowth117]

Updated the tool to fix some files. There was a weird thing they were doing in material name reading, but it's fixed now. Still need to read the extra data in cathedral_garden.cmsh, cathedral_new_firstlevel.cmsh, and level_09_cliffs_displaced.cmsh. 

To copy from my github on the other broken one:

Of note, c03statue_break.cmsh seems to be improperly written. Extra data seems to be written after its normal footer, which by itself wouldn't be an issue, but even after cutting this down so the proper data can be read, it seems the decompressed size stored there is incorrect or the compressed buffer is just bad or incomplete. The extra data after this appears to be for another compressed buffer, but there's not much there and it doesn't have its own decompressed size to check sadly.

So yeah, that one's fucked. Won't be fixing it probably. Will try to fix the other ones outside of issues purely related to normals since idk what the game is doing there.

 

But yeah tldr everything in that list of busted files I posted above, other than the ones mentioned in this post, are able to convert more or less.

https://github.com/Shadowth117/Aqua-Toolset/releases

Edited May 2 by Shadowth117

[kingceryn]

Hey!  So this is great work, I was looking for some rough models to clean up in Blender, maybe some STLs for printing, but this is way more!  Does anyone happen to have any OBJs, and texture JPEGs exported I might be able to use?  Or a link to the assets you all used with the tool to export them?

[michalss]

10 hours ago, kingceryn said:

Uh not sure why I was “warned” lol. Got what I needed, so thanks 👍

Read the rules, you about asking for models, copyright material for download. That is the reason, also written in the warn message. Your post has been reported by members who actually read the rules. Forum is not about share any extracted copyright material, but about revers engineering.

[squall789]

On 5/3/2024 at 3:51 AM, Shadowth117 said:

Updated the tool to fix some files. There was a weird thing they were doing in material name reading, but it's fixed now. Still need to read the extra data in cathedral_garden.cmsh, cathedral_new_firstlevel.cmsh, and level_09_cliffs_displaced.cmsh. 

To copy from my github on the other broken one:

Of note, c03statue_break.cmsh seems to be improperly written. Extra data seems to be written after its normal footer, which by itself wouldn't be an issue, but even after cutting this down so the proper data can be read, it seems the decompressed size stored there is incorrect or the compressed buffer is just bad or incomplete. The extra data after this appears to be for another compressed buffer, but there's not much there and it doesn't have its own decompressed size to check sadly.

So yeah, that one's fucked. Won't be fixing it probably. Will try to fix the other ones outside of issues purely related to normals since idk what the game is doing there.

 

But yeah tldr everything in that list of busted files I posted above, other than the ones mentioned in this post, are able to convert more or less.

https://github.com/Shadowth117/Aqua-Toolset/releases

I'm curious, what is the extra data in cathedral_garden.cmsh, cathedral_new_firstlevel.cmsh, and level_09_cliffs_displaced.cmsh for?

 

[Dmnslyrx8]

11 hours ago, squall789 said:

I'm curious, what is the extra data in cathedral_garden.cmsh, cathedral_new_firstlevel.cmsh, and level_09_cliffs_displaced.cmsh for?

 

it's due to an extra flag that was never noticed, and extra data from that flag being included after the material data if I recall Shadowth's message about it.

Still waiting for that fix to work with those so I can make a render of the cathedral and the land around it, which shockingly fits into place w/o having to edit positions or anything.

Edited May 12 by Dmnslyrx8

[Shadowth117]

On 5/12/2024 at 1:55 AM, squall789 said:

I'm curious, what is the extra data in cathedral_garden.cmsh, cathedral_new_firstlevel.cmsh, and level_09_cliffs_displaced.cmsh for?

 

Well, that's what we don't know. And to be honest with you, I probably won't look into it past figuring out enough to parse through it when I have the time to look at that. But others are more than welcome to look at it when it's ready and make their own conclusions.

PS4 games can technically be modified so if someone wants to edit it all to see what exactly happens, they can, but that means repacking or otherwise editing the containing psarc, repacking a new FPKG, praying you repacked correctly so the game doesn't just crash and then praying that your edit doesn't simply break the game on its own. And this is without mentioning the lengthy copy times to and from a PC.

Suffice to say someone else is welcome to test all that if they want.

 

Edited Tuesday at 06:10 PM by Shadowth117

[Shadowth117]

Alright, the 3 that didn't work now work. The extra data looks to be bounding related for one set, and then maybe mesh separation related for the next. But yeah, I expect it's something debugging related given how few models it's in and the fact that all 3 have the same types of extra data there. It's a reasonably significant amount of data so I'd imagine they wouldn't want to compile it in typically.

The bounds could be used for something in the final, but I kind of doubt it. Not sure what to make of the mesh separation looking bit, but it is what it is. For now the output seems fine there.

https://github.com/Shadowth117/Aqua-Toolset/releases

Feel free to report on the models that are busted looking after conversion. 

Edited Wednesday at 09:40 PM by Shadowth117