Shadow of the Colossus Remake (PS4) Models

[Quasar728]

Hello, my goal is to extract models from the PS4 remake of Shadow of the Colossus. I found QuickBMS and have read the documentation and watched various videos on how to use it. I cannot find the right .bms script to use though and I have not tried to write it myself, hoping that someone here can help. I have the .pkg file for the US version of the game (CUSA08034) and have tried extracting the .pkg file using quickbms_4gb_file.exe with the 2 different .bms scripts. First with teamico.bms. This didn't work. Here's the output 
 

offset           filesize   filename
--------------------------------------
  0000000000200000 0          0000000000000000.dat
  0000000034333038 758132831  

it's not possible to create that file due to its filename or related
incompatibilities (for example already exists a folder with that name), so
now you must choose a new filename for saving it.
if you press ENTER a new name will be generated automatically.
old: 
new:

Next I tried pkg_cnt.bms because inspection of the .pkg file in a hex editor shows the first 4 bytes are `7F 43 4E 54` (Decoded text = .CNT). 
This somewhat worked but certainly did not extract all the files. Here's a screenshot of the extracted files.

It seems like this must have been done before because I saw this video

Can anyone help me understand what I'm doing wrong? Any help would be greatly appreciated.

[Quasar728]

Update: I was able to extract the .pkg using orbis-pub-chk from PS4-Fake-PKG-Tools-3.87. Yay! I learned that the key for fake packages is all zeros. This gave me some .psarc files which I then unpacked using UnPSARC_v1.5. Now it appears I have all the game data (including models). The only issue now is they are all .cmsh, .cclm, and .cskl files which I don't know how to view yet (see screenshot). After extensive googling I still haven't found anything useful.  If anyone has experience with these files and knows of a tool to convert them to something more familiar (e.g. .obj) or import them into Blender please let me know! 

[piken]

For the mesh and skeleton files, you could try 3D Model Researcher and SkelFinder to visualize the data and figure out their format.

[Quasar728]

Thank you for the suggestion! These seem like great tools! I had a look 3D Model Researcher and tried to see if I could get it to work with one of the .cmsh files. After comparing two .cmsh files in a hex editor I suspected that these files are encrypted. I then ran binwalk on one and here was the output. 

MD5 Checksum:  2c3021a26ec4f20cbcc16498becfa5e2
Signatures:    411

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
1388446       0x152F9E        Certificate in DER format (x509 v3), header length: 4, sequence length: 15012

I guess that means I'm out of luck if these files the .cmsh files are encrypted. Unless I misunderstand something. 

Another interesting thing I found is that the .cclm and .cskl files don't appear to be encrypted because they contain matching bytes at the beginning and the .cclm file references the .cmsh file. Here's a comparison of the beginning of two of each type of file.



Will someone confirm that this means I can't use .cmsh file, or tell me if that's not the case? Is it possible that the .cclm file contains a key for the .cmsh file? That seems weird and insecure but maybe. Thanks! 

Edited March 17 by Quasar728
rewording

[id-daemon]

6 hours ago, Quasar728 said:

Will someone confirm that this means I can't use .cmsh file, or tell me if that's not the case? Is it possible that the .cclm file contains a key for the .cmsh file? That seems weird and insecure but maybe. Thanks! 

None of these files look encrypted

[Quasar728]

I'm new to this so could you explain your reasoning to help me understand? What are the signs that the .cmsh files are not encrypted? I thought it might be because all but 1 bytes are different between the two cmsh files, they look random, and binwalk said there was a DER file signature.

Well, after further googling I found that the hex signature for the DER format is just two bytes 0x3082. So it looks like binwalk just happened to find those bytes at offset 0x152F9E and interpreted that as a DER signature. Now I'm thinking the .cmsh files are just the mesh data and maybe the .ccli files have the format information. If there was a header with the formatting of the .cmsh files within the .cmsh files themselves then that should match across different .cmsh files.  

Also here's what I get when I use the file command

colossus_03_knight.cmsh: data
colossus_03_knight.cclm: DOS executable (COM, 0x8C-variant)
colossus_03_knight.cskl: DOS executable (COM, 0x8C-variant)

Not quite sure where to go from here but I'll look into DOS executable (COM, 0x8C-variant).

Is it allowed to post these files in case an adventurous user would like to help me figure this out?

[DKDave]

3 hours ago, Quasar728 said:

I'm new to this so could you explain your reasoning to help me understand? What are the signs that the .cmsh files are not encrypted? I thought it might be because all but 1 bytes are different between the two cmsh files, they look random, and binwalk said there was a DER file signature.

Well, after further googling I found that the hex signature for the DER format is just two bytes 0x3082. So it looks like binwalk just happened to find those bytes at offset 0x152F9E and interpreted that as a DER signature. Now I'm thinking the .cmsh files are just the mesh data and maybe the .ccli files have the format information. If there was a header with the formatting of the .cmsh files within the .cmsh files themselves then that should match across different .cmsh files.  

Also here's what I get when I use the file command

colossus_03_knight.cmsh: data
colossus_03_knight.cclm: DOS executable (COM, 0x8C-variant)
colossus_03_knight.cskl: DOS executable (COM, 0x8C-variant)

Not quite sure where to go from here but I'll look into DOS executable (COM, 0x8C-variant).

Is it allowed to post these files in case an adventurous user would like to help me figure this out?

Yes, you can post some samples.

cclm is probably the information about the meshes

cskl is probably skeleton data

cmsh seems to be the mesh vertex data

 

 

 

[Quasar728]

Great! Here are the cclm, cskl, and cmsh for two models. I've zipped them so that they can be attached in this comment. The zip contains a directory SotC_test_models containing these 6 files. 

 

SotC_test_models.zip

[CharlieV]

This won't be any help, but there was a thread back on Xentax, I've been watching/attempting to investigate this game for years and previously provided samples myself, nobody ever got anywhere with anything, not even textures. The person who made the tool which works on Bluepoint's later Demon's Souls Remake said that despite the same file extensions it's completely different and they couldn't help. Any progress would be amazing, but if the past is anything to go by there isn't enough interest by those with the skills needed. 

[Shadowth117]

On 3/22/2024 at 12:01 AM, CharlieV said:

This won't be any help, but there was a thread back on Xentax, I've been watching/attempting to investigate this game for years and previously provided samples myself, nobody ever got anywhere with anything, not even textures. The person who made the tool which works on Bluepoint's later Demon's Souls Remake said that despite the same file extensions it's completely different and they couldn't help. Any progress would be amazing, but if the past is anything to go by there isn't enough interest by those with the skills needed. 

About that, at the time I thought it was different, but it's more complicated than that. For w/e reason cmsh went through a number of variations which ended up supported by the final game. DeS is a little like this, but with far less variation. Basically though, the format varies drastically based on the header. BluePoint seems allergic to pointers so most stuff in their formats is inlined, meaning you really need to understand it to get things to happen. Note too that you can't expect formats to be byte aligned to something nice like 0x4, or 0x10, unlike most game formats. Some files do it, but the models don't tend to.

Some models are very much like DeS and those are supported by my tool actually. 

There's a few format variants to consider, but the unsolved ones all start with a header with 8C0A followed by a variable length buffer size, then a subtype, something else variable length (sometimes a size?), and then a secondary subtype.  The subtypes are pretty important. The 8C0A are also flag information, not just some magic, but for SOTC, this is the only 'unsolved' variation.

To illustrate this better, in this instance we have 8C 0A, a 3 byte size, 89, 3 more bytes, and then 02. You can also see the material name, defaultSG, and the 0SOP, MRN, NAT, and XET representing position, normal, tangent, and texture mapping/UV respectively, but unlike what you'd see for some files, it seems like what's here is compressed or something. 

 That compression is kind of a problem. Really unsure what the deal is there. IIRC, other data in the file isn't compressed and you'll see files like files with 82 instead of 89 in that flag set which are straightforward. 

From what I know, 0x88 and 0x89 are models with this 'compressed' looking header info going on. 0x80 and 0x81 just seem to not have that header info at all. 82 is like DeS.

The secondary subtype flag is typically 0x2 for normal models, but has been observed as 0x5, 0xD and 0x15 for models which explicitly reference another file in the header. These can show. So you can have an 0x89 for instance that has the 0x5 and is actually just referencing another model, but acting as an LOD of sorts. Haven't delved into these a ton, in DeS they use instaLOD, but roll their own system here I haven't looked into a ton.

So in summary:

Convertable right now:

Some of these 0002, 0005, 010A, CC0A, 8C0A * 82 

Problem models:

Anything with 8C0A and primary flag 80, 81, 88, or 89.

Anything with secondary flag 05 (But these should all be LOD models)

 

Haven't looked much at CSKL for SOTC or animations, though IIRC they're different to DeS. Also note DeS's CMDL is really just a bunch of references to CMSH files, materials and junk.

 

I'm not gonna claim it's the prettiest, but my code for all this is here:

https://github.com/Shadowth117/PSO2-Aqua-Library/tree/master/AquaModelLibrary.Data/BluePoint

My debug function for sorting some of these models out is here:

https://github.com/Shadowth117/Aqua-Toolset/blob/96d8160696a3e869782c4e036df95f1070674b20/AquaModelTool/AquaModelToolUI.cs#L4176

And my latest releases are here, in which you can convert what currently works:

https://github.com/Shadowth117/Aqua-Toolset/releases

 

Feel free to throw a PR at me if you manage to figure this out more or just message me in Discord or Twitter or wherever. Would love to get this working, but I have other projects going on and so I probably won't circle back for a long time unless someone else has any ideas. 

 

Edit: Made a slight update just now. I made an update to better handle some things in DeS and didn't go back for SOTC regressions. Some SOTC models should work now, though, namely 0002 ones. Might need to go back and check things again a bit. DeS and SOTC have some fundamental format differences in what is similar that aren't readily visible, but you can do some things to get both to read. 

Like I said though, would definitely appreciate some help if somebody wants to go further here. I think this is at least a good start. Here's a stalagtite model:

Edited March 25 by Shadowth117

[DKDave]

Just a quick observation - all of your sample files have some of these 0x8c0a headers in them.  These signify Oodle Mermaid compression.

The cclm and cskl seem to only be 1 compressed block of data.  The decompressed size is contained in the last 12 bytes of the file along with some type text such as MLOC or LEKS.  These 12 byte blocks of info always seem to follow the compressed data.

The cmsh file seems to be lots of these compressed blocks, but no proper file table to identify their offsets, unless that info is elsewhere.

 

[Shadowth117]

23 hours ago, DKDave said:

Just a quick observation - all of your sample files have some of these 0x8c0a headers in them.  These signify Oodle Mermaid compression.

The cclm and cskl seem to only be 1 compressed block of data.  The decompressed size is contained in the last 12 bytes of the file along with some type text such as MLOC or LEKS.  These 12 byte blocks of info always seem to follow the compressed data.

The cmsh file seems to be lots of these compressed blocks, but no proper file table to identify their offsets, unless that info is elsewhere.

 

That's really interesting to know if it's an oodle compression type, I'll check into that.

That footer you mentioned is just kind of a part of BluePoint's formats, compressed or not. It always corresponds to the extension, but in Demon's Souls they seemingly didn't use compression like that as far as I know so the uncompressed size wasn't much of a factor. 

 

Edit: For reference btw, yeah they're kind of allergic to offsets. It's all kind of inlined, you have to just know what the structs are and read things out. Definitely a bit scary sounding to me for runtime, but pretty cool to see done in its own way. The games manage to be fairly stable despite it.

Edited March 26 by Shadowth117

[Shadowth117]

Okay, yeah. this looks right. Thanks again for the callout! I'll see what's left and try to get something out soon.

Edit: Model in question:

Edited March 26 by Shadowth117

[Shadowth117]

Okay, so I was away on a trip for a few days, but I came back and started working on it again. I actually may have gotten it. I need to test this exhaustively, but... it looks about right. 

I did run into something where some files in the .psarc didn't extract correctly as far as I can tell. Gonna try different methods and see what to recommend. 

One thing I will say is that files reference each other in a way that they're intended to be accessed from a single file structure (Materials can just be in whatever folder relative to the 'root' essentially). So I'm gonna end up recommending the user extracts the contents of the psarcs to Shadow's Image0 folder, to match the layout Demon's Souls PS5 has by default. 

 

[Shadowth117]

Here's a release for now that supports SOTC cmdl files: https://github.com/Shadowth117/Aqua-Toolset/releases

Hope this is useful to someone or another. To quote from my Github:

To use this properly, you should extract the contents of all of the .psarc archives to the same directory as the eboot.bin, that is, cachedFiles.psarc, pak0.psarc, pak1.psarc, pak2.psarc, pak3.psarc, and pakN.psarc. pakN.psarc will need an extractor capable of handling files greater than 32 bit length, due to it being more than 12gb.

I've uploaded a copy of the QuickBMS script, brink.bms, I know works for the .psarc files alongside this. Its credits are included inside, but it seems to be by Komiga on Github, though there's not a repo to my knowledge as it was just uploaded loose on Xentax some time back.

Note that like Demon's Souls PS5, the LOD meshes which references another mesh file at the top of the file are NOT supported, though I don't see them as a priority given they're LODs.

 

While I've uploaded the .psarc script here for mirroring's sake, I implore anyone interested in the model tool to go through the link for the most up to date possible binary.

brink.zip

Edited April 4 by Shadowth117

[Dmnslyrx8]

Before I even attempt to do this myself, does the tool also grab textures also?

edit:
Texture files are also Oodle Mermaid compression, and split into 1 chunk, and then there's some with multiple chunks.

edit2:
So tested the tool out, works pretty well so far! Only thing that bothers me is how meshes are separated, clearly it's not correct, so looks like some mesh groups are either getting combined into 1, or they're set via compressed blocks in the data.

Edited April 6 by Dmnslyrx8

[Shadowth117]

13 hours ago, Dmnslyrx8 said:

Before I even attempt to do this myself, does the tool also grab textures also?

edit:
Texture files are also Oodle Mermaid compression, and split into 1 chunk, and then there's some with multiple chunks.

edit2:
So tested the tool out, works pretty well so far! Only thing that bothers me is how meshes are separated, clearly it's not correct, so looks like some mesh groups are either getting combined into 1, or they're set via compressed blocks in the data.

Yeah I'd been interpreting this off the cmsh material info since there's some stuff that corresponds to counts with the faces or verts per material. For a lot of models in Demon's Souls this seemed sensical though there were a few cases like this. The weird thing with SOTC models though is that it references vertex indices and NOT face indices here while Demon's Souls references actual faces. There's not a lot unaccounted for either so I'm not entirely sure how else I can go about separating the data. If anyone gets ideas on that for this model, I'd be interested to hear.

This is 'wander.cmsh', I assume.

Edit:

I have been looking into texture stuff by the way. The way they note texture resolution is a bit weird. Demon's Souls CTXR seems to denote an aspect ratio along with the buffer size stuff. Not entirely sure what they're doing in SOTC yet, though they're pretty similar. The pixel format for both is based on the first byte/int32, after Oodle decompression.

Edited April 6 by Shadowth117

[Dmnslyrx8]

Yep that's indeed wander.cmsh. wander_lod1 looks almost like his PS2 model from what ive seen lmao

Did notice some models export with broken vertex normals tho for stuff that has no bones or vertex weights.

 

I've also tried to decompress the blocks inside the cmsh and the texture files (so that I can use rawtex to export em), but had no luck, but does seem like ur data for mesh groups is missing, just not sure where at. Tried looking into the cmat + csdr data, but didn't get far into that besides parameter values and the such.

Edited April 7 by Dmnslyrx8

[Quasar728]

I want to give an enormous thank you to @Shadowth117! Your tools were easy to use and I'm finally able to load these models in Blender. You're a hero! I made a quick video demonstrating how to convert a cmdl to fbx and open in Blender. It's very simple but nice to show that it works. Sorry for the terrible sound quality. 

 

[Shadowth117]

On 4/6/2024 at 11:53 PM, Quasar728 said:

I want to give an enormous thank you to @Shadowth117! Your tools were easy to use and I'm finally able to load these models in Blender. You're a hero! I made a quick video demonstrating how to convert a cmdl to fbx and open in Blender. It's very simple but nice to show that it works. Sorry for the terrible sound quality. 

  Glad it's working for you!

On 4/6/2024 at 7:50 PM, Dmnslyrx8 said:

Yep that's indeed wander.cmsh. wander_lod1 looks almost like his PS2 model from what ive seen lmao

Did notice some models export with broken vertex normals tho for stuff that has no bones or vertex weights.

 

I've also tried to decompress the blocks inside the cmsh and the texture files, but had no luck, but does seem like ur data for mesh groups is missing, just not sure where at. Tried looking into the material data, but didn't get far into that besides parameter values and the such.

I think you might be confused on something; after mermaid decompression I don't think there are compressed blocks. Can you explain what you're talking about? For Wander.cmsh all sections are read; I ensured it reads all the way through to the footer. While there likely is some element being misunderstood, the vertices and triangle data don't really leave anything to the imagination. If you breakpoint the cmsh code and debug it, it'll be pretty clear why all this is, but even just reading it there should be enough to understand things quickly just looking at that and the hex data.

 

 

[09williamsad]

I modified Aqua-Toolset for batch usage, then converted all the cmdl and cmsh models.

It seems to have converted all the significant models without issue, or at least I did not see any missing that are notable.

Converted list attached.

 

For textures, the only information I could find is https://reshax.com/topic/21-tool-to-compressdecompress-texture-ctxr-from-shadow-of-the-colossus-ps4/#comment-1538

I tried to use that tool for decompressing some textures, but it exported 0KB files.

SotCPS4ConvertedModels.zip

[Shadowth117]

4 hours ago, 09williamsad said:

I modified Aqua-Toolset for batch usage, then converted all the cmdl and cmsh models.

It seems to have converted all the significant models without issue, or at least I did not see any missing that are notable.

Converted list attached.

 

For textures, the only information I could find is https://reshax.com/topic/21-tool-to-compressdecompress-texture-ctxr-from-shadow-of-the-colossus-ps4/#comment-1538

I tried to use that tool for decompressing some textures, but it exported 0KB files.

SotCPS4ConvertedModels.zip 132.3 kB · 0 downloads

So, the textures are relatively straightforward and even follow the same pixelformat designation as Demon's Souls. Grabbing a buffer and manually converting it isn't that bad actually. I've been looking into simplifying this and having things be able to convert just from the model or ctxr, but grabbing the buffers and converting them properly gets a bit weird. Demon's Souls and SOTC also seem to determine resolution and buffer separation differently. It gets weirder with textures which aren't square as they sometimes get split into squares in the mipmaps.

Now, a lot of times the highest quality texture is in the chunk0.ctxc, but especially in SOTC, they are often as well just in the .ctxr. Either way, I'm still trying to determine how the file itself designates what's going on there and ideally get all of the mipmaps to extract properly. CTXRs can also contain multiple textures, in which case all will use the same mipmap count and same resolutions as far as I'm aware.

SOTC and Demon's Souls have an area after the external texture reference block in the ctxr which seems to contain info on resolution etc in some manner, but it's not straightforward. 

For instnace, in Demon's Souls, you have these for BC1 textures:

 

The c2000 is 2048x2048, the c6050 is 4096x4096, the final is 1024x2048. It's apparent that the 4th byte, with first nibble C, and the 6th byte, with first nibble 8, have a 2nd nibble that together correlate to the aspect ratio and size. You'll notice I said aspect ratio and 'correlates' rather than simply saying multiple of 1024. And that's because we have other instances such as this, which is 3840x2560. A 3/2 ratio, but clearly not an integer multiple of 1024, nor even an integer multiple of 2.

 

But that aside, Shadow is its own beast here, as while it has these values set with likely the same types even, it uses them differently. Here is a square 1024x1024 texture in SOTC:

Though there's familiarity in the numbers used, the aspect ratio pattern is clearly not present. 

I haven't looked as much at SOTC just yet for this, this is pretty recent research, though clearly if they're all by chance just square textures we avoid a lot of the hassle here by simply calculating based on the buffer sizes and pixel block sizes. But I don't know just yet. I'll try to see what I can manage here.

Feel free to shout out if I'm just missing something very obvious.

Edit:

For reference, I've been using daemon's rawtex tool here (I hope it's fine to attach it here), as the textures use ps4 swizzling (demon's souls uses ps5 swizzling clearly). Here's what I have for pixel formats for now. 

RawTex.7z

RawTex is clearly not ideal longterm here so any help into researching how all this works would be appreciated so I can make a more customized tool for the job. The .ctxc files are easy enough to convert since once uncompressed they're simply a texture buffer with a 0xC byte footer (chunk0 is always the largest, in observation), but anything within a .ctxr will need a tad more effort to copy out. 

Note again that SOTC texture files are USUALLY oodle compressed so you'll need to run them through this in Aqua Model Tool or an equivalent for now or RawTex won't be able to read them.

 

Edit 2:

Also also, you can already multi select. So as long as what you want is in the same folder, you don't need to make a whole batch convert functionality. Furthermore, Souls Model Tool, a tool bundled with Aqua Model Tool, already allows this directly for cmdls, so you can simply make .bat commands or drag and drop etc. without modifying anything.

Edited April 13 by Shadowth117

[CharlieV]

Had time to test 3 of them, most seem to work just fine and all are square so far, usually 2k but sometimes 1k. Seems all the fur is generated in real-time using a 'heat map' so the models don't have fur, and they don't bother to have any texture underneath it. c02 has very little fur so it's all perfect, c03 almost the entire body is fur so there's no texture apart from one patch of skin under the right arm and it shows that correctly, but c01's body is all kinds of messed up. There are 3 textures for the body but none of them map correctly, sometimes just a small partial match such as the 3 middle fingers, but unlike c03 that's not how it's supposed to be because the entire face/legs don't work and they're quite clearly one piece in the texture, maybe something wrong with the UV's on that one? 
c01 with the face texture applied but not working

c02 all good

c03 where you can see the fur areas are blanked with a repeating low res pattern but correct skin patch

[Dmnslyrx8]

Well I figured out how to decompress the files... was doing my command prompt all wrong -_-
But yeah, most of the textures are squares, altho there's some that's not squared.

But, it seems one texture has a wrong alpha channel?
ex:  wander_hair_col.chunk0.ctxc

also red channel (roughness map) seems to be a wrong texture for anything related to "wander_cloth_lower_rgh", texture doesn't match up with anything. All other "rgh" textures are fine however. Also Iris texture seems wrong too, it like duplicates itself? Unsure if that's accurate or not, been a few years since I've touched the game.


Edit: Also didn't know ur tool had a oodle decompressor built in.. that would've saved me days of pulling my hair out lmao

Edited April 14 by Dmnslyrx8

[Shadowth117]

6 hours ago, Dmnslyrx8 said:

Well I figured out how to decompress the files... was doing my command prompt all wrong -_-
But yeah, most of the textures are squares, altho there's some that's not squared.

But, it seems one texture has a wrong alpha channel?
ex:  wander_hair_col.chunk0.ctxc

also red channel (roughness map) seems to be a wrong texture for anything related to "wander_cloth_lower_rgh", texture doesn't match up with anything. All other "rgh" textures are fine however. Also Iris texture seems wrong too, it like duplicates itself? Unsure if that's accurate or not, been a few years since I've touched the game.


Edit: Also didn't know ur tool had a oodle decompressor built in.. that would've saved me days of pulling my hair out lmao

I wanna say rgh is 'roughness' and it seems at a glance like the lower matches up okay when separating out the alpha map. The hair does look weird with the alpha to me, but it's hard to really say how it's being used so maybe that's honestly how it's supposed to be, as weird as it is.

 

 

On another note, if you could point out the textures that are NOT square resolutions, that would be very helpful for making something better for this.