This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=3508

dissectionalone, posted Thu Dec 08, 2016 8:46 pm (19367)

---

Hi guys,

First time posting here.

Is there anyone with coding knowledge willing to create a parser of some sort or a script to import a seemingly unknown (more like unspecified) 3d format? I'm assuming the format must be some form or variation of XSI (associated with Softimage XSI) but I can't say that for sure.

The reason I'm assuming it must be XSI (or something related) is because the file is from the Xbox 360 game Project Gotham Racing 4.

Luigi Auriemma kindly created a script that can extract the contents of the pak_hrd and some of the .pak files.

The .pak files' content I've seen maintains it's original file extensions and are mostly text files.

The .pak_hrd are given the .dat extension by quickbms (these are the ones I suspect to be XSI or XSI related)

Any help would be highly appreciated.

Many thanks in advance

Cheers

dissectionalone, posted Tue Jan 03, 2017 11:24 pm (19937)

---

Here's the BMS extracted file in case anyone is interested or just curious

https://www.mediafire.com/folder/syra7z ... _Extracted

Regards

Acewell, posted Wed Jan 04, 2017 2:34 am (19941)

---

this doesn't look like any xsi file i have ever seen, i can't see any obvious
patterns or data that make sense either maybe is encrypted or something.

id-daemon, posted Wed Jan 04, 2017 7:33 am (19942)

---

Nothing that bad. Data is compressed with zlib. Separate data blocks for geometry, faces and textures.

Acewell, posted Wed Jan 04, 2017 9:23 am (19944)

---

yep you can run it through offzip to get uncompressed headerless files

Code:

offzip -a 0000000000000000.dat c:\offzip\extracted

dissectionalone, posted Wed Jan 04, 2017 11:08 am (19945)

---

Many thanks for the replies, guys

I'll look into it

dissectionalone, posted Wed Jan 04, 2017 12:36 pm (19946)

---

A noob question: How did you determine that specific file's extension? Or did you take the data and made a png file from the contents of the .dat archive?

Acewell, posted Thu Jan 05, 2017 4:28 am (19950)

---

dissectionalone wrote:

Or did you take the data and made a png file from the contents of the .dat archive?

yes that, the texture data was typical big-endian and tiled dxt for X360. texture data is easy to identify
when you look at enough of them. there was no header so i used file size to determine the dimensions
and then experimented with formats from there and used Noesis to convert it png so i could post it here.

dissectionalone, posted Thu Jan 05, 2017 2:13 pm (19960)

---

I've been trying to figure out ways of identifying all those data blocks, trying to extract that Lamborghini Diablo model but had no luck so far. Those headless binary files feel like advanced chinese

id-daemon, posted Thu Jan 05, 2017 5:27 pm (19964)

---

all "header" information is stored in .dat file itself, between ZLIB compressed blocks. Coordinates are not float, but shorts probably. Faces done with tri-strips with FFFF terminators. I just took a quick look at that.

dissectionalone, posted Thu Jan 05, 2017 7:03 pm (19967)

---

id-daemon wrote:

all "header" information is stored in .dat file itself, between ZLIB compressed blocks. Coordinates are not float, but shorts probably. Faces done with tri-strips with FFFF terminators. I just took a quick look at that.

I'll try to dig up the info from there. Thanks for the info

dissectionalone, posted Mon Jan 09, 2017 8:50 pm (20040)

---

I've tried looking at the original file, without using the offzip command to extract all the data and generate a bunch of headless .dat files, and tried searching for certain strings using a hex editor (you know trying to locate some standard format reference like DDS or FBX) and found a couple but I'm not sure if those are correct or actual meaningful references that can Id the formats.

Acewell, posted Mon Jan 09, 2017 9:49 pm (20043)

---

in 0000000000000000.dat it looks like each file has an uncompressed 51 byte header preceeding each compressed block
i guess a really good bms script could write out the headers and append them to the decompressed file data.
there might be a flag which tells what type of file it is
the last 8 bytes of each header has the uncompressed size and compressed size in big endian byte order

dissectionalone, posted Mon Jan 09, 2017 11:15 pm (20044)

---

That flag might probably be what makes quickbms decide what file extension to give the extracted content right?
Could you elaborate a bit on that idea of the headers with a screenshot, please? Just so I can make sure I'm following.

Acewell, posted Mon Jan 09, 2017 11:34 pm (20045)

---

well you would set script to read the flag in the header and give certain files a better extension.
when you use offzip to extract the files they are named by their starting offset, 000046e3.dat
is the first file extracted at 0x46e3 and if you go back 51 bytes in the main dat file you are at
the start of the header for that file at 0x46b0

dissectionalone, posted Tue Jan 10, 2017 12:06 am (20046)

---

So the first byte at 0x46e3 which has a 78 that indicates it's zlib compressed? Or am I confusing it? Shouldn't the beginning of the header have an indication of what type file it is?

Mondraconus, posted Fri Aug 18, 2017 11:26 am (25677)

---

Acewell wrote:

yep you can run it through offzip to get uncompressed headerless files

Code:

offzip -a 0000000000000000.dat c:\offzip\extracted

00082fc6.dat.png

Sorry if I post this too late, but if may I asking to you how do you convert that converted bms .dat file to .png ? I have been using the way you told me, by using offzip, but it not convert to .png ? How you change it to .png file ? Can you tell me step by step to do ? Thank you

Regards

Acewell, posted Fri Aug 18, 2017 3:32 pm (25697)

---

i explained that already in this post
viewtopic.php?p=19950#p19950

QuickBMS and offzip just extract the data, i used Noesis to open and convert the image data.

Mondraconus, posted Fri Aug 18, 2017 6:58 pm (25703)

---

Acewell wrote:

i explained that already in this post
viewtopic.php?p=19950#p19950

QuickBMS and offzip just extract the data, i used Noesis to open and convert the image data.

I tried it by using noesis, but it can't preview it with a error dialogue box appearing... Any help ?

Mondraconus, posted Sat Aug 19, 2017 4:04 am (25723)

---

Mondraconus wrote:

Acewell wrote:

i explained that already in this post
viewtopic.php?p=19950#p19950

QuickBMS and offzip just extract the data, i used Noesis to open and convert the image data.

I tried it by using noesis, but it can't preview it with a error dialogue box appearing... Any help ?

OK, first I would say thanks to Acewell for introducing a simple way to extract and convert to PNG file. I really loved this game, but I need texture of areas inside it. I need shop sign, poster, anything signs that have unique itself to some places, like Japanese signs in Tokyo, Francais signs in Quebec, or Chinese signs in Macau. I'll give you one example of extracted .DAT files from Macau_sunny.PAK file, maybe you can rip one of them ( not all ), and show the PNG to me here please ?

What I've been done for this until I get error :

1. Using QuickBMS to extract Macau_sunny.pak file inside to "Extract" folder.
2. Copy "00000019.dat" file inside AC00 folder ( in Extract folder ) to Offzip folder.
3. Then running offzip to extract 00000019.dat file, but return error ( see "Error using offzip.png" attachment ). Dunno why, so I pick another example.

4. Now, I using the same way from step 1 to AlfaRomeo_SZ.pak_hrd file.
5. Copy "00000000.dat" file inside SCNE folder ( in Extract folder ) to Offzip folder.
6. Then running offzip to extract 00000000.dat file, it success now ( see "Offzip success for 00000000.dat file.png" attachment ).
7. Now, opening Noesis to convert by using batch process from .DAT to .PNG file, it also can't see the preview.
8. It don't doing anything, says "Nothing was exported" ( see "batch process.png" attachment ).
9. It just created an empty folder with the same name as the batch file, but no PNG or any file inside it ( see "batch result.png" attachment ).

OK, that's it. I don't know how to make it converted as yours to PNG file. I also attach these file ( "00000019.dat" and "00000000.dat" ) to here, maybe you can try to see these files until it converted to PNG like yours before please ? If you success try it, can you explain again step by step to me what I'm doing wrong before, so next I can do it by myself. Thank you.

I just need a confirmation if this .DAT file can be ripped and I can see some signs of the area in this file...

Acewell, posted Sat Aug 19, 2017 4:18 pm (25736)

---

there is no plugin to to open these *.dat files or magically detect the type of data, it has to be done manually.
offzip gives the *.dat files no useful file names and they are headerless.
it is up to each person to open the dat files in a hex editor and identify the data type,
some are image data, some are face indices, some are vertex blocks etc...

your 00000019.dat sample looks like vertex data for a track model to me.


your 00000000.dat file looks like it has not been run through offzip yet. i still see some compressed blocks

for the png example i converted before, i had to open some dats in a hex editor and selected one with image data,
then i had to input information into a temp python script by hand to begin trial and error regarding width and height
based on file size, then the format based on known X360 tiled formats. the best i can do to help you further is provide
the temp script which is not automated, you have to input the values for trial and error which means you still have to
teach yourself how to identify the data through experience. i hope this was clearer.

Mondraconus, posted Thu Aug 24, 2017 4:12 am (25825)

---

Acewell wrote:

there is no plugin to to open these *.dat files or magically detect the type of data, it has to be done manually.
offzip gives the *.dat files no useful file names and they are headerless.
it is up to each person to open the dat files in a hex editor and identify the data type,
some are image data, some are face indices, some are vertex blocks etc...

your 00000019.dat sample looks like vertex data for a track model to me.


your 00000000.dat file looks like it has not been run through offzip yet. i still see some compressed blocks

for the png example i converted before, i had to open some dats in a hex editor and selected one with image data,
then i had to input information into a temp python script by hand to begin trial and error regarding width and height
based on file size, then the format based on known X360 tiled formats. the best i can do to help you further is provide
the temp script which is not automated, you have to input the values for trial and error which means you still have to
teach yourself how to identify the data through experience. i hope this was clearer.

Okay, provide me the temp script. I'll try it. Yup. That's the hardest, I think I must see some tutorial on youtube how to identify the data since hex has many ANSI code rows that's hard to translate into one. Dunno how you identify it, maybe there a special ANSI code or Hex block that I must see in ? I'll never give up because this the only way I can get Macau textures.

Assetto corsa, Raceroom has also the Macau track, even the track is almost same like Project Gotham Racing 4 Macau track, but loss many textures inside. Project Gotham Racing 4 has Macau track detailed more. That's why I'll never give me for this one.

Regards

Acewell, posted Thu Aug 24, 2017 4:55 am (25826)

---

Mondraconus wrote:

Okay, provide me the temp script.

okay here ya go

this Noesis python texture script goes in noesis\plugins\python folder.
you have to set the width (line 16), height (line 17) and format (line 15) in the script,
there are seven X360 formats listed for testing, enter the number of the format to test!

Mondraconus wrote:

Dunno how you identify it

the best i can tell you is open a bunch of known textures in a hex editor and inspect the patterns closely, and do the
same with a bunch of different known model files too, your pattern recognition skills will get better the more you see them.

tex_dat.zip

Mondraconus, posted Thu Aug 24, 2017 5:57 am (25828)

---

Quote:

Mondraconus wrote:

Dunno how you identify it

the best i can tell you is open a bunch of known textures in a hex editor and inspect the patterns closely, and do the
same with a bunch of different known model files too, your pattern recognition skills will get better the more you see them.

If I may ask you again, do you remember what the image file do you get from the last .dat that you convert to .png before ( dds, png, dxt, or else ) ? I already see the tutorial, according to it I must use "find" command and type image format in there ( dds, png, or else ) and if I find it in the bunch of HEX at the right, that means .dat contains image data depend on what format I typing in.

PS : thanks for the code, now I'm still tried to search from the hex editor for the correct image file one. Hope I can find it and soon I can extract and convert it to .png format as you do before.

Regards