weiyun, posted Wed Apr 05, 2017 12:46 pm (21985)

---

warrantyvoider wrote:

made your code into a browser, works nice sofar!



PS: the game also loads the decrypted files, so thats it, from here on I just need to reuse my DAI code^^ (or anyone else his fav. frostbite source)

PPS: I can use my old DAI code almost 1:1, lol

Good job! I also tried to understand the data formats.
It seems layout.toc is the root entry point.

Code:

list superbundles: All superbundle name which points to .sb and .toc file. 
list fs: All fs files which is a virtual file system, it's the same format as toc files. In mea there is only one entry: initfs_Win32.
int head: Don't know what it's for.
object installManifest: It contains all cat package info, like id, name, superbundles.

The .toc file is basically pointers to .sb files.

Code:

list bundles
{
string id;
int offset;
int size;
} //the offset and size points to the corresponding .sb file.
list chunks
{
Guid id;
byte[] sha1;
} // chunk info (only chunks0.toc and loc.toc have this list.
bool cas;
string name;
bool alwaysEmitSuperbundle;

I also made a visualizer for those structs.

warrantyvoider, posted Wed Apr 05, 2017 1:39 pm (21987)

---

yes, this is true, here the general loading idea:

layout toc contains all superbundle names (.sb files)
normal toc, list the bundles inside a sb
normal toc also may reference chunks by sha1
if toc has property cas=true, then the sb file contains the bundle meta data, if its false, chunkdata is stored in the sb file
bundles contain ebx,res and chunks, all which have references by sha1, take that and look through all cat files for that sha1
the cat file then has the offset and size inside the cas file for it, thats where the actual data is

greetz

warrantyvoider, posted Wed Apr 05, 2017 7:18 pm (22000)

---

added support for reading raw chunk data, you can select a single chunk or a list of chunks...



and load them into the VFS, where you can preview them or export



greetz WV

Release.rar

Devisaur, posted Wed Apr 05, 2017 7:34 pm (22001)

---

warrantyvoider wrote:

added support for reading raw chunk data, you can select a single chunk or a list of chunks...



and load them into the VFS, where you can preview them or export



greetz WV

So this maybe just be me but everytime I try to run the application it cycles. The file manager freezes and nothing happens. I can't even find the process in task manager to end. Is it possible you'd be willing to upload the Visual Studio project?

OClear, posted Thu Apr 06, 2017 1:50 am (22003)

---

Instructions on how to use Explorer tool?

How to load toc/sb and cat/cas files all at the same time?
I can only load one file type, not all. Tool resets itself when trying to load more files.

warrantyvoider, posted Thu Apr 06, 2017 3:30 am (22008)

---

See, thats why i should not release programs before i spend a year making tuts for it. No i dont wanna research, code and then also explain everything!

Yes, you first load all tocs by selecting the game exe, then follow the 2 screenshots above

OClear, posted Thu Apr 06, 2017 3:40 am (22009)

---

I don't have any game executable. Just the files posted here by OP. toc/sb and cas/cat are in different folders.
How to get Explorer tool to understand this without game executable?

warrantyvoider, posted Thu Apr 06, 2017 3:51 am (22010)

---

aaah, ok, in that case create a folder, put all of them in that and create an empty textfile called f.e. mea.exe or whatever. just if you dont have all CAT/CAS files (in subfolders) the probability to find a specific chunk by its sha1 is small

PS: attached my current version, I moved the CAT stuff from VFS (=Virtual File System= stuff in games memory) to FS (=FileSystem=real existing files) where it makes more sense

PPS: here a list of all files in the gamedirectory and its subfolders https://pastebin.com/SBrHmbN0

Release.rar

OClear, posted Thu Apr 06, 2017 5:19 am (22011)

---

Great, thanks! Everything under FS (cat and toc) now gets loaded.

michalss, posted Thu Apr 06, 2017 5:35 am (22012)

---

warrantyvoider wrote:

aaah, ok, in that case create a folder, put all of them in that and create an empty textfile called f.e. mea.exe or whatever. just if you dont have all CAT/CAS files (in subfolders) the probability to find a specific chunk by its sha1 is small

PS: attached my current version, I moved the CAT stuff from VFS (=Virtual File System= stuff in games memory) to FS (=FileSystem=real existing files) where it makes more sense

PPS: here a list of all files in the gamedirectory and its subfolders https://pastebin.com/SBrHmbN0

You have been good with DAI in the past, any chace to get text out of the archives ? Looks compressed and encrypted

warrantyvoider, posted Thu Apr 06, 2017 6:01 am (22013)

---

I had no time yet to look at the actual chunk contents, but as far as ive seen, each entry is missings its header and I dont see no magic for encryption anywhere. dunno, im busy with the two filesystems before I can look at content data

greetz

PS: about cas data:


cat tells you to load a region from cas, then start reading blocks, with this 8 byte header. cat files can also contain a second list, where one entry is 80 bytes and has additional 20 bytes (in compare to a normal entry) that look like encryption keys again... (because no block header to see when reading where it says there, and also theres no cas number as far as I can see)

weiyun, posted Thu Apr 06, 2017 9:33 am (22015)

---

warrantyvoider wrote:

I had no time yet to look at the actual chunk contents, but as far as ive seen, each entry is missings its header and I dont see no magic for encryption anywhere. dunno, im busy with the two filesystems before I can look at content data

greetz

PS: about cas data:


cat tells you to load a region from cas, then start reading blocks, with this 8 byte header. cat files can also contain a second list, where one entry is 80 bytes and has additional 20 bytes (in compare to a normal entry) that look like encryption keys again... (because no block header to see when reading where it says there, and also theres no cas number as far as I can see)

cat file 010 Editor Template

Code:

uint flag; //0x01CED100
int zero;
byte signaturedata[0x224]; //rsa signature info, 0x224 bytes
byte nyan[0x10]; //NyanNyanNyanNyan, 0x10 bytes
int list1count;
int list3count;
int list2count;
byte padding[0xc];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown;
int cas_num;
} list1[list1count];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown1;
int unknown2;
int size2;  // size2 should equal to size
byte unk1[0x28];
} list2[list2count];
struct {
byte unknown[0x3c];
} list3[list3count];

cas is compressed in chunks using zstandard
format:

Code:

int decompressedsize;
ushort flag?;
ushort compressedsize;
byte[] compresseddata; //compressed data starts with magic header 0xfd2fb528

Decompression code

Code:

        [DllImport("libzstd.dll")]
        static extern UIntPtr ZSTD_decompress(IntPtr dst, int dstCapacity,
                              IntPtr src, int compressedSize);

        public static byte[] Decompress(byte[] buffer, long decompressedsize)
        {
            byte[] decbuffer = new byte[decompressedsize];
            int offset = 0;
            var cmphandle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
            var decmphandle = GCHandle.Alloc(decbuffer, GCHandleType.Pinned);
            using (BinaryReaderEx br = new BinaryReaderEx(new MemoryStream(buffer), Endianness.Big))
            {
                while (br.BaseStream.Position                 {
                    int framedecsize = br.ReadInt32();
                    short flag = br.ReadInt16();
                    ushort framesize = br.ReadUInt16();
                    int decsize = (int)ZSTD_decompress(decmphandle.AddrOfPinnedObject()  offset, decbuffer.Length,cmphandle.AddrOfPinnedObject()   (int)br.BaseStream.Position, (int)framesize);
                    offset  = decsize;
                    br.BaseStream.Position  = (int)framesize;
                }
            }
            cmphandle.Free();
            decmphandle.Free();
            return decbuffer;
        }
        

I decompressed the texture data with the correct size, but the pixel format seems wrong.

warrantyvoider, posted Thu Apr 06, 2017 9:51 am (22017)

---

*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?

weiyun, posted Thu Apr 06, 2017 10:36 am (22019)

---

warrantyvoider wrote:

*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?

Only cat in the patch directory has list3. It's related to resource patching.

Code:

struct {
byte sha1_1[0x14];
byte sha1_original[0x14];
byte sha1_replacement[0x14];
} list3[list3count];

Where sha1_1 is unique. I think it serves as the id of this entry.
sha1_original is only found in original package.
sha1_replacement is found in list1 of the same package.

weiyun, posted Thu Apr 06, 2017 10:40 am (22021)

---

weiyun wrote:

warrantyvoider wrote:

*SPAM*, really? zstandart? I tried so many others, damn^^ thank you, I will include this in my code, cat/cas structure was known, just not the algo

greetz

PS: have you seen a CAT with entries in the 3rd list?

Only cat in the patch directory has list3. It's related to resource patching.

Code:

struct {
byte sha1_1[0x14];
byte sha1_original[0x14];
byte sha1_replacement[0x14];
} list3[list3count];

Where sha1_1 is unique. I think it serves as the id of this entry.
sha1_original is only found in original package.
sha1_replacement is found in list1 of the same package.

list1 of cat file has a lot of duplicated chunks, both within the same cat and across different cats.
list2 chunk seems encrypted.

OClear, posted Thu Apr 06, 2017 10:54 am (22022)

---

weiyun wrote:

Code:

struct {
byte unknown[0x3c];
} list3[list3count];

ayainstallpackage\cas.cat has (72999, 0, 31) entries. This third list has 80 bytes each, not 60.

weiyun, posted Thu Apr 06, 2017 11:00 am (22023)

---

OClear wrote:

weiyun wrote:

Code:

struct {
byte unknown[0x3c];
} list3[list3count];

ayainstallpackage\cas.cat has (72999, 0, 31) entries. This third list has 80 bytes each, not 60.

Note that list3 count appears before list2count.

Quote:

int list1count;
int list3count;
int list2count;

warrantyvoider, posted Thu Apr 06, 2017 11:12 am (22024)

---

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)

FartingSquirrel, posted Thu Apr 06, 2017 11:34 am (22025)

---

Devisaur wrote:

FartingSquirrel wrote:

Hey guys!
As I saw, you all were talking about the textures of the game, but is there anything about the text files for localization purposes? Did someone try to get them?

Not so far -- my main goal is to pull all the text files. I'd imagine what would need to occur is something similar to what they did for Dragon Age Inquisition(http://daitools.freeforums.org/). However it's a multiple step process to break it all down and I have yet to find a source that explains what they're doing/how they're doing it or even comments their code.

Well, it's good that someone's trying. Good luck with it.

Trip, posted Thu Apr 06, 2017 2:25 pm (22027)

---

Code:

int decompressedsize;
ushort flag?;
ushort compressedsize;
byte[] compresseddata; //compressed data starts with magic header 0xfd2fb528

the flag is the compression format as noted by swinei on 1st page of this thread

warrantyvoider, posted Thu Apr 06, 2017 3:03 pm (22028)

---

is there a list of which flag is what compression?

weiyun, posted Thu Apr 06, 2017 3:53 pm (22029)

---

warrantyvoider wrote:

is there a list of which flag is what compression?

I figured out the list2 is compressed using zstandard and then encrypted using AES 128 CBC Mode. That's why there is extra bytes added to the chunk. But I didn't look into key generation details yet.

warrantyvoider, posted Thu Apr 06, 2017 4:47 pm (22030)

---

weiyun wrote:

I figured out the list2 is compressed using zstandard and then encrypted using AES 128 CBC Mode. That's why there is extra bytes added to the chunk. But I didn't look into key generation details yet.

how did you figured out its AES 128 CBC? debugging? and what extra bytes do you mean? the ones at the end of the 80 byte entries in cat or the actual data in cas?

anyway, I added the compression algo to my program and can read all list1 entries fine, so all those chunks can be read and exported just fine now


Flags from what ive seen sofar

Code:

0x0F70 = ZStandard compressed
0x0071 = Uncompressed, not last block
0x0070 = Uncompressed, last block

greetz WV

Release.rar

weiyun, posted Thu Apr 06, 2017 5:57 pm (22031)

---

warrantyvoider wrote:

how did you figured out its AES 128 CBC? debugging? and what extra bytes do you mean? the ones at the end of the 80 byte entries in cat or the actual data in cas?

I debugged the exe file using x64dbg. It seems the key is fixed, but not hardcoded. I'm not sure if game uses other keys for decryption, but I didn't find any exception either.
The decrypted data is normal compressed chunk. The actual encrypted chunk data is padded to multiples of 16 since AES is a block cipher.

Decrypt code:

Code:

public static byte[] Decrypt(Stream s, int size)
        {
            byte[] key = { 0x91, 0xC2, 0x1D, 0xC7, 0x65, 0x95, 0x0E, 0xB0, 0xC3, 0x38, 0xD7, 0x3D, 0xA6, 0xD7, 0x4B, 0x1C };
            var aes = AesCryptoServiceProvider.Create();
            aes.Mode = CipherMode.CBC;
            aes.KeySize = 128;
            using (CryptoStream cs = new CryptoStream(s, aes.CreateDecryptor(key, key), CryptoStreamMode.Read))
            {
                byte[] res = new byte[size];
                cs.Read(res, 0, size);
                return res;
            }

        }

It seems the conversation file has another layer of compression.
And the texture format code 0x42 doesn't look like DXT5.

OClear, posted Thu Apr 06, 2017 7:27 pm (22035)

---

This is just raw texture data from chunk with no header. Where is texture info?

256x256, DXGI_FORMAT_BC7_UNORM_SRGB (format 99)

weiyun wrote:

It seems the conversation file has another layer of compression.
And the texture format code 0x42 doesn't look like DXT5.

Where does "texture format code 0x42" come from?