warrantyvoider, posted Thu Apr 06, 2017 7:53 pm (22036)

---

weiyun wrote:

I debugged the exe file using x64dbg. It seems the key is fixed, but not hardcoded. I'm not sure if game uses other keys for decryption, but I didn't find any exception either.
The decrypted data is normal compressed chunk. The actual encrypted chunk data is padded to multiples of 16 since AES is a block cipher.

Decrypt code:

Code:

public static byte[] Decrypt(Stream s, int size)
        {
            byte[] key = { 0x91, 0xC2, 0x1D, 0xC7, 0x65, 0x95, 0x0E, 0xB0, 0xC3, 0x38, 0xD7, 0x3D, 0xA6, 0xD7, 0x4B, 0x1C };
            var aes = AesCryptoServiceProvider.Create();
            aes.Mode = CipherMode.CBC;
            aes.KeySize = 128;
            using (CryptoStream cs = new CryptoStream(s, aes.CreateDecryptor(key, key), CryptoStreamMode.Read))
            {
                byte[] res = new byte[size];
                cs.Read(res, 0, size);
                return res;
            }

        }

this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?

I tried:
offset:0x02EC3AC7
size:0x11F6C0
path:streaminginstall\ayainstallpackage\cas_02.cas

taken from the single 80 bytes entry at the end of the cat file in the same folder

I attached my cutout for compare

weiyun, posted Thu Apr 06, 2017 8:15 pm (22037)

---

warrantyvoider wrote:

this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?

I tried offset 0x3df791f, size 0x111df0 in mecdefaultinstallpackage\cas_01.cas and it works fine.
Maybe different bundles use different keys.

Rick, posted Thu Apr 06, 2017 10:54 pm (22039)

---

Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.

OClear, posted Fri Apr 07, 2017 12:57 am (22040)

---

warrantyvoider wrote:

PS: attached my current version, I moved the CAT stuff from VFS (=Virtual File System= stuff in games memory) to FS (=FileSystem=real existing files) where it makes more sense

How to clear data under VFS -> CHUNKS? "Load selected item to VFS" keeps on adding chunks every time it's used, and doesn't delete the old ones.

weiyun, posted Fri Apr 07, 2017 1:49 am (22041)

---

Rick wrote:

Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.

Thanks for the info. Here is the yaml file. Can you find the key location in game exe? I didn't find any hardcoded keys.

Code:

--- # encrypt embargoed assets (YAML format)
start:
 # Unique keys for each text language
 - key: 91c21dc765950eb0c338d73da6d74b1c
   keyid: d66a3e06
   filters:
     - game/localization/config/texttable/en/.*
 - key: ff67f81f63309f430756fa5ee8a49cf6
   keyid: 13f2f24d
   filters:
     - game/localization/config/texttable/br/.*
 - key: 09059e69d726ba9cc6cae5e7ea87959f
   keyid: e4b3b4dd
   filters:
     - game/localization/config/texttable/de/.*
 - key: 940fb798bcbc72375201ed143ab2338a
   keyid: 4608ff24
   filters:
     - game/localization/config/texttable/es/.*
 - key: 559813d02d2d498bda5c03161b82aba8
   keyid: 1e7a889c
   filters:
     - game/localization/config/texttable/fr/.*
 - key: 51724f6c718da5041214c3239c23a463
   keyid: 759bed13
   filters:
     - game/localization/config/texttable/it/.*
 - key: 216dad98a17d11e4b0175f1bc0828037
   keyid: 707deb43
   filters:
     - game/localization/config/texttable/pl/.*
 - key: ac7bb33878a14cc66daf80ff94b8089f
   keyid: 1d171a1b
   filters:
     - game/localization/config/texttable/ru/.*
 # Encrypt certain assets which are only available in the full game to prevent access to people playing the trial
 - key: 9d68ee1b349b8eb6ef4cf37197e1604d
   keyid: 827e55c8 
   filters:
     # M2
     - game/levels/crit/crit_vlt/crit_vlt.*

Here is the revised template.

Code:

//--------------------------------------
//--- 010 Editor v6.0.2 Binary Template
//
// File: Frostbite Cat
// Author: weiyun
// Revision:
// Purpose:
//--------------------------------------
uint flag; //0x01CED100
int zero;
byte signaturedata[0x224]; //rsa signature info, 0x224 bytes
byte nyan[0x10]; //NyanNyanNyanNyan, 0x10 bytes
int normalChunkCount;
int patchChunkCount;
int encryptedChunkCount;

byte padding[0xc];
struct {
byte sha1[0x14];
int offset;
int size;
int unknown;
int cas_num;
} normalChunk[normalChunkCount];
struct {
byte sha1[0x14];
int offset;
int size;
int zero;
byte casnum;
byte unknown[3];
int size2;  // size2 should equal to size
char keyid[8];
byte unk1[0x20];
} encryptedChunk[encryptedChunkCount];
struct {
byte sha1[0x14];
byte sha1_original[0x14];
byte sha1_patch[0x14];
} patchChunk[patchChunkCount];

weiyun, posted Fri Apr 07, 2017 3:35 am (22043)

---

OClear wrote:

Where does "texture format code 0x42" come from?

It's in the itexture header chunk, its sha1 is located in the sb file.

warrantyvoider, posted Fri Apr 07, 2017 4:32 am (22044)

---

weiyun wrote:

warrantyvoider wrote:

this seems not to work for me, can you tell me an offset&size in a casfile, where this works for you?

I tried offset 0x3df791f, size 0x111df0 in mecdefaultinstallpackage\cas_01.cas and it works fine.
Maybe different bundles use different keys.

I can verify this works for me too, so yay, the keys are not install specific. also thank you again for the working code btw, you got pm

Rick wrote:

Yes, crypto key used is based on the "key id" in the catalog. The keys are hardcoded into the game exe, but you can find them in initfs_Win32, see Scripts/CasEncrypt.yaml.

hi rick, its been a while^^ thanks for the info!

OClear wrote:

How to clear data under VFS -> CHUNKS? "Load selected item to VFS" keeps on adding chunks every time it's used, and doesn't delete the old ones.

you cant yet, the problem is, theres no such thing as multiselect in treeview controls, so Ill code an extra dialog for that, but thats not my focus now, sry, just restart for now ?\_(?)_/?

OClear, posted Fri Apr 07, 2017 11:24 am (22046)

---

warrantyvoider wrote:

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)

I have problems decompressing with zstd library. I get error "Unknown frame descriptor". I compile zstd library with legacy support and still get same error. My sample starts with magic id, so I don't know why it fails.

warrantyvoider, posted Fri Apr 07, 2017 12:46 pm (22048)

---

OClear wrote:

warrantyvoider wrote:

yeah, from the 80 byte entries, the last 20 bytes are probably the encryption key. btw your zstd code doesnt work as it is, but if I put it into a file with ending .zst (starting with zstandard header) it can decompress it fine (f.e. zstd -d test.zst)

I have problems decompressing with zstd library. I get error "Unknown frame descriptor". I compile zstd library with legacy support and still get same error. My sample starts with magic id, so I don't know why it fails.

this code assumes you give it compressed chunk data (with 0x0F70 header)

Code:

using ZstdNet;
public static byte[] DecompressChunk(byte[] data)
        {
            MemoryStream m = new MemoryStream(data);
            MemoryStream res = new MemoryStream();
            while (m.Position < m.Length)
            {
                m.Seek(6, SeekOrigin.Current);
                ushort size = (ushort)(m.ReadByte() * 0x100   m.ReadByte());
                byte[] buff = new byte[size];
                byte[] buff2 = null;
                m.Read(buff, 0, size);
                try
                {
                    using (var decompressor = new Decompressor())
                    {
                        buff2 = decompressor.Unwrap(buff);
                    }
                }
                catch { }
                if (buff2 != null)
                    res.Write(buff2, 0, buff2.Length);
            }
            return res.ToArray();
        }

https://github.com/skbkontur/ZstdNet

warrantyvoider, posted Fri Apr 07, 2017 3:38 pm (22050)

---

back home and back coding^^ started by making initFS browseable too



EDIT1:

Yay, I can load entire bundles in my VFS and display them! now I need to find one that uses encryption...



EDIT2:

now I can also decrypt the data on the fly and display it too, this is one of these "embargoed" items... (data\win32\loctext\en.toc -> .../tl_dlc -> load only res in bundle)
so now it automatically: decrypts, decompresses and puts all the pieces back together!



EDIT3: attached my current version; added an option to clear the VFS without restart; if you click on the top node in the structure view of a bundle and press F1 (shortcut for loading stuff), it will load the entire bundle into VFS. same goes for all ebx/res/chunk if you click a node named like that and single import if you pick one of its children nodes

PS: also thanks to everyone that contributed code and information so far, you will be added to the credits later, definitly!

Release.rar

lyutor1945, posted Fri Apr 07, 2017 9:02 pm (22053)

---

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

Devisaur, posted Fri Apr 07, 2017 9:08 pm (22054)

---

lyutor1945 wrote:

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by DawnlessSky who as said they don't plan on working on it.*

*Edited to fix initial mistake.

lyutor1945, posted Fri Apr 07, 2017 9:57 pm (22057)

---

Devisaur wrote:

lyutor1945 wrote:

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by WarrantyVoider who happens to be one spearheading, along with a few other people, the work in this forum.

Oh, sorry! Didn't know it. Looks like I'm goof:) Can I ask you another question then. WarrantyVoider and DawnlessSky it is one and the same person or different people?

Devisaur, posted Fri Apr 07, 2017 10:02 pm (22058)

---

lyutor1945 wrote:

Devisaur wrote:

lyutor1945 wrote:

Hi guys! How are you? Has anyone tried to consider DAI mod Tool? Both games on the Frostbite engine, and may be able to create a similar tool for MEA based on the DAI mod tool:)

DAI Mod Tool is made by WarrantyVoider who happens to be one spearheading, along with a few other people, the work in this forum.

Oh, sorry! Didn't know it. Looks like I'm goof:) Can I ask you another question then. WarrantyVoider and DawnlessSky it is one and the same person or different people?

I stand corrected I'm conflating two separate people -- but the answer to your original question can be found http://dawnatwork.tumblr.com/ --
"Hello!! I'm sure this questions has already been asked a hundred times, but will there be an update for DAI Tool or similar tool for Mass Effect Andromeda?
lyutor1945things

Not from me, no. Others will probably make one though."

warrantyvoider, posted Fri Apr 07, 2017 10:21 pm (22059)

---

simple answer, dawnless used my code and research, and made her own tool without mentioning me (afaik), so that tool came after DAItoolsWV (btw, the same goes for ehams tool, forgot how he called his)

greetz

crackedmind, posted Fri Apr 07, 2017 10:55 pm (22060)

---

warrantyvoider wrote:

back home and back coding^^ started by making initFS browseable too



EDIT1:

Yay, I can load entire bundles in my VFS and display them! now I need to find one that uses encryption...



EDIT2:

now I can also decrypt the data on the fly and display it too, this is one of these "embargoed" items... (data\win32\loctext\en.toc -> .../tl_dlc -> load only res in bundle)
so now it automatically: decrypts, decompresses and puts all the pieces back together!



EDIT3: attached my current version; added an option to clear the VFS without restart; if you click on the top node in the structure view of a bundle and press F1 (shortcut for loading stuff), it will load the entire bundle into VFS. same goes for all ebx/res/chunk if you click a node named like that and single import if you pick one of its children nodes

PS: also thanks to everyone that contributed code and information so far, you will be added to the credits later, definitly!

I'm writting my own tool with C , using you algo. While debugging, i see, you are not correctly parsing .toc files. You suggest that types 0x82, 0x87 is completly different, but it's not true.

I found some information here. Type splits into flags and typecode. 0x82 - Flag = 0x4 Type=0x2
Flag 4 means that we need to skeep reading null terminated name for this field. Also, requiredChunks after that parses as list of 0x0F types (0x8F with flag).


Sample from my json dump

Code:

        {
            "alwaysInstalled": false,
            "estimatedSize": 871620188,
            "friendlyName": 0,
            "id": "??\u0012O\u0019?%8??#?CTu",
            "installBundle": "Win32/streaminginstall/ayainstallpackage",
            "license": "",
            "mandatoryDLC": false,
            "maxSizeMB": -1,
            "name": "streaminginstall/ayainstallpackage",
            "optionalDLC": false,
            "requiredChunks": [
                "7?\t???\r??r??w??",
                "????\u0012C;??\\????\u0006",
                "GBWT ? ?m\u0015????L",
                "n??5?????\"??\u001b?S",
                "n??5?????;??\u001b?S",
                "n??5?????7??\u001b?S",
                "q??(I?T???[??[",
                "q??(I?T???[????[",
                "q??(I?T???[????[",
                "u????bu[\u001f?%???? ",
                "u????bu[\u001f?%???? ",
                "u????bu[\u001f?%???? ",
                "?\u0002??\u0012C???\u0012qX??d\u0010",
                "??f\u0001d???Q??",
                "??f\u0001d???Q??\u0001`?J\u0010",
                "??f\u0001d???Q??\u001fu?J\u0010",
                "???v6?8??}aFl?C",
                "??S?m?r\u001b????Z?2X",
                "??S?t?r\u001b????Z?2X",
                "??S?x?r\u001b????Z?2X"
            ],
            "saveLocked": true,
            "superbundles": [
                "Win32/game/levels/hubs/hub_aya_unc/hub_aya_unc_level",
                "Win32/game/levels/crit/crit_vlt/crit_vlt",
                "Win32/game/levels/hubs/hub_aya/hub_aya_level"
            ],
            "testDLC": false
        }

p.s. thanks to weiyun for decrypt code.

OClear, posted Sat Apr 08, 2017 1:08 am (22062)

---

warrantyvoider wrote:

this code assumes you give it compressed chunk data (with 0x0F70 header)

Code:

WORD w1;         // 0x100 or zero
WORD w2;
WORD comp_type;   // 0x700F
WORD w3;
// zstd magic id follows

-snip-
Solved.

weiyun, posted Sat Apr 08, 2017 1:23 am (22063)

---

OClear wrote:

Does 0x100 mean there is an extra 256 bytes in the beginning? Dictionary?
Or maybe multi-part data? Or just unknown flags?

No, It's big endian, the actual value is 0x10000, which is the uncompressed chunk size.

weiyun, posted Sat Apr 08, 2017 1:35 am (22064)

---

warrantyvoider wrote:

simple answer, dawnless used my code and research, and made her own tool without mentioning me (afaik), so that tool came after DAItoolsWV (btw, the same goes for ehams tool, forgot how he called his)

greetz

I ported your DAI code for stringtable and it works. Game uses TTF as font file this time.

OClear, posted Sat Apr 08, 2017 2:04 am (22065)

---

weiyun wrote:

OClear wrote:

Does 0x100 mean there is an extra 256 bytes in the beginning? Dictionary?
Or maybe multi-part data? Or just unknown flags?

No, It's big endian, the actual value is 0x10000, which is the uncompressed chunk size.

Ah, that makes sense. I didn't know what he was doing here:

Code:

 ushort size = (ushort)(m.ReadByte() * 0x100   m.ReadByte());

He could have read it as a big-endian word instead. It works out the same.

Is that the reason my decompressor is failing? Is the zstd compressed data written in big-endian format too?
CAT file LE, CAS file BE, game programmers can't write consistent data.

Rick, posted Sat Apr 08, 2017 2:45 am (22066)

---

OClear wrote:

Is that the reason my decompressor is failing? Is the zstd compressed data written in big-endian format too?

Are you passing it too much/too little data?

Edit: to clarify, I've yet to have any issues with decompression of any data via Zstd.

https://github.com/gibbed/Gibbed.MassEffectAndromeda/blob/master/projects/Gibbed.Frostbite3.Unbundling/CompressionHelper.cs#L82

(note: my code is very much WIP)

OClear, posted Sat Apr 08, 2017 3:37 am (22067)

---

Thanks, I didn't realize the zstd compressed data was split into multiple parts (failed when size >= 65536). I think I can get it working now.

OClear, posted Sat Apr 08, 2017 10:59 am (22070)

---

crackedmind wrote:

Sample from my json dump

Do you have a tool/script that can convert .sb files into readable json format?

crackedmind, posted Sat Apr 08, 2017 11:59 am (22071)

---

OClear wrote:

Do you have a tool/script that can convert .sb files into readable json format?

Not yet finished. I Planning to finish today.

warrantyvoider, posted Sat Apr 08, 2017 1:47 pm (22072)

---

crackedmind wrote:

I'm writting my own tool with C , using you algo. While debugging, i see, you are not correctly parsing .toc files. You suggest that types 0x82, 0x87 is completly different, but it's not true.

I found some information here. Type splits into flags and typecode. 0x82 - Flag = 0x4 Type=0x2
Flag 4 means that we need to skeep reading null terminated name for this field. Also, requiredChunks after that parses as list of 0x0F types (0x8F with flag).


Sample from my json dump

Code:

        {
            "alwaysInstalled": false,
            "estimatedSize": 871620188,
            "friendlyName": 0,
            "id": "??\u0012O\u0019?%8??#?CTu",
            "installBundle": "Win32/streaminginstall/ayainstallpackage",
            "license": "",
            "mandatoryDLC": false,
            "maxSizeMB": -1,
            "name": "streaminginstall/ayainstallpackage",
            "optionalDLC": false,
            "requiredChunks": [
                "7?\t???\r??r??w??",
                "????\u0012C;??\\????\u0006",
                "GBWT ? ?m\u0015????L",
                "n??5?????\"??\u001b?S",
                "n??5?????;??\u001b?S",
                "n??5?????7??\u001b?S",
                "q??(I?T???[??[",
                "q??(I?T???[????[",
                "q??(I?T???[????[",
                "u????bu[\u001f?%???? ",
                "u????bu[\u001f?%???? ",
                "u????bu[\u001f?%???? ",
                "?\u0002??\u0012C???\u0012qX??d\u0010",
                "??f\u0001d???Q??",
                "??f\u0001d???Q??\u0001`?J\u0010",
                "??f\u0001d???Q??\u001fu?J\u0010",
                "???v6?8??}aFl?C",
                "??S?m?r\u001b????Z?2X",
                "??S?t?r\u001b????Z?2X",
                "??S?x?r\u001b????Z?2X"
            ],
            "saveLocked": true,
            "superbundles": [
                "Win32/game/levels/hubs/hub_aya_unc/hub_aya_unc_level",
                "Win32/game/levels/crit/crit_vlt/crit_vlt",
                "Win32/game/levels/hubs/hub_aya/hub_aya_level"
            ],
            "testDLC": false
        }

you made me realize I have to rewrite that entire code for my tool, which im at currently, so thanks for the info


my reading class sofar, let me know if I do something wrong

Code:

public static class FBJSON
    {
        public class Field
        {
            public byte flags;
            public byte type;
            public bool hasName;

            public string name;
            public ulong size;
            public object data;

            public Field(Stream s)
            {
                byte b = (byte)s.ReadByte();
                flags = (byte)(b >> 5);
                type = (byte)(b & 0x1f);
                hasName = (flags & 4) == 0;
                if (hasName) name = Helpers.ReadNullString(s);
                long start = s.Position;
                switch (type)
                {
                    case 1://list
                        size = Helpers.ReadLEB128(s);
                        data = new List();
                        while (s.Position - start                             ((List)data).Add(new Field(s));
                        s.ReadByte();
                        break;
                    case 2://dict
                        size = Helpers.ReadLEB128(s);
                        data = new Dictionary();
                        while (s.Position - start                         {
                            Field obj = new Field(s);
                            ((Dictionary)data).Add(obj.name, obj);
                        }
                        s.ReadByte();
                        break;
                    case 6://bool
                        data = s.ReadByte() == 1;
                        break;
                    case 7://binary string
                        size = Helpers.ReadLEB128(s);
                        data = Helpers.ReadNullString(s);
                        break;
                    case 8://int32
                        data = Helpers.ReadInt(s);
                        break;
                    case 5://unknown
                    case 9://int64
                        data = Helpers.ReadLong(s);
                        break;
                    case 0xf://uuid
                        data = new byte[0x10];
                        s.Read((byte[])data, 0, 0x10);
                        break;
                    case 0x10://sha1
                        data = new byte[0x14];
                        s.Read((byte[])data, 0, 0x14);
                        break;
                    case 0x13://payload blob
                        size = Helpers.ReadLEB128(s);
                        data = new byte[size];
                        s.Read((byte[])data, 0, (int)size);
                        break;
                }
            }


            public TreeNode[] ToNodes()
            {
                List result = new List();
                TreeNode t;
                int count = 0;
                switch (type)
                {
                    case 1:
                        foreach (Field f in (List)data)
                        {
                            t = new TreeNode((count  ).ToString());
                            t.Nodes.AddRange(f.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 2:
                        foreach (KeyValuePair pair in ((Dictionary)data))
                        {
                            t = new TreeNode(pair.Key);
                            t.Nodes.AddRange(pair.Value.ToNodes());
                            result.Add(t);
                        }
                        break;
                    case 6:
                        result.Add(new TreeNode((bool)data ? "true" : "false"));
                        break;
                    case 7:
                        result.Add(new TreeNode((string)data));
                        break;
                    case 8:
                        result.Add(new TreeNode(((int)data).ToString("X8")));
                        break;
                    case 5:
                    case 9:
                        result.Add(new TreeNode(((long)data).ToString("X16")));
                        break;
                    case 0xf:
                    case 0x10:
                    case 0x13:
                        result.Add(new TreeNode(Helpers.ByteArrayToHexString((byte[])data)));
                        break;
                }
                return result.ToArray();
            }
        }

        public static List ReadFields(Stream s)
        {
            List result = new List();
            long len = s.Length;
            while (s.Position                     result.Add(new Field(s));
            return result;
        }
    }

EDIT: there are more types, stay tuned^^