This topic was archived here:

https://web.archive.org/web/20230000000000fw_/https://www.zenhax.com/viewtopic.php?t=1326

0wn3df1x, posted Wed Sep 02, 2015 7:15 pm (7136)

---

Hi there! I need some help with .tim archives from Renowned Explorers.

Example: https://drive.google.com/file/d/0B4gOIO ... ZOVGc/view
Game: http://store.steampowered.com/app/296970/

Ekey, posted Wed Sep 02, 2015 8:12 pm (7138)

---

Fast info

Code:

struct TIMHeader
{
   uint32_t   dwID; //IRUY
   uint32_t   dwVersion; //144
   uint32_t   dwUnknown;
   uint32_t   dwFiles; // Probably
   uint32_t   dwNTableOffset; // Offset for table with names
   uint32_t   dwZero;
   uint32_t   dwNTableSize; // Uncompressed size for names table (876726)
   uint32_t   dwNTableZSize; // Compressed size for names table (76598)
   uint32_t   dwITableOffset; // Offset for table with entry index
   uint32_t   dwZero;
   uint32_t   dwITableZSize; // Compressed size for entry index table (96529) (uncompressed size but it must be > 679464)
   uint32_t   dwUnknown; // 171
   uint32_t   dwXTableOffset; // Another table offset (dunno what this)
   uint32_t   dwZero;
   uint32_t   dwXTableZSize; // Compressed size for another table 1 (33401) (uncompressed size but it must be > 131072)
   uint32_t   dwZero;
   uint32_t   dwYTableOffset; // Another table 2 offset (dunno what this)
   uint32_t   dwZero;
   uint32_t   dwXTableZSize; // Compressed size for another table 2 (279943) (uncompressed size but it must be > 339732)
};

Used lzham_codec as compression.

abbeybas, posted Wed Sep 02, 2015 8:36 pm (7140)

---

Hello from AbbeyGames.

I have no intention of making the files in this archive invisible, so I'll provide the format of our archive format here. What do you want to use it for if I may ask? The file simply contains all our game content there is nothing of real interest in there.

Code:

struct TIMHeader
  {
    // Archive id 'YURI'
    uint32_t id = 'YURI';

    // Size of the archive header
    uint32_t headerSize = sizeof(TIMHeader);

    // Version format of the archive (0 = default)
    uint16_t version = 0;

    // Number of files contained in the archive
    uint32_t fileCount;

    // Offset from the beginning of the archive to the directory listing
    uint64_t directoryListingOffset;

    // The uncompressed size of the directory
    uint32_t directoryListingSize;

    // The compressed size of the directory listing
    uint32_t directoryListingCompressedSize;

    // Offset from the beginning of the archive to the file registry
    uint64_t fileRegistryOffset;

    // Compressed size of the file registry
    uint32_t fileRegistryCompressedSize;

    // Offset from the beginning of the archive to the hashtable
    uint64_t hashtableOffset;

    // Size of the hashtable when compressed
    uint32_t hashtableCompressedSize;

    // Offset from the beginning of the archive to the bucket table
    uint64_t bucketTableOffset;

    // Size of the bucket table when compressed
    uint32_t bucketTableCompressedSize;

    // Directory listing MD5 before compression
    uint8_t directoryListingDigest[16];

    // File registry MD5 before compression
    uint8_t fileRegistryDigest[16];

    // Hash table MD5 before compression
    uint8_t hashtableDigest[16];

    // Bucket table MD5 before compression
    uint8_t bucketTableDigest[16];
  };

  struct TIMHashEntry
  {
    // Offset to the first bucket for the given hash
    uint32_t startBucketIndex;
  };

  struct TIMBucketEntry
  {
    // Hash of the filename with the default hashing algorithm
    uint32_t hash;

    // Hash of the filename with a different algorithm than used for the hash table
    uint32_t otherHash;

    // The index in the file table
    uint32_t fileIndex;
  };

  enum class TIMFileFlags : uint8_t
  {
    kNone = 0,
    kCompressed = 1   };

  struct TIMFile
  {
    // The offset of the file from the beginning of the archive
    uint64_t fileOffset;

    // Size of the file uncompressed
    uint32_t uncompressedSize;

    // Size of the file compressed
    uint32_t compressedSize;

    // Flags for the file (see TIMFileFlags)
    uint8_t flags;
  };

  struct TIMDirectoryInfo
  {
    // The size of the struct including its children
    uint16_t size;
    
    // The number of characters that make up the name of the directory
    uint16_t nameLength;

    // The number of sub directories
    uint16_t numDirectories;

    // The number of files in the directory
    uint16_t numFiles;

    // Followed by nameLength char's
    // Followed by numDirectories TIMDirectoryInfo's
    // Followed by numFiles TIMFileInfo's
  };

  struct TIMFileInfo
  {
    // The number of characters that make up the name of the file
    uint16_t nameLength;

    // Followed by nameLength char's
  };

Have fun! I'll leave the rest up to you as an exercise.. (But just let me know if you want a more thorough description..)

Cheers from Bas@AbbeyGames!

Ekey, posted Wed Sep 02, 2015 8:46 pm (7141)

---

abbeybas wrote:

Hello from AbbeyGames.

I have no intention of making the files in this archive invisible, so I'll provide the format of our archive format here. What do you want to use it for if I may ask? The file simply contains all our game content there is nothing of real interest in there.

Woah! Hi! .. Usually it is necessary for translating game to another languages or just for collecting models / textures etc. Thanks for the detailed information of archive

0wn3df1x, posted Wed Sep 02, 2015 9:02 pm (7142)

---

abbeybas wrote:

Hello from AbbeyGames.

I have no intention of making the files in this archive invisible, so I'll provide the format of our archive format here. What do you want to use it for if I may ask? The file simply contains all our game content there is nothing of real interest in there.

Cheers from Bas@AbbeyGames!

Hello, dear devs! I'm not coder, only translator.

We admire your "Renowned Explorers" project. Our team is looking forward to make a Russian localization for it. We hope you'll get wise to our ambition to make Russian version of your wonderful game. But in order to realize our desire, we need tools. For example, to get the fonts, textures and text from content.tim. (Lockit)

Ekey, posted Wed Sep 02, 2015 9:12 pm (7143)

---

Hmmm... the structure of the header does not match that in the archive

abbeybas, posted Wed Sep 02, 2015 10:19 pm (7146)

---

Ow shoot, yeah we have since moved to version 1 but I forgot to update our documentation. The fileCount however, should match. I parse the header in code simply by reading in the structure specified in the code as raw C memory. There should be around 33000 files in the archive atm.

We have always planned on translating the game and it's set up the be translatable. At this point however there is quite a lot of text in the game and it would be too expensive to do all translations at once. That's why we chose to only do english and in the future add new translations. The archive contains a binary encoded file that contains all the translations for the game. However we'd love it for people to help us out with the translations so if you want to help please send an email to [email protected] and let's set something up!

Cheers!
Bas

Ekey, posted Wed Sep 02, 2015 10:27 pm (7147)

---

abbeybas wrote:

Ow shoot, yeah we have since moved to version 1 but I forgot to update our documentation. The fileCount however, should match. I parse the header in code simply by reading in the structure specified in the code as raw C memory. There should be around 33000 files in the archive atm.

Oh. Ok

abbeybas wrote:

We have always planned on translating the game and it's set up the be translatable. At this point however there is quite a lot of text in the game and it would be too expensive to do all translations at once. That's why we chose to only do english and in the future add new translations. The archive contains a binary encoded file that contains all the translations for the game. However we'd love it for people to help us out with the translations so if you want to help please send an email to [email protected] and let's set something up!

You could make the individual archives with language data's like: language_en.tim, language_fr.tim, language_ru.tim etc. and load them first by priority depending on the selected language in Steam. I know one team that will help translate game to Russian language!

aluigi, posted Thu Sep 03, 2015 12:04 pm (7150)

---

The version field is padded to 32bit (perfectly normal for struct on 32bit systems without pragma pack) but there are a couple of new fields, one after fileRegistryCompressedSize (0xab) and another after hashtableCompressedSize (0).

The compression algorithm doesn't match because lzham produces a file of 0 bytes.
I tried comtype lzham of quickbms, I tried also older versions of quickbms with same result.
Ekey, what lzhma API did you use for the decompression?

Ekey, posted Thu Sep 03, 2015 2:09 pm (7158)

---

aluigi wrote:

Ekey, what lzhma API did you use for the decompression?

They use 10.8.1

aluigi, posted Fri Sep 04, 2015 4:10 pm (7167)

---

ok, I guess I use an old version in quickbms. going to update it in the next quickbms

aluigi, posted Sat Sep 05, 2015 11:34 am (7197)

---

No Ekey, the latest version is the same with DEST_BUF_TOO_SMALL while using lzham_decompress and -3 with lzham_z_uncompress.
clog "out.dat" directoryListingOffset directoryListingCompressedSize directoryListingSize

Can you show what are you exactly using?

aluigi, posted Sat Sep 05, 2015 12:18 pm (7205)

---

If I use LZHAM_DECOMP_FLAG_READ_ZLIB_STREAM I get error 11 (LZHAM_DECOMP_STATUS_FAILED_BAD_RAW_BLOCK) and the correct output size but the output is all zeroes.

Ekey, posted Sat Sep 05, 2015 1:07 pm (7213)

---

Well I did some tests.

Table_Compressed_Original.dat - Compressed table from TIM archive
Table_Decompressed_Original.dat - Decomrpessed table
Table_Compressed_lzhamtest.dat - Compressed by lzhamtest_x86 with comp level 4

As you can see Table_Compressed_lzhamtest.dat it differs from the original only in the existing header with 13 bytes.

aluigi, posted Sat Sep 05, 2015 2:51 pm (7215)

---

How you got Table_Decompressed_Original.dat ?

Ekey, posted Sat Sep 05, 2015 2:52 pm (7216)

---

aluigi wrote:

How you got Table_Decompressed_Original.dat ?

Debugger

aluigi, posted Sat Sep 05, 2015 3:14 pm (7218)

---

That explains why I have this problems, I guess they use a fixed dictionary or similar.
I have made all the possible tests and only using the unbuffered option I get at least this:

Code:

TEST 7 33 0 (76598 876726)
  b6 60 00 00 0b 00 1b 00 71 00 05 00 00 00 08 00   .`......q.......
  61 72 69 61 6c 09 ff ff ff ff ff ff ff ff ff ff   arial...........
  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................
  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff   ................

The 0xff you see there mean that the output is only 0x16 bytes so there is something invalid and is not caused by the API or quickbms.

aluigi, posted Sat Sep 05, 2015 3:18 pm (7219)

---

@abbeybas any explanation?

Ekey, posted Sat Sep 05, 2015 3:32 pm (7220)

---

If lzham modified I do not see sense to continue

aluigi, posted Sat Sep 05, 2015 6:24 pm (7226)

---

I agree.
Considering that it was planned to be a 100% documented format, it's taking too much time and effort.

The only positive thing is that the next quickbms will have a correct implementation of lzham.

zespri, posted Wed Sep 09, 2015 12:24 am (7324)

---

Ekey wrote:

As you can see Table_Compressed_lzhamtest.dat it differs from the original only in the existing header with 13 bytes.

Well, I personally can't see that. These two files in the archive are very different. There are indeed 26 bytes in common after the first extra 13, but after that I cannot see any resemblance. The size is also quite different.

abbeybas, posted Mon Sep 14, 2015 8:42 am (7470)

---

My apologies for leaving this thread for so long.

So indeed we use 10.8.1. We did not modify the original algorithm.

To decompress we use the following flags:

Code:

decompressParams.m_struct_size = sizeof(lzham_decompress_params);
decompressParams.m_dict_size_log2 = dictionarySize;
decompressParams.m_table_update_rate = LZHAM_FASTEST_TABLE_UPDATE_RATE;
decompressParams.m_decompress_flags = 0;
decompressParams.m_num_seed_bytes = 0;
decompressParams.m_pSeed_bytes = nullptr;
decompressParams.m_table_max_update_interval = 0;
decompressParams.m_table_update_interval_slow_rate = 0;

We use variable sized compression dictionaries. In all places where compression is used the dictionarySize is calculated based on the size of the input with:

Code:

std::min(std::max(floor_to_pow_2(uncompressedSize), LZHAM_MIN_DICT_SIZE_LOG2), 25);

floor_to_pow_2 is defined as:

Code:

// http://graphics.stanford.edu/~seander/bithacks.html#IntegerLogLookup
uint32_t floor_to_pow_2(uint32_t v)
{
  static const char LogTable256[256] =
  {
#define LT(n) n, n, n, n, n, n, n, n, n, n, n, n, n, n, n, n
    - 1, 0, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
    LT(4), LT(5), LT(5), LT(6), LT(6), LT(6), LT(6),
    LT(7), LT(7), LT(7), LT(7), LT(7), LT(7), LT(7), LT(7)
#undef LT
  };

  unsigned int t, tt; // temporaries

  if ((tt = (v >> 16)) != 0)
    return (t = (tt >> 8)) ? 24   LogTable256[t] : 16   LogTable256[tt];

  return (t = v >> 8) ? 8   LogTable256[t] : LogTable256[v];
}

I hope this helps you guys!

aluigi, posted Mon Sep 14, 2015 8:52 am (7472)

---

Thanks abbeybas, that explains why only the first bytes where correctly extracted.
It's the first time I used lzham and I guess that field can be easily brute forced to guess the correct value for the extraction.
I will try. thanks

zespri, posted Mon Sep 14, 2015 9:26 am (7473)

---

Just tried that, it works. Brute forcing is not likely to work as sometimes you do not get with wrong dictionary size, just gibberish. It could work if we always knew the exact uncompressed size but it looks that it is not always known. Thanks.