Glacier 1 *.STR file format

[wssdude]

(Little preamble:) Unsure if I should post each archive format separately... I'll start with this one, as I have it best described atm. I reversed bunch of other audio-related formats of Glacier 1 games so I plan to slowly put them all here. Take this as an appetizer 😛 

Streams files of Glacier 1 games can be read on their own, they contains all of the required data. It should actually be read before any scenes when someone wants to do anything audio-related to have best support, unlike older Glacier 1 games which had streams.wav.

Data in file can be separated into following sections, some have clear indices some can be implicitly inferred:
- header
- block of WAV data (also contains LIP-encoded segments in some data, current exact structure of these is unknown...)
- block of WAV headers (different format than headers in *.WHD files, is much simpler and more concise)
- file name table (file names match those in *.WHD and *.SND files, there are some extras though contained within this file so this is not full subset!)
- records table (start marked in header along with records count)

Block of WAV data seems to be aligned on 0x100 boundary (which coincidentally seems to also be size of header and offset to block of WAV data...). Rest of the file does not seem to have any specific alignment.

Any WAV data may be encoded in LIP segments which have variable length. Header of the LIP chunks seems to have size of 0xF00 or 0x1000 (with first header containing 'LIP ' magic). Each record seems to contain a field which can be checked to see if data contains LIP segments or not without the need to rely on comparing magic of each data block. For distance-based records, you will have to look into master record to see if LIP encoding is used. There may be multiple LIP segments in the data block, but only first one has magic in first four bytes. Due to variable data length, we have to find out the right size of the LIP segment first before parsing. It seems to appear roughly every ~4 seconds, but naive formula of `average byte rate * 4` just roughly yields what the LIP segment is. Therefore, there is some guessing work that has to be done on the algorithm side to extract data properly. If anyone could help with reversing these LIP segments, it would be great! They do not seem to correspond to speech necessarily.

Current detection method for LIP segments relies on the fact that archive is aligned, we know roughly where the offset should be and that we can calculate exact size of each data block (next block offset - current block offset). There is also additional observation to be made that nearly all LIP segments seem to have around half of their data filled with zeroes. We can also notice that when we subtract real data size, aligned on 0x100 boundary, from whole data block size, we get amount of bytes belonging to LIP segments. We can then calculate from this size amount of LIP segments in the data block. There may be only one such segment (calculated size of all LIP segments is <= 0x1000), which does not require us to do any magic - we just have to skip past the header and read real data right after it. Note that the size of the block may be 0xF00 and not 0x1000 so <= and skipping whatever offset you get is probably best course of action until the segments are bit better understood. If there are more segments, we can proceed with calculation of segment size (as the data is interleaved in a way described above).

As mentioned before, we roughly know when each of LIP segments appears in the audio file (it is roughly equivalent to 4 seconds, leaving last block unaligned most of the time with smaller size). We should try to pattern match buffer of size 0x780 filled with zeroes, masking each found offset with ~0xFFF (which will left-align on 0x1000) and taking closest offset to the one we predicted. We then read in minimum from "data block bytes left to read" and this "found LIP segment offset", skip 0x1000 bytes to get "divider offset" for the encoded block and copy each part of the segment into its own buffer. In the end, we are left with complete LIP data and complete WAV data.

Block of WAV data is organized in such a way that it has all non-distance-based entries at the beginning and all distance-based entries at the end. There is no clear block of LIP data, it seems to be mixed randomly in-between all of the entries so no reliable distinction in the block. Distance-based entries point to same data offset, there are always exactly three such pointers (2 defined in *.WHD which have their copy in *.STR file also, 1 is only defined in *.STR file). There cannot be other number of "duplicates" pointing to same data offset than 1 (none) or 3 (distance-based entry, 1 for master and 2 for near/far data). Third entry we mentioned is STR file only, it is the true data definition used by the sound graph. If LIP data is present, master record has appropriate flag set. Note that master record should not really be used for other things, as its parameters are not exactly the same always and correct ones are located directly in the entry. 

TODO - add information about distance-based records structure, it is also interleaved...

Due to all this, recommended way to get to the actual data is to pre-calculate all individual WAV data block sizes and resolve LIP segment sizes for each record along with detecting which records are distance-based. 
 
TODO - add parsing process used by Glacier 1 Audio Tool which seems to have correct export

Note that format information of data, along with data sizes, offsets, names, etc. are all the same as one can find in their equivalent records in *.WHD files. So there is no need to reference *.WHD files for any sort of information for extraction of the *.STR files (unlike older Glacier 1 games).

Below are simple C++ headers which should help anyone interested to get started with the file format I hope!

V1 is for Hitman: Blood Money
V2 is for Kane & Lynch: Dead Men and Mini Ninjas
V3 is for Kane & Lynch 2: Dog Days (TODO - missing information+header!)

//
// Created by Andrej Redeky.
// SPDX-License-Identifier: Unlicense
//
// Extended format information: https://reshax.com/topic/27-glacier-1-str-file-format
//

#pragma once

enum class STR_LanguageID_v1 : uint32_t
{
  Default = 0,
  English = 1,
  German = 2,
  French = 3,
  Spanish = 4,
  Italian = 5,
  Dutch = 6
};

struct STR_Header_v1
{
  char id[0xC] = {'I', 'O', 'I', 'S', 'N', 'D', 'S', 'T', 'R', 'E', 'A', 'M'}; // always "IOISNDSTREAM"
  uint8_t unkC[0x4]; // always seems to be a sequence 09 00 00 00
  uint32_t offsetToEntryTable = 0; // points at the STR_Footer, right after string table ends
  uint32_t entriesCount = 0; // same as number of STR_Data entries in STR_Footer
  uint32_t dataBeginOffset = 0x100; // offset to beginning of data probably, but it is like this even for PC_Eng.str which does not have such size and has no data...
  uint8_t unk1C[0x8]; // always seems to be a sequence 00 00 00 00 01 00 00 00
  STR_LanguageID_v1 languageId = STR_LanguageID_v1::Default; // specifies which language data is contained within the archive
};

enum class STR_DataFormat_v1 : uint32_t
{
  INVALID = 0x00,
  PCM_S16 = 0x02,
  IMA_ADPCM = 0x03,
  OGG_VORBIS = 0x04,
  DISTANCE_BASED_MASTER = 0x11
};

// beware that this is really 3 different headers, as there is no padding... didn't know how to name things so left it like this for now..
struct STR_DataHeader_v1
{
  // PCM_S16, IMA_ADPCM, OGG_VORBIS and DISTANCE_BASED_MASTER have following bytes
  STR_DataFormat_v1 format; // specifies how data should be read
  uint32_t samplesCount; // samples count
  uint32_t channels; // number of channels
  uint32_t sampleRate; // sample rate
  uint32_t bitsPerSample; // bits per sample

  // all PCM_S16, IMA_ADPCM and DISTANCE_BASED_MASTER have following bytes on top
  uint32_t blockAlign; // block alignment

  // all IMA_ADPCM have following bytes on top
  uint32_t samplesPerBlock; // samples per block
};

struct STR_Entry_v1
{
  uint64_t id; // probably some ID, is less than total entries count, does not match its index
  uint64_t dataOffset; // offset to beginning of data, beware of the distance-based records which alias the same index!
  uint64_t dataSize; // data size
  uint64_t dataHeaderOffset; // offset to table containing header
  uint32_t dataHeaderSize; // size of STR_DataHeader_v1 (unused fields from the structure are left out)
  uint32_t unk24; // unknown number
  uint64_t fileNameLength; // length of filename in string table
  uint64_t fileNameOffset; // offset to filename in string table
  uint32_t hasLIP; // 0x04 when LIP data is present for current entry, 0x00 otherwise
  uint32_t unk3C; // unknown number
  uint64_t distanceBasedRecordOrder;  // if 0, entry is not distance-based, otherwise denotes data order of individual records in data block (or is simply non-zero for master record)
};

enum class STR_LanguageID_v2 : uint32_t
{
  Default = 0,
  English = 1,
  German = 2,
  French = 3,
  Spanish = 4,
  Italian = 5,
  Dutch = 6
};

struct STR_Header_v2
{
  char id[0xC] = {'I', 'O', 'I', 'S', 'N', 'D', 'S', 'T', 'R', 'E', 'A', 'M'}; // always "IOISNDSTREAM"
  uint8_t unkC[0xC]; // always seems to be a sequence 09 00 00 00 XX XX YY YY 00 00 00 00 where XX XX changes with language and game and YY YY is same for a game (Kane & Lynch: Dead Man has this sequence E1 46, Mini Ninjas has this sequence 4C 4A)
  uint32_t offsetToEntryTable = 0; // points at the STR_Footer, right after string table ends
  uint32_t entriesCount = 0; // same as number of STR_Data entries in STR_Footer
  uint32_t dataBeginOffset = 0x100; // offset to beginning of data probably, but it is like this even for PC_Eng.str which does not have such size and has no data...
  uint8_t unk24[0x8]; // always seems to be a sequence 00 00 00 00 01 00 00 00
  STR_LanguageID_v2 languageId = STR_LanguageID_v2::Default; // specifies which language data is contained within the archive
  uint8_t unk30[0x8]; // always some sequence 38 XX XX XX XX XX XX XX where XX is same for a game (Kane & Lynch: Dead Man has this sequence 00 A1 01 18 EE 90 7C, Mini Ninjas has this sequence 00 00 00 00 00 00 00)
};

enum class STR_DataFormat_v2 : uint32_t
{
  INVALID = 0x00,
  PCM_S16 = 0x02,
  IMA_ADPCM = 0x03,
  OGG_VORBIS = 0x04,
  UNKNOWN_MASTER = 0x1A
};

// beware that this is really 2 different headers, as there is no padding... didn't know how to name things so left it like this for now..
struct STR_DataHeader_v2
{
  // PCM_S16, IMA_ADPCM, OGG_VORBIS and UNKNOWN_MASTER have following bytes
  STR_DataFormat_v2 format; // specifies how data should be read
  uint32_t samplesCount; // samples count
  uint32_t channels; // number of channels
  uint32_t sampleRate; // sample rate
  uint32_t bitsPerSample; // bits per sample
  uint32_t unk14 = 0;
  uint32_t unk18 = 0;
  uint32_t blockAlign; // block alignment

  // all IMA_ADPCM have following bytes on top
  uint32_t samplesPerBlock; // samples per block
};

struct STR_Entry_v2
{
  uint64_t id; // probably some ID, is less than total entries count, does not match its index
  uint64_t dataOffset; // offset to beginning of data, beware of the distance-based records which alias the same index!
  uint64_t dataSize; // data size
  uint64_t dataHeaderOffset; // offset to table containing header
  uint32_t dataHeaderSize; // size of STR_DataHeader_v2 (unused fields from the structure are left out)
  uint32_t unk24; // unknown number
  uint64_t fileNameLength; // length of filename in string table
  uint64_t fileNameOffset; // offset to filename in string table
  uint32_t hasLIP; // 0x04 when LIP data is present for current entry, 0x00 otherwise
  uint32_t unk3C; // unknown number
  uint64_t unk40;  // OLD INFO: if 0, entry is not distance-based, otherwise denotes data order of individual records in data block (or is simply non-zero for master record)
};

[RubyJabberwocky]

I don't know if this is any help to you, since I don't really know about audio codecs or headers, or hex, or anything about that but...I've kinda managed to get what I think was a clean extraction of Kane & Lynch 2 Dog Days, maybe the info on how I did it could be helpful to you to kinda reverse engineer what the tools did or something?

[wssdude]

On 11/14/2023 at 8:08 PM, RubyJabberwocky said:

I don't know if this is any help to you, since I don't really know about audio codecs or headers, or hex, or anything about that but...I've kinda managed to get what I think was a clean extraction of Kane & Lynch 2 Dog Days, maybe the info on how I did it could be helpful to you to kinda reverse engineer what the tools did or something?

I dont think I need it at this moment (currently rewriting some parts of the tool to have a separate lib), but you may always post what you have found 😛 

[RubyJabberwocky]

On 11/15/2023 at 5:46 PM, wssdude said:

I dont think I need it at this moment (currently rewriting some parts of the tool to have a separate lib), but you may always post what you have found 😛 

I've sent you a Discord DM explaining all the tools that got me kind of somewhere in detail, cause I can't explain it in short for the life of me, and I'm kind of new to the site!