Hi, needed help to extract the files inside FILE.CDF with the FILE.CDD as the header/TOCs
UFS.zip
(I actually truncated the FILE.CDF so it can be attached here easily. The original size of it was around 570MB)

These formats is also used by other Silicon Dreams (Gusto's old studio before they were reformed) titles on PS2 like Lego Football Mania/Soccer Mania and UEFA Champions League Season 2001-02

Edited Saturday at 12:30 PM2 days by BloodRaynare

FILE.CDD (the TOC)

The CDD is a tree of nodes. All offsets inside it are relative to the start of the CDD.

• Header: 3 bytes (03 04 03 in UFS, meaning unknown), then the root folder's child list.

• Every node starts with its name in ASCII. The last character is ORed with 0x80 (bit 7 set), which is how the end of the name is marked.

• If the byte right after the name is FE, the node is a file. Anything else means it's a folder.

• Folder: a list of 3-byte big endian offsets to its child nodes, ended by FF.

File entry, after the name:

FE        marker
3 bytes   sector, little endian (offset in CDF = sector * 0x800)
4 bytes   stored size in the CDF, little endian
4 bytes   unpacked size, little endian
            FFFFFFFF = stored as-is
            00000000 = stored as-is (only the sound files: BD/HD/SB/MIB/MSB/WAD)
            anything else = compressed

UFS has 1666 files: 1085 compressed, 489 stored with FFFFFFFF and 92 sound files stored with 0. Files are packed back to back in the CDF, each one starting on a new 0x800 sector.

Compression (custom LZ)

• Every compressed file starts with a 4-byte value. Skip it. Its purpose is unknown.

• Control bits come from 32-bit little endian flag words, read lowest bit first. When a word runs out, the next 4 bytes of the stream are read as a new one, so the flag words sit in between the data bytes.

Tokens:

0      literal: copy 1 byte
1 0    short match: read 1 byte B, distance = 0x100 - B (1..256)
       then read 2 more flag bits (first one is the high bit), length = 2..5
1 1    long match: read 16-bit little endian T
       T == 0      end of stream
       distance = 0x2000 - (T >> 3)
       length   = (T & 7) + 2
       if (T & 7) == 0, read 1 more byte E:
         E & 0x80  distance += 0x2000
         length = (E & 0x7F) + 2
         if (E & 0x7F) == 0, length = next 16-bit little endian value (no +2)

Distances are stored as negative values, so a short match byte of FC means 4 bytes back. The samples have a single 01 byte left over after the end marker.

Other notes

• .FAB files (animations) start with v1.0 and have an URBN tag in the header. .TRE files are small stored files holding a 4-byte value and the path of the matching .FAB.

• Full extractor attached: ufs_cdf_extract.py. Usage: python ufs_cdf_extract.py FILE.CDD FILE.CDF out_folder (add -l to only list the files).

ufs_cdf_extract.py

10 hours ago, Hazza12555 said:

FILE.CDD (the TOC)

The CDD is a tree of nodes. All offsets inside it are relative to the start of the CDD.

• Header: 3 bytes (03 04 03 in UFS, meaning unknown), then the root folder's child list.

• Every node starts with its name in ASCII. The last character is ORed with 0x80 (bit 7 set), which is how the end of the name is marked.

• If the byte right after the name is FE, the node is a file. Anything else means it's a folder.

• Folder: a list of 3-byte big endian offsets to its child nodes, ended by FF.

File entry, after the name:

FE        marker
3 bytes   sector, little endian (offset in CDF = sector * 0x800)
4 bytes   stored size in the CDF, little endian
4 bytes   unpacked size, little endian
            FFFFFFFF = stored as-is
            00000000 = stored as-is (only the sound files: BD/HD/SB/MIB/MSB/WAD)
            anything else = compressed

UFS has 1666 files: 1085 compressed, 489 stored with FFFFFFFF and 92 sound files stored with 0. Files are packed back to back in the CDF, each one starting on a new 0x800 sector.

Compression (custom LZ)

• Every compressed file starts with a 4-byte value. Skip it. Its purpose is unknown.

• Control bits come from 32-bit little endian flag words, read lowest bit first. When a word runs out, the next 4 bytes of the stream are read as a new one, so the flag words sit in between the data bytes.

Tokens:

0      literal: copy 1 byte
1 0    short match: read 1 byte B, distance = 0x100 - B (1..256)
       then read 2 more flag bits (first one is the high bit), length = 2..5
1 1    long match: read 16-bit little endian T
       T == 0      end of stream
       distance = 0x2000 - (T >> 3)
       length   = (T & 7) + 2
       if (T & 7) == 0, read 1 more byte E:
         E & 0x80  distance += 0x2000
         length = (E & 0x7F) + 2
         if (E & 0x7F) == 0, length = next 16-bit little endian value (no +2)

Distances are stored as negative values, so a short match byte of FC means 4 bytes back. The samples have a single 01 byte left over after the end marker.

Other notes

• .FAB files (animations) start with v1.0 and have an URBN tag in the header. .TRE files are small stored files holding a 4-byte value and the path of the matching .FAB.

• Full extractor attached: ufs_cdf_extract.py. Usage: python ufs_cdf_extract.py FILE.CDD FILE.CDF out_folder (add -l to only list the files).

ufs_cdf_extract.py

Just tried the tool and works for the most of the files though some of the compressed one fails, probably due to slightly different compression method
Some example messages of the failed ones:

[67/1666] FAIL ANIMS/CHOREO/GOALCELEBRATIONS/NETBUSTER/AGILITY/GOAL_AGILITY_04.FAB: got 48732 bytes, expected 88540

[68/1666] FAIL ANIMS/CHOREO/GOALCELEBRATIONS/NETBUSTER/FLAIR/GOAL_FLAIR_01.FAB: got 39470 bytes, expected 155996

could you send the full FILE.CDF file

13 hours ago, Hazza12555 said:

could you send the full FILE.CDF file

Sorry just saw your reply anyway here's the full FILE.CDF
https://drive.google.com/file/d/1m7QnmfoRxDwTH_PS8gzowlju4SKu97Y0/view?usp=sharing

Edited 5 hours ago5 hr by BloodRaynare