Havok middleware

[Maxwell]

Havok has been around since 2000s. It is commonly associated with game physics, however there is more to it than meets the eye.
Havok has and is still used in countless games to this day. It is a almost complete toolset for numerous problematics.

This topic will cover broad solutions so reader experience will be required about:

• Basic knowledge about binary formats
• Basic knowledge about XML format
• How ELF or PE works
• C/C++ data structure layout, member alignment, vtables
• Data endianess
• Virtual memory

So, what Havok covers?

• Animation (presumably discontinued), covers animation algorithms, animation compression and their bindings. It is used to animate transforms, parameters and triggers (aka annotations).
  • Animation classes have hka prefix.
• Behavior (presumably discontinued), is used for blending animations, inverse kinematics and overall animation control. It is a set of state machines and is very similar to Animation Blueprints used in Unreal Engine 4.
  • Behavior classes have hkb prefix.
• AI is used to create and control artificial intelligence (the old one) notably navigation (via nav meshes) and path finding. It can be linked to physics world as it can be used for navigation.
  • AI classes have hkai prefix.
• Destruction is used to generate and work with dynamic destruction and object fracture and deformations. It is similar to Chaos in Unreal Engine 4.
  • Destruction classes have hkd prefix for legacy and hknd, hkcd prefix for new versions.
• Physics is probably most used and most known usage. It's used to define entire world of physics, it's shapes, constraints and algorithms for collisions and physics that ties everything together.
  • Physics classes have hkp prefix for legacy and hknp prefix for new versions.
• Extra stuff, like models for debug rendering, scripting engine and metadata are also present.
  • Extra classes have hkx prefix.

In addition to these above solutions, developers can integrate their custom classes into Havok itself.

Why care?

Havok uses numerous file formats that can store any data and it's entirely up to the developers what format they want use or even if they want to use them at all.
Havok uses at least 5 file formats.

Packfile format

NOTE: In order to fully understand packfile format, basic knowledge about C/C++ language is essential. This format is universal store format that is almost always used.

It's mostly used in a binary form or very rarely in XML form
Packfiles are using sections in order to separate different kind of data.
Most common sections are:

• __classnames__ that store all classes used within packfile, it stores only class name and it's signature hash. It's always used in binary format, never in XML.
• __types__ stores complete class information, their names, base classes, member names and member types. This section is optional and is mostly omitted from packfiles.
• __data__ finally store data values

XML Packfile

Stores data as XML text. This format is platform independent but slow to load compared to it's binary counterpart.
Following code snipped describes XML layout for packfiles.

<hkpackfile>
    <hksection name="__types__">
    <hkobject class="hkClass" name="hkGroupFilterDesc">
        <hkparam name="name">hkGroupFilter</hkparam>
        <hkparam name="declaredMembers" numelements="1">
        <struct>
            <hkparam name="name">nextFreeSystemGroup</hkparam>
        </struct>
        </hkparam>
    </hkobject>
    </hksection>
</hkpackfile>

Packfile always starts with `hkpackfile` node. This node may have additional attributes like packfile version, SDK version, or other metadata.
Only children allowed for `hkpackfile` node are `hksection` with designated name attribute describing section.
Only children allowed for `hksection` node are `hkobject` that will encapsulate data for specified class. When XML is loaded, it will create class instance. Class name attribute is mostly used for referencing so other classes can use it.
Every `hkobject` must have `hkparam` children. Eventually they store all the data as text format. Such data is further parsed and then assigned to designated class member.

For example, above XML packfile will represent data for following class declaration in C++class

struct hkClassMember {
  hkString name;
};

class hkClass {
  hkString name;
  hkArray<hkClassMember> declaredMembers;
};

Binary packfile

This format is a first choice to store data. It's super fast to load and data are in native format. That's why it's first choice and is commonly used.
Under the hood, this format is same as ELF or PE (executables or objects files produced by compilers), because it stores data as memory images.
Packfiles are loaded into virtual memory as a whole and then patched (patching occurs for pointers and class vtables). In one way, they are an extension to the main executable.

NOTE:. If you want to truly understand the complexity of such format, do research on ELF format.

This format have certain limitations, it's platform specific. And I will shortly explain why.
But firstly let's explore file header.

struct hkxHeaderData {
  u32 magic1; // 0x57E0E057
  u32 magic2; // 0x10C0C010
  u32 userTag;
  u32 version;
  u8 pointerSize;
  bool littleEndian;
  bool reuseBaseClassPadding;
  bool emptyBaseClassOptimization;
  s32 numSections;
  s32 contentsSectionIndex;
  s32 contentsSectionOffset;
  s32 contentsClassNameSectionIndex;
  s32 contentsClassNameSectionOffset;
  char contentsVersion[16];
};

Where:

• magic1 and magic2 are two endian independent magic numbers.
• userTag can be any number specified by user, it's metadata and does not affect packfile in any way.
• version is a version of packfile, this does not have affect on data in any way except layout of section and header data.
• pointerSize can be either 4 for 32bit systems or 8 for 64bit systems.
• littleEndian will specify data endianess. If false, big endian is used.
• reuseBaseClassPadding is compiler specific, it is used mostly by GNU GCC compiler. In order to understand do research on data layouts and struct data padding for in C language.

Some examples of common data layouts:

Platform	pointerSize	littleEndian	reuseBaseClassPadding
Windows x64	8	✔	❌
Windows x86	4	✔	❌
XBOX 360	4	❌	❌
PS3	4	❌	✔
Linux x64 or PS4	8	✔	✔

And here comes the biggest limitation of this format. You cannot load packfile from different platform.
For example packfile from PS3 cannot be possibly loaded on Windows x64 platform, because the data layout is different than platform it's being loaded into.

This is it for now. I can provide more info if there are requests for it.

Tagfile

I won't go into depth with this format, since it's very rarely used.
Tagfile are platform independent. They can be as XML or as binary data. XML schema is different than packfile. It must have class type info and in addition does not store default data.
Binary format starts with 0xDO11FACECAB00D1E magic number.
The rest of data is very similar to Profobuf Wire format or JSON binary formats like CBOR, BSON, MsgPack or others.
Again, I won't go into detail unless there is an interest in it.

New format

This format is used since 2015 and replaces packfiles. In it's own sense, it's hybrid format from both packfiles and tagfiles.
Tagfiles are no longer loaded as a whole and data is stored into data chunks instead of sections.

Each data chunk begins with following structure:

struct hkChunk {
    u8 flags;
    u24 size;
    u32 chunkMagic;
};

Where:

• flags describe chunk, there seems to be only 0x40 which indicates either following chunk is not group
• size tells size of entire chunk including hkChunk, data are in big endian.
• chunkMagic is chunk type and is in ASCII

Some chunk types:

• TAG0: Root chunk where everything begins
• SDKV: String value containing SDK version (and possibly layout data)
• DATA: Contains native class data just like in packfiles
• TYPE: Marks groups with class and type descriptions
• TSTR: Type names
• TNAM: Index that is used to build type names
• FSTR: Class member names
• TBOD: Index that is used to build classes or types
• THSH: Class signatures
• INDX: Marks groups with pointer patches for DATA
• ITEM: Index for class entries
• PTCH: Index for pointer patches

Chunks are grouped as:

TAG0 {
    SDKV,
    DATA,
    TYPE {
        TPTR,
        TSTR,
        TNAM,
        FSTR,
        TBOD,
        THSH,
        TPAD,
    },
    INDX {
        ITEM,
        PTCH,
    }
}

Known tools

Project Anarchy

Project Anarchy was official Havok SDK for registered developers durin Intel acquisition era.
It contained Preview Tool to preview 3D shapes and simulate physics.
Content Tools was set of build tools to generate and compile Havok data to numerous formats for all platforms.
And Havok source code with compiled libraries for legacy physics and animations.
It also contained official plugins for Autodesk software like 3ds max or Maya.
Official website is no longer available, but there are backups online. Project ararchy contained versions from 550 to 2014.

HavokLib

Unofficial open source library currenty only for Havok Animation. This library is focused on multi-platform binary packfile loading. It can convert binary packfiles (old and new) to XML. HavokLib also contains cli toolset for conversion of Havok animation data into GLTF format.

HavokLib can be found at: https://github.com/PredatorCZ/HavokLib

HavokMax

Unofficial native plugin for 3ds max that uses HavokLib. Can import binary packfiles (old and new). Can export skeletal and animation data to XML format. HavokMax is no longer maintained and is only as archive.

HavokMax can be found at: https://github.com/PredatorCZ/HavokMax

Detailed info can be found at: https://lukascone.wordpress.com/2019/03/21/havok-3ds-max-plugin

HavokNoesis

Unofficial native plugin for Noesis that uses HavokLib. Can import binary packfiles (old and new). Can export skeletal and animation data to XML format. HavokNoesis is very old, it's possible to update it, if there is legitimate interest.

HavokNoesis can be found at: https://github.com/PredatorCZ/HavokNoesis

Detailed info can be found at: https://lukascone.wordpress.com/2019/05/13/havok-noesis-plugin

This is it for now

I haven't even scraped the surface of this behemoth of a middleware.
It took way too much time than I anticipated to make even this little text.
As I mentioned mutiple times, more research can be added. If there is a demand.
Feel free to share your research as well.
Thank you for reading this.

[michalss]

Very nice

[abso1utezer0]

Thank you for everything you've done, it's definitely helped the Epic Mickey community and I over the past year.

Could you try to look into packfiles with collision data? It's one of the only formats left for Epic Mickey that isn't modifiable. I've been able to write a tool to grab the mesh data and convert it to .obj, but it doesn't actually parse the packfile and is pretty janky.

Also, how would we use this hkx -> xml functionality?

Thanks again.

[Skykila]

Unfortunately HavokToolset was only briefly mentioned. After all, HavokToolset is a very convenient and multifunctional tool. 
https://github.com/PredatorCZ/HavokLib/releases

Edited November 12, 2023 by Skykila

[Grix]

Very useful, would you happen to know more about .HKT files? I'm part of the WWE 2K modding commuity and these control the collision. I've attached a sample.

Would appreciate any input since this is one of the last remaining files that we wish to modify, but hit a roadblock since we're unfamiliar with the format. 

Thanks!

WWE18_01_RAW.zip

[Maxwell]

On 11/11/2023 at 12:53 AM, abso1utezer0 said:

Also, how would we use this hkx -> xml functionality?

It's a library feature. There isn't currently any toolset for convenience as of now. It would be easy to add tho.

 

On 11/11/2023 at 12:53 AM, abso1utezer0 said:

Could you try to look into packfiles with collision data?

I guess you only think of collision shapes? I haven't looked much at collisions or shapes. They use some traditional shapes like spheres, capsules, boxes, convex shapes, k-DOPs, polytopes but also state machines and complicated trees for polytopes.

I have thought about researching it, but it's quite complicated topic. I tried to be more grounded and research only some commonly used animation classes.

 

On 11/12/2023 at 1:11 PM, Grix said:

would you happen to know more about .HKT files

This is a new format. Can be converted to xml. It's for 2019 SDK so I don't know, if library supports it.

[Skykila]

On 11/12/2023 at 4:11 PM, Grix said:

Very useful, would you happen to know more about .HKT files?

DSAnimStudio contains utilities despacito2.exe and despacito_test.exe which convert .HKT to .XML.

[mrmaller1905]

What is MOPP, and what does that stand for?

Edited December 7, 2023 by mrmaller1905

[Maxwell]

On 12/7/2023 at 4:41 PM, mrmaller1905 said:

What is MOPP, and what does that stand for?

It stands for Memory Optimized Partial Polytope (hopefully I remember that correctly).

In a easy way, it's data collection of physics shapes.

In more technical sense, it's compiled code that runs on their custom virtual machine, used for collision detection.

I believe it was removed for new physics.

[Aoba]

Awesome, now I know that platforms are not compatible with each other. I thought 2013-2.5 would work 2013-3.0 because I'm trying to fix a broken goalkeeper for PES2016. But only tools I found are just for different version. I kinda tried to used tools for other like Skyrim/Sonic but it wouldn't work. It will either caused in T-Pose then followed by a game crash.

Edited March 20 by Aoba
hk2013

[Maxwell]

On 1/1/2024 at 3:56 PM, Aoba said:

platforms are not compatible with each other

To put it into a perspective, there are multitude of factors.

1st is data layout must match it's intended platform

2nd is pretty volatile, and that is version. You cannot load data for newer version in older version.

You can load data intended for older version, but there are 2 rules. 1st rule, the application trying to load older data must contain compatibility module/library. 2nd rule, application cannot load deleted classes. This contains either classes from opted out modules, or classes that was removed between versions. For example hkaDeltaComperssedAnimation has been deleted in version 2011, so you cannot load such data in version 2011 and later.

It's pretty complicated and there is no guarantee the app (where you try to load data for older versions) actually contains compat module, since mostly there is no need for it.

So it's always better to stick with exact version no matter what.

To convert data from older versions to newer one, you can use AssetCC tools from project Anarchy.

[Aoba]

On 1/29/2024 at 6:22 PM, Maxwell said:

To put it into a perspective, there are multitude of factors.

1st is data layout must match it's intended platform

2nd is pretty volatile, and that is version. You cannot load data for newer version in older version.

You can load data intended for older version, but there are 2 rules. 1st rule, the application trying to load older data must contain compatibility module/library. 2nd rule, application cannot load deleted classes. This contains either classes from opted out modules, or classes that was removed between versions. For example hkaDeltaComperssedAnimation has been deleted in version 2011, so you cannot load such data in version 2011 and later.

It's pretty complicated and there is no guarantee the app (where you try to load data for older versions) actually contains compat module, since mostly there is no need for it.

So it's always better to stick with exact version no matter what.

To convert data from older versions to newer one, you can use AssetCC tools from project Anarchy.

It's just unfortunate because PES2015-2016 is on Havok 2013-3.0-r1.0 which aren't available anywhere. And converting from old to new won't get recognized by the game either. So I guess there's no way to make custom animations for it because we can only read it.

Edited March 20 by Aoba
havok2013

[Dr. Sheldon Cooper]

Hey guys, sorry for bumping in to the conversation, i have been trying to find for ways to convert from 2013.3.0-r1 to obj if geometry or XML if just text file, and back from XML/OBJ to 2013.3.0-r1 yet i cant find anything from these searches here that would work, no experience with the format, so i am not sure what would i need in order to convert these and if needed to convert back, i included some samples if any of you might happen to know, can please let me know?

https://archive.org/search?query=creator%3A"Havok+technologies"

HKX_2013.3.0-r1_samples.zip

Edited March 19 by Dr. Sheldon Cooper

[Dr. Sheldon Cooper]

I tried to mess with the files once again, i can open them in havok 2014-1-0.

I cant figure out how to convert them to anything else besides XML, or of its even possible to convert them to OBJ, to modify, change or add to the collisions.

If i try to convert them to HKX they files are very large compared to original and also i am unable to convert them to lower version as the originals which is 2013.3.0-r1.

If anyone might happen to have any clues what would I need to do, that'd be great.

[Maxwell]

I am surprised that you can even open those files in a first place.

The samples you posted are not genuine havok files, they contains 16 bytes of some kind of custom data before the actual havok file starts.

I wasn't able to convert those into xml via assetcc, but hctStandAloneFilterManager is able to load them and convert them into xml via Write to Platform filter.

There is no current way to convert shapes into obj, because they are compiled polytopes for collisions anyway. The only way so far is preview tool, that internally converts polytopes into mesh as a debug.

[Dr. Sheldon Cooper]

12 hours ago, Maxwell said:

I am surprised that you can even open those files in a first place.

The samples you posted are not genuine havok files, they contains 16 bytes of some kind of custom data before the actual havok file starts.

I wasn't able to convert those into xml via assetcc, but hctStandAloneFilterManager is able to load them and convert them into xml via Write to Platform filter.

There is no current way to convert shapes into obj, because they are compiled polytopes for collisions anyway. The only way so far is preview tool, that internally converts polytopes into mesh as a debug.

Indeed the first 16 bytes are Dunia engine header on top of the actual HKX file, which i am assuming tells the game executable how to handle it at runtime? I have been scratching my head trying to figure them out in how to deal with them but i think i hit a wall that is why i asked in here.

 

I too tried to convert to XML via assetcc but not a single chance in hell it would work.

You might happen to know a way at all to deal with them besides the preview tool?

Thank you for responding, much appreciated.

 

Edit:

Ah i just noticed here that you already mentioned they where never researched and rather complicated tree, damn, sad to see it was a dead end.

Edited April 17 by Dr. Sheldon Cooper

[Maxwell]

On 4/18/2024 at 1:25 AM, Dr. Sheldon Cooper said:

You might happen to know a way at all to deal with them besides the preview tool?

Like I wrote here:

On 4/17/2024 at 1:28 PM, Maxwell said:

hctStandAloneFilterManager is able to load them and convert them into xml via Write to Platform filter.

There is a Preview Tool and Filter Manager

[Dr. Sheldon Cooper]

6 hours ago, Maxwell said:

Like I wrote here:

There is a Preview Tool and Filter Manager

Yes i have tested that as well as you saw per screenshot above, i kind of figured there might been other ways to deal with that i am unaware of, such collision HKX files other than the preview and to export to XML via filter.

Thank you though.

[mrmaller1905]

Does London Racer: World Challenge use Ipion Virtual Physics (IVP) or Havok, and does the executable contains IVP_Real_Object?

[Maxwell]

On 6/21/2024 at 10:55 AM, mrmaller1905 said:

Does London Racer: World Challenge use Ipion Virtual Physics (IVP) or Havok, and does the executable contains IVP_Real_Object?

Yes, London Racer: World Challenge uses Ipion and it does contain IVP_Real_Object.