This has been an odd one. When I try to use Inverse Storage (ZYX) on the TBN vectors of a file I'm analyzing with Advanced mesh reaper: eXtreme Edition (AXE), it completely misreads the data. For example, if I were to tell it to read the highlighted vector data in the fifth column from the attached image, telling it to start at the offset 5E71 with byte data type, inverse storage, and a 20 byte stride, it will read the first set like normal - 81, 79, D4 - but then insist that all other following sets start with the last byte it processes of the first set, D4. 

So the Data Interpretation readout gives me this for Normals as described above:
0: -1.000000 0.952756 -0.346457 -> 2.027776

1: -0.346457 0.039370 0.149606 -> 0.143964

2: -0.346457 0.291339 0.795276 -> 0.837374

3: -0.346457 0.228346 -0.228346 -> 0.224316

4: -0.346457 0.795276 0.149606 -> 0.774878

5: -0.346457 0.228346 0.425197 -> 0.352967

6: -0.346457 0.448819 0.803150 -> 0.966520

This leaves me having to make a copy of the file and manually swap two bytes of every TBN vector just to try and test it in non-inverse form - a very time consuming task. And if I'm wrong about which three bytes of each set of four are the XYZ/ZYX of the vector, I'm going to have to do the swapping process all over again with different pairs of bytes.

If it's worth anything, this is what the Digest with info about things like the bounding box says:

616 normals read.

start address: 0x5e71

end address: 0x8e80

bounding:

-1.000000 -1.000000 -1.007874

-0.346457 0.992126 1.000000

pivot:

-0.673228 -0.003937 -0.003937

Do I need to manually set a bounding box based on data somewhere in the file? Or am I right that inverse storage is just bugged?

Hi, no one can answer your question(s) (except the author of AXE himself, Bigchillghost) if you don't upload the example file/bytes block you worked with.

At least give the bytes from 0x5E60 up to 0x5F00 for easy copying. (No one likes to 'create' 160 bytes from your screenshot manually.)

Of course, of course! My apologies. 
 I hope this helps. The verts and UVs are interwoven into one block of "X, Y, Z, U, V" repeating starting at 0x10 with float data type and strides of 20, the face indices are down at 0xBCD0 as shorts or "UINT16", and the TBN vectors are four bytes each (one "blend" byte followed by ZYX based on our comparing which ones change most across mirrored pairs of verts) starting at 0x5E70, though that one blend byte seems to make it so you have to put the offset one byte later at 0x5E71. Their stride is 20 in order to skip over the eight bytes of occlusion/vert color data. A separate MDLB file for the models often has strings naming parts of the data that can be found in the MDG file, and it always lists "normal", "binormal", and "tangent" in that order, though we'll have to see if that holds true once this issue is figured out.

This is the same Ga'hoole game I've discussed before, and Krome studios seemed to make all the models to-scale with actual owls so you're going to want to either scale them up or prepare to zoom in a LOT to see them at all. I found a scale of 250 on all axes was a good starting point. All you need to render is one submesh - for stiff armor like this they put all the LOD models in one file as separate submeshes.
a001_helm_enemy.mkmesh.mdg.zip

Edited February 11Feb 11 by gryphongirl

Thanks!

Quote

start address: 0x5e71

Why address 0x5E71 and not 0x5E70? Typo?

Where did you put it? Vertex attributes, normals address box?

Stride 20 and data type word,  right?

I can't seem to get your values:

Quote

0: -1.000000 0.952756 -0.346457 -> 2.027776

(where the -1.0 is confusing to me)

Edited February 11Feb 11 by shak-otay

1 hour ago, shak-otay said:

Thanks!

Why address 0x5E71 and not 0x5E70? Typo?

Where did you put it? Vertex attributes, normals address box?

Stride 20 and data type word,  right?

I can't seem to get your values:

(where the -1.0 is confusing to me)

Data type Byte, or possibly Char, for the normals. We're almost certain based on the strings in the MDLB that the file contains all three of normals, binormals, and tangents. Someone mentioned that one byte of each four could be something related to "blend", so I tried to focus on which bytes changed for certain pairs of symmetrical vertices to understand which corresponded to XYZ, and from that we believe inverse storage was needed. We suspected the first byte of each four is the "blend" - in this case, that first 0A - so the XYZ or ZYX would be one byte further back at 0x5E71, which I put in the normals address box. With inverse storage, that -1.000000 would be from the byte "81" at 0x5E73.

Edited February 11Feb 11 by gryphongirl

1 hour ago, gryphongirl said:

Data type Byte, or possibly Char, for the normals.

 

Quote

and manually swap two bytes of every TBN vector

Ok, misunderstood, what you did. You swapped x and z, right?

Can you provide a plc file (Parameter list)?

 

Edited February 11Feb 11 by shak-otay

I think I also may have forgotten what should have been an obvious mention that this is Big Endian as well ^^'. I'm only doing positions and normals so far as an example of the weird way it handles the data for normals. Also including what I see in the preview - because so many of them have their X forced to be the same, they end up with a "combed over" look.

 

ACK, hang on, just realized that had it set to Char from testing something. My bad! This is how it looks with Byte normals: still messed up with data not read right.

Thanks, and yeah, it was the char thingie!

(Endianness doesn't  matter for TBN  vectors with byte/char type , btw.)

For "the data not read right": is there any documentation for AXE's TBN (other than the AXE_Demo.pdf, where 'inverse storage' isn't mentioned)?

Can it be that AXE "expects" 3 bytes for a TBN component, not 4?

Quote

but then insist that all other following sets start with the last byte it processes of the first set, D4

Strange. And why do normals have negative components for type byte?

For char it's clear, when the most significant bit is set, the value is negative.

How did you convert 0x81, 0x79, 0xD4 for unsigned byte values?

div by 256?

 

Edited February 11Feb 11 by shak-otay

I haven't seen any such documentation, unfortunately. I'm also not sure if it expecting 3 bytes instead of 4 could affect it - I have no way of telling it to read 4 vs. 3, and if I don't tell it to use inverse, it reads the data fine without replacing all other X values with normal 0's Z, which implies that "expectation" shouldn't cause the glitch if they both have it.

The only other thing I can think of is its mention of a "bounding box" - notice how the second set of coordinates for the box in each instance has a first "X" value that matches the Z of normal #0 and the X it forces all the other normals to have. I wonder if there's something weird about the code for how the bounding box is set up when inverse storage is used, and that forces all the values to stay within the reaches of the box? There's a way to tell it to read certain data at an offset in the model file as the bounding box. I might have to experiment with setting that to something unrestrictive like -1, -1,, -1 and 1, 1, 1 to see how it turns out.

(Without 'inverse storage'.) How do you get 0.662745 -0.050980 0.011765 from 0xD4, 0x79, 0x81 for unsigned byte values? Or is it signed bytes?

Once I know the expected values it should be possible to create code.

bytes/256.0, 3x (3 bytes, then skipping one byte) per line:

5e71: 0.828125 0.472656 0.503906  0.613281 0.105469 0.500000  0.335938 0.132813 0.738281  
5e85: 0.074219 0.019531 0.507813  0.984375 0.121094 0.113281  0.023438 0.523438 0.894531  

treating bytes as signed:

5e71: -0.328125 0.472656 -0.003906  -0.113281 0.105469 -0.000000  0.335938 0.132813 -0.238281  
5e85:  0.074219 0.019531 -0.007813  -0.484375 0.121094 0.113281  0.023438 -0.023438 -0.394531  

5E71: -0.167969 0.472656 -0.492188
5E85:  0.074219 0.019531 -0.488281

Edited February 11Feb 11 by shak-otay

It doesn't give me options for "signed" or "unsigned" in AXE from what I saw - just "Bytes" - but given some of the results it reads are positive values and some are negative values, I would hazard a guess at "signed". In fact, my sister and I tested the normals values for a pair of matching verts on this model, #0 and #308. The possible normals data (one blend byte and 3 XYZ bytes) were 0A D4 79 81 for #0 and 0A D4 7A 7F for #308. Looking at those verts on the model, it seems reasonable that no matter which type of TBN vector this actually was (in case the order of the MDLB strings lied to us) they should differ most on the X axis, with the X value being positive/negative versions of the same number. For the two normals listed in this pair, that tracks: 81 is -127 while 7F is 127 as signed Ints. 79 vs 7A is much less significant - 121 vs 122, both positive, possibly just the result of some accidental asymmetry.  As for how those bytes for #0 become those floats - I do not know. That's all stuff AXE handles.

1 hour ago, shak-otay said:

(Without 'inverse storage'.) How do you get 0.662745 -0.050980 0.011765 from 0xD4, 0x79, 0x81 for unsigned byte values? Or is it signed bytes?

Once I know the expected values it should be possible to create code.

bytes/256.0, 3x (3 bytes, then skipping one byte) per line:

5e71: 0.828125 0.472656 0.503906  0.613281 0.105469 0.500000  0.335938 0.132813 0.738281  
5e85: 0.074219 0.019531 0.507813  0.984375 0.121094 0.113281  0.023438 0.523438 0.894531  

treating bytes as signed:

5e71: -0.328125 0.472656 -0.003906  -0.113281 0.105469 -0.000000  0.335938 0.132813 -0.238281  
5e85:  0.074219 0.019531 -0.007813  -0.484375 0.121094 0.113281  0.023438 -0.023438 -0.394531  

I see you've now specified without inverse storage, but the answer is still the same: I give it the same starting offset, tell it data type of "Byte", and it generates those values as the XYZ. The only thing I can think of that might be throwing it off is if the bounding box is affecting things.

Sorry, I meant, "how do you get the values calculating them manually"?

It's  funny that my calculated values do not come close to the AXE values, even taking into account that the divisor (256.0) might be different.

The first three bytes should match, at least.

edit: seems I made a fault with "2 complement".

Edited February 11Feb 11 by shak-otay

For the mentions of -127 and 127 as values, I highlighted each byte in my hex editor of choice, HexFiend, and tell it to tell me what each one is as a Signed Int. If you're talking about floats, I have no idea how to calculate them manually. I might see if I can do some math with the values to pin down what it's doing in the near future but for now I'm clueless on what the float values should be.

well, got it (the AXE values for char):

-0.335938 0.945313 -0.984375
 0.148438 0.039063 -0.976563

(Corrected 2's complement and factor 2, divisor 128 instead of 256)

(Tomorrow I'll continue... with a tool)

btw, 127 is better - inverse storage values:

-0.992126 0.952756 -0.338583  -1.000000 0.212598 -0.771654  -0.519685 0.267717 0.677165  
-0.984252 0.039370 0.149606  0.228346 0.244094 -0.023622  -0.204724 -0.952756 0.047244  
0.905512 0.291339 0.795276  -0.480315 -0.818898 0.354331  -0.732283 -0.511811 -0.960630  
-0.700787 0.228346 -0.220472  0.826772 0.401575 0.850394  -0.094488 -0.889764 0.047244  
-0.779528 0.795276 0.149606  0.685039 -0.692913 -0.149606  -0.141732 -0.795276 -0.070866  
0.511811 0.228346 0.425197  -0.070866 0.811024 0.614173  0.897638 -0.070866 -0.464567 

0.433071 0.448819 0.803150
-0.803150 -0.803150 -0.110236
0.000000 -0.055118 0.811024
0.000000 -0.055118 0.181102
-0.566929 0.881890 -0.314961
-0.661417 -0.968504 -0.188976
-0.811024 0.409449 0.307087
0.000000 -0.874016 -0.062992
-0.346457 0.220472 0.559055
-0.385827 0.472441 0.307087
0.362205 0.858268 -0.574803
0.314961 -0.992126 0.929134
-0.590551 0.944882 -0.314961
0.000000 -0.055118 -0.440945
0.385827 0.669291 0.677165
0.976378 0.385827 -0.834646
-0.330709 -0.960630 -0.984252
-0.992126 0.291339 -0.204724
-0.244094 0.228346 -0.606299
-0.606299 0.795276 -0.362205
-0.228346 -0.803150 -0.259843
-0.535433 0.842520 0.055118
0.000000 -0.244094 0.921260
0.000000 -0.716535 0.433071
-0.173228 -0.606299 -0.551181
-0.724409 0.708661 0.834646
-0.173228 -0.606299 -0.551181
0.708661 0.905512 -0.173228
0.708661 0.905512 -0.173228
0.606299 -0.181102 -0.559055
-0.433071 0.023622 -0.299213
0.606299 -0.181102 -0.559055
-0.433071 0.023622 -0.299213
0.755906 -0.692913 0.952756
0.755906 -0.692913 0.952756
0.165354 -0.204724 -0.228346
-0.692913 -0.527559 0.929134
0.598425 0.133858 0.149606
-0.173228 -0.779528 -0.574803

Edited February 11Feb 11 by shak-otay

Thank you for this! I'm not usually one to mess with these programs, but I might just take a glance at the source code when I too am better rested to see if something is just set up weird. I'm glad to hear I'm not just imagining it with something being glitchy. The left hand column from those first sets you showed there match what I was getting with Char, other than the glitch replacing every X with -0.346457 on mine. AXE's source code is available to download so I might just open it up in XCode or BBEdit and see what I can do.

17 hours ago, gryphongirl said:

AXE's source code is available to download

Where? I checked Bigchillghost's Github and there's the zipped exe only.

Anyways, here's my TBN tool (feel free to request a different output format, see Makeobj_log.obj) :

TBN.zip

edit: downside: uses a fixed count of 616 and stride of 20 (so 616x20 bytes are covered from start address).

Edited February 12Feb 12 by shak-otay

You're right! I saw a zip titled "Source code" under version 1.3.0 but that was just the UI picture and readme from the main Github page.

Thank you for the TBN tool! Do you mind if I share this with the rest of the team of people working on RE'ing this game? I'm a Mac user so I'll have to figure out a good way to run the exe (don't spend your time trying to solve that for me, I have plenty of options like Crossover and a VM). Obj will work for now on armor/environment pieces, though there's also character models whose skeletons I'm still figuring out which would need something like FBX that supports bones. That's what AXE outputs, although it claims to put out binary FBX files that Blender likes when it actually puts out ASCII FBX which requires me to use another conversion tool. Although, if I can use AXE on an output obj file where the normals are set up better, that might be a good step of the process...

I also want to double check ahead of time since I can't test it yet (work has begun, and today promises to be pretty busy): What parameters can I give the exe, such as stride, offset, and count for different things? To borrow terminology from your program Hex2Obj, although armor, weapon, and environment files in this game are laid out in VB style bringing together position and UV (and later TBN and colorset) like two sides of a zipper, character models are Seq style with everything in its own block, and there's many other files that have a flag suggesting they're some other format I'll have to study. I also have to control how many/which submeshes are used or in some files like this helm I'll end up with multiple LOD models layered over each other in one file, while other models like characters need every single submesh. 

While I may not have the source code, I do have a sister who is crazy enough (I say with affection/awe) to enjoy low level code like Assembly, and even reverse engineered the code of the Wii version of this game enough with the help of a symbol table to find functions looking for data about "strips" in the MDG files, proving it was using tristrips instead of the x360 version's triangles. Depending on how large the EXE for AXE is, I almost wonder if she would be able to sort out the bug. Model Researcher Pro was almost in a state where she could find the bug with the data-highlighting system that discouraged me from using that, though not quite close enough. So the question is mostly whether it's looked down on to patch someone else's extraction program you don't have the source code to. If MRPro can do everything except the skeleton, I almost wonder if I could just have AXE do multi-source with MRPro's output for the main mesh and the MDLB file for the skeleton.

Thank you again for all your help! You've made it so I can confirm without a doubt AXE was messing up the normals when things were inverted. 

1 hour ago, gryphongirl said:

Thank you for the TBN tool! Do you mind if I share this with the rest of the team of people working on RE'ing this game?

It was intended for that. (It uses a fixed count of 616, btw. Because there's only one param input box. I could add a 2nd one for the count, if required.)

Quote

when it actually puts out ASCII FBX which requires me to use another conversion tool.

Noesis should be a good candidate.

Quote

Depending on how large the EXE for AXE is, I almost wonder if she would be able to sort out the bug.

Maybe it's easier to contact Bigchillghost? (He doesn't seem to be active here any more but I think he could be interested to fix the bug himself.)

Edited February 12Feb 12 by shak-otay

Okay note to self - if I think I lost a draft of a reply, I just gotta try tapping the box to type and might find out it's still waiting here.

I opened an Issue on the AXE Github with most of the more benign (no hint of what game) screenshots/data presented here and with thanks for making the program and BigChillGhost has released what appears to be a fix for this and some other issues, including something that reads like it might let me export Binary FBX instead of ASCII, so fingers crossed! Will test it during work downtime today and either edit this reply or post a new one with the results!

Though small warning to anyone reading who hasn't read the patch notes yet - WinXP support has been dropped. In my case I should be fine, just gotta figure out how to replace the EXE Crossover on my Mac is looking at.

I actually fixed that issue in the middle of 2024, yet didn't bother to publish it. You can download the patched version from the release page now. (Well, your reply was quicker than mine😃)

Just for some clarifications:

1. Char stands for signed char while Byte for unsigned char, that's a common sense for anyone with C language or win32 background thus why the app chose the term;

2. For unit vector encoded as Byte, the formula for decoding is:

(unsignedByteValue / 255.0) * 2 - 1.0

And please do send my regard to your sis. That's a real precious spark in her!

19 minutes ago, Bigchillghost said:

2. For unit vector encoded as Byte, the formula for decoding is:

(unsignedByteValue / 255.0) * 2 - 1.0

The -1.0 will lead to signed bytes, doesn't it? (The '-' for some values for type Byte was the thing confusing me.)

Quote

while Byte for unsigned char

 

Yes, the core idea is to scale an unsigned char into range of 0 to 2, then minus 1 to shift the entire interval into [-1, +1].

Finally have the symmetry I was looking for! Definitely going to keep trying the normals data of more meshes and see if it's just these models at the scale of an owl or smaller being so tiny that makes their normals not always stick out and their binormals/tangents not always lay flat - though maybe shiny metal armor like this (which seems meant to have a copper metal texture or similar) is just given much more dramatic lighting.