Leisure Suit Larry: Magna Cum Laude .WGG Files

[Anti6]

Hi, so I made a script to unpack and repack the games JAM files, inside there are a bunch of assets, one of which is the WGG filetype, it contains the information that the game uses to create a model, I wrote the attached script in order to convert it into a .OBJ, so that it can be viewed in blender. I might update it further if I run into problems with the rest of the files, especially the level files seem like they can be painful to convert properly. I also have not yet identified all of the fields, so at this stage changing a model and then repacking them into this might not be feasable. 

Enjoy

For the Jam unpacker and repacker please see https://reshax.com/topic/1034-leisure-suit-larry-magna-cum-laude-jam-files/?do=findComment&comment=5446

I posted this in "Game Archive" as well, and don't know how to remove it or move it to the correct place.

 

WGG_convert.py

[Anti6]

Update, will now unpack each object into a single file and won't create an extra random empty .obj file.

Also changed how the script selects which file to unpack. You can just drop the .wgg file onto the script now instead of editing the 'input_file'.

To texture the file, you can search for the dds or bmp file that is associated with it. Not sure how I would code it in yet.

Also as I didn't mention it previously. The script might look awful to some, I am very new to python and I am using these types of little projects to teach myself.

Before this I did not know what a obj file was, how it works, or how 3d models work.

 

WGG_convert.py

Edited September 24 by Anti6
attached wrong file.

[Anti6]

Update, will now unpack most files. (Haven't tested all 379 files. Around 40 with different sizes have been tested)

Commented in the file so you can see what is going on and why.

Some issues remain.

There are lighting(I am assuming due to ignorance) "objects" in some files that have no vertex chunk number - nor co-ordinate, same for face, these usually just contain 1 vertex x, y, z co-ordinate with v1,v2,v3 face data.

I have no idea how to fetch this data.

I am guessing through a lot of this but it seems to work fine from my tests.

I changed the y and z co-ordinates from the previous files. To the correct way, ie x,y,z not x,z,y. I did this previously because of blender.( I am new to blender)

There are also some objects that spawn at or around 0,0,0. These seem to be controlled by WKA files, which should be animation files of some kind.

I would appreciate any insight anyone has where I might have missed something or where they can fill in the unknowns.

Textures seem to apply fine throughout but I have no way currently of automating this, any help on this would also be appreciated since the texture data is mostly in other files.

 

 

WGG_convert.py

Edited September 30 by Anti6
Attached correct file. Need to flip uvy for textures

[Anti6]

So I am running into problem with the last part of this converter.

Some of the files have a different format for objects, and it just does not make any sense.

This explanation is specifically to the files I attached, but can be adapted to others.

---

{Position 1} + {pointer to last chunk} is where this portion of data starts. (192)

The entries seem to be 16 bytes long, and there seem to be 117{unknown 11} of them. (117*16=1872)

Directly after this, there are what seem to be vertex values that are 32 bytes long, an there is 154{unknown 12} of them. (154*32=4928))

{Unknown 7} is the size of the first portion(what seem to be faces) of the unknown section. (1872)

{Unknown 8} is the size of the faces and vertex together.(1872+4928=6800)

{Unknown 11} is the amount of faces for the object if {number of table entries} = 0.

{Unknown 12} is the amount of faces for the object if {number of table entries} = 0.

---

Face data looks like this:  18 20 19, 1, 0, 0

4 Bytes - face 1?,4 Bytes - face 2?,4 Bytes - face 3?,2 Bytes - unknown,1 Bytes - unknown,1 Bytes - unknown

Vertex data look like this: 0.050289 0.998726 0.004093, -12.794792, 0, 0, 68

4 Bytes - x float, 4 Bytes -y float, 4 Bytes -z, 4 Bytes - Unknown float, 4 Bytes - Face number,4 Bytes - Unknown, might be object ID,4 Bytes - other vertex number or ffffffff,4 Bytes - next vertex number or ffffffff

---

I have no idea how I would go about making this usable? Can anyone help?

WGG_convert.py WGG.7z

[shak-otay]

Fst: without a model sample noone can follow your thoughts

2nd: I'd suggest to focus on one problem after the other.

The "0,0,0" objects require a position applied. Maybe found in transform matrices, 4x4 for example, where the first 3 rows usually give the rotation (3 euler angles for example).

4rd row should contain x y z 1 (or x y z w). (Sometimes there's 4x3 matrices only.)

Principially it's a good idea trying to get a complete format description, like you seem to do it.

But there's some people like me who take the short path first: extracting one sub mesh of a 3D model to have a base for understanding the format

Edited October 2 by shak-otay

[Anti6]

Hi shak-otay

Thanks for the reply, I did upload the file in question as well as the converter that would allow you to follow my thoughts in the previous post.

I attached a different file to this one, this is a hybrid, with files I have already figured out how to extract and then one that I can't, which is the bulk of the rest of the file.

This should be the last problem to understand the file structure, all indications are that the 2 sections are related, I just don't understand how to structure the data in a way that makes a usable model.

All my understanding of 3d models are literally from reversing this filetype. Unfortunately I dont understand what you mean with transform matrices, rotation, or euler.

Any help would be appreciated, and I'll make sure to read up on the terms you mentioned.

I attached a different one where all items are extarctable for whoever doesn't have access to the games .jam files.

LLIBMAIN.7z WGG_convert.py GENLESBA.7z

Edited October 2 by Anti6

[shak-otay]

Hi Anti6 and thanks! This is what I found on a quick hack. Looks like a sensible part of mesh (though I thought the FVFsize to be 24 instead of 12):

LLIBMAIN WGG, one sub mesh maybe (edit: FIcnt is 8135)

(Maybe you have the complete mesh already, as one could guess from the picture in your opening post?)

Used your script in blender. Had to force the output txt file being sent to known path by adding a drive letter.

But the .obj is sent by blender somewhere. Need to search for it. Or add the drive letter, too.

edit: ok, found it. ILLaFone.obj. (Did you notice the vertices are "nan", not a number? Since the Fone is loaded you can ignore it probably but it should be fixed somehow. Maybe use less digits?)

(I'll check the problematic WGG now, but I fear I won't dig as deep as you did...)

Well, the 34 vertices at 0x1b50 should form a bottle. Strange, that they don't.

For LLIBMAIN there's 2 possible vertex start addresses, 0x25C and 0x7B390, vCnt= 905 949. Both form the same mesh part.

Edited October 3 by shak-otay

[Anti6]

Hi shak-otay, thanks for the reply, I didn't know you can use the script in blender to be honest🙈. I'll have to look into that, I just dropped the file onto it, or if the filename is hardcoded then have it in the same folder. In the LLIBMAIN there are 4 extractable .obj, namely ILLaFONE.obj, LBKyPdLn.obj, LBKyPdSn.obj, and SCRNSVR.obj. I decided to use the "mesh name" in the file to name them, though the parts often have different name, in this case ILLaFONE.obj = LBCarp, LBKyPdLn.obj = LBAdd2, LBKyPdSn.obj = LBAdd3, and SCRNSVR.obj = LBScrnSV. Looking at output_mesh_dump.txt and the script should explain how I got those, when the file is completely understood I'll explain the file like I did with the JAM unpacker.

I did note that the fourth float is nan, so if you look at the script it should explain somewhat, but sometimes there are extra vertices depending on the vertex size. Since .obj doesn't use any extra vertices except x,y,z , I didn't think it a issue to export them, as they will just be disregarded. Honestly, I don't know if the extra vertices are supposed to be RGBA or something other than weight. For instance some of the character models have 5 floats, x,y,z and then 2 more which might be weight and rgba or something to do with the bones. I exported them for incase someone in the future figures out how to use them, but it is somewhat explained in the script. exporting them does not break the .obj, so I left it in for sake of completion.

Edit: When I say 5 floats, I mean, x,y,z,float1,float2 the normals x,y,z and u,v is there as well and follows after. Also not all models have the fourth float as nan.

The issue with this file is that there are supposed to be 11 objects, 7 of which are in table 0, and should be contained in the extra data I have yet to figure out.

If you check the output_mesh_dump.txt that the script creates, table 0 contains the mesh that I am struggling with table 1 to table 4 does extract properly. The offset for the data in question is 18752 (0x4940) - this is what I assume to be face data. It contains 8135 16-byte lines, which end at 148912 (0x245b0) which is also where what I assume to be the vertex data starts. The vertex data has 11119 32-byte lines, which end at 504720 (0x7B390). It seems that they are somehow linked, as the vertex data contain number that match of the face data i.e. 0-8134, and two field that match other vertex data i.e. 0-11118, but sometimes the 2 fields are ffffffff, so I am unsure, how these vertex co-ordinates would link to eachother and how the face data should be changed to match. There are alot of duplicates scattered throughout. The attached WGG_converter.py dumps the content of these parts to the end of output_mesh_dump.txt, for ease, I have attached it hereto. The last 8 lines of the script are the extraction of these line if you want to know the formats, and each line has the line number infront, this isn't in the hex data, but I try to use it to match these together.

Hope this helps clarify some things.

WGG_convert.py output_mesh_dump.7z

Edited October 3 by Anti6
Clarified 5 floats

[Anti6]

Looking at the last part of your response with regards to cola.wgg. I am wondering if the unknown face data and unknown vertex data aren't somehow to do with transform matrices or something, but how would you apply it to the vertex data that you mentioned starts at 0x1b50(I somehow completely missed that as it isn't linked like it would be normally, but it would explain quite a bit). As 154 is divisable by 34, I wonder if this file doesn't contain 4 different transformations for the object in question. Might be worth looking into, it's a better explenation than I have had untill now.

Edited October 3 by Anti6
incorrect cal

[shak-otay]

1 hour ago, Anti6 said:

Hi shak-otay, thanks for the reply, ...

It contains 8135 16-byte lines, which end at 148912 (0x245b0) which is also where what I assume to be the vertex data starts. The vertex data has 11119 32-byte lines, which end at 504720 (0x7B390).

Hi Anti6, at 0x245B0 I see normals only. All important vertices seem to reside before 0x4560, with different FVFsizes. (Dunno what the copy of 0x25C at 0x7B390 is good for.)

Missed some faces, btw.

(Bigchillghost could tell us the meaning of every byte, I guess. But his appearances are getting rarer...)

Durik might have a better understanding than me, also.

Edited October 3 by shak-otay

[Anti6]

Well, i'll be damned, the face data matches the vertex data fine, when I tested in my idiocy I forgot to add 1 to each face. Still need to figure out what the padding means.

If I apply that to your program it looks fine, though I will still have to figure out how to divide this into 7 parts:

If you say that 0x245B0 contains normal data, then I'll take you at your word and work from there, though what would the 4th float be? The other 16 bytes still seem to be linking data.

I do still wonder why there are duplicate faces, and why the normals have references to the faces, maybe each vertex has more than one normal? Also odd that the UV information seems to be missing, but maybe I should throw out my assumptions and just look for it. I already failed to realize the vertex information is there, so who knows, maybe it's also hiding somewhere.

It also seems that not all files have the vertex information in the same format, LLIBMAIN has it where it is expected, at 0x25c, but colA has it at the end only where it should be at 0x188 normally.

Thank you, with this I should be able to finish building the converter(I hope). Feel stupid for not figuring it out myself. Occam's razor I suppose. I did learn quite a bit, which is what matters.

Quote

(Dunno what the copy of 0x25C at 0x7B390 is good for.)

I honestly don't know why either, almost every single one of these files duplicates the first chunk at the end of the file.

Still have a lot to learn before this little project is finished.

[shak-otay]

1 hour ago, Anti6 said:

 

If you say that 0x245B0 contains normal data, then I'll take you at your word

Well, "my words" are almost always wild guessing, so don't rely too much on them.

How did you get 0x2F44, btw? Other than the counts I couldn't seem to find it in your outout log.

stumbled over Watto tool, which can extract jam files, allegedly. This is a reduced version.

(Dunno if it's worth to mention because there's a working extractor in this forum already.)

Edited October 3 by shak-otay

[Anti6]

Quote

Well, "my words" are almost always wild guessing, so don't rely too much on them.

Well, I am assuming you have quite a bit more experience, so leaning on your guesses makes sense to me.

Quote

How did you get 0x2F44, btw? Other than the counts I couldn't seem to find it in your outout log.

So the log does indicate it, but it's somewhat split. In this case all vertices are collected into one place, sometimes they are split, but generally(except for the instance with the building faces that have duplicates you helped me with) they are all together. In the log, there is Chunk 0 to Chunk 4, vertices start at "Pointer to First Vertex Chunk: 456"(0x1c8)+88 bytes for the header so 0x220 and end at  "Pointer to faces: 17664"(0x4500) + 88 for the header so 0x4558, they are split into "Number of Vertex Chunks: 5", each 'chunk' starts with a 60 bytes header, Header bytes 4 - 8 indicate how many vertices in the chunk, and bytes 56-60 indicates how big the vertices are. Chunk 0 is 953 and 12, So it ends at 0x2F08, where the next one starts if there is more than 1. After the header is a collection of names, after that is a collection of tables, the table header tells you about how many pieces make up the mesh and the name of the mesh, the table data then follows the header(usually), the table data tells you which chunk to use for vertex data, which chunk to use for face data. ILLaFone table header starts at 0x120, table data starts at 0x138. The first 0x4 tells me to use vertex chunk 1, the second 4 bytes tells me to use face chunk 0, the next 4 bytes are for vertex chunks that have no faces(Guess for now), next 4 bytes tells me how many vertices to skip to find where the vertex data starts, and the next 4 tells me how many face to skip, next 2 bytes is unknown, the next 2 bytes tells me how many vertices to use, the next 2 is unknown, the next 2 tells me which object this table data belongs to, the next 2 tells me how many faces to use. So in the case of ILLaFone, vertex chunk = 1, vertex to skip = 0, vertex count = 196, chunk 1 starts at 0x2f08. It is mostly the same for faces with some differences in header. vertex header is 60 bytes, so you have to skip 60 bytes to get the vertex data which gives you 0x2f44.

I hope this makes sense, you should be able to follow through the log. The script does try to explain some most it, but I haven't created a doc for the layout yet, so the logic needs to be reversed if you want to figure out how it gets what.

Quote

stumbled over Watto tool, which can extract jam files, allegedly. This is a reduced version.

Yehp, but it can't repack. Atleast not without paying for the full version, not sure if the full version can, but I figured out how to, so I am sure that they figured it out as well.

Quote

(Dunno if it's worth to mention because there's a working extractor in this forum already.)

Which is why I wrote the one in this forum, I linked it in my first post. it's not just an extractor, you can unpack and modify then repack the files, it is to help with game modification.

Edited October 5 by Anti6
Clarified third quote reply, fixed dec - hex conversion errors

[Anti6]

Hi, here is the final version.

As far as I can see and tested it extracts everything correctly, some files have duplicate mesh data at 2 different places, in these cases, only one gets exported.

When I have some time I'll write documentation on the layout for future use, I still can't figure out what the extra section is, it seems like transformation data. But it isn't needed to extract the mesh, so I am conveniently ignoring it, though I am saving it as "normals" in the obj file that is generated🙈😁. These normals might not be of any use to anybody. The can be easily removed from the script or the .obj in question.

WGG_convert - Final.py

[Meth0d]

Faces are stored as tristrips ?

[Anti6]

Not sure where that could be true, except for 1 filetype for there is only vertex data, though usually it only contains 1 vertex.

Faces are usually stored separately in the file.

 

[Anti6]

Hi

I hope this helps, please use it in conjunction with the script if anything is unclear in either. I got lazy in some parts of the script.

Documentation for WGG files:

Header - 88 Bytes (All pointers and offsets always starts after the header)
8 Bytes - Magic(HVSIWGG)
4 Bytes - Version1, Always 2
4 Bytes - Version2, Always 13
4 Bytes - EndOffset, also Filesize
8 Bytes - Object Separator, used to separate objects in file
4 Bytes - Last Chunk Pointer
4 Bytes - Unknown, usually 0
4 Bytes - Amount of Tables
4 Bytes - Size of object list
4 Bytes - Number of Vertex Chunks
4 Bytes - Pointer to start of vertex information
4 Bytes - Number of Face Chunks
4 Bytes - Pointer to start of face information
4 Bytes - Number of vertex chunks with no face information, called "Positional Chunk" herein 
4 Bytes - Face EndOffset
4 Bytes - Size of Last chunk to EndOffset
4 Bytes - Unknown, might be pointer to start of object list, always 0
4 Bytes - Size of "Unique Face data"
4 Bytes - Size of "Unique Face data" and "Unique Transformation Data"
4 Bytes - Size of unlisted chunk, mostly unused
4 Bytes - Size of Last chunk(Duplicate)

Object List:(For each 16 bytes in "Size of object list")

8 Bytes - Object Name
8 Bytes - Object Separator

Tables:(For each "Amount of Tables")

Table Header: (24 Bytes)
4 Bytes - Number of entries in table
4 Bytes - Size of "Unique Face data" in objects
4 Bytes - Number of objects
8 Bytes - Name of table
4 Bytes - Size of "Unique Transformation Data" in objects

Table Data:(40 Bytes - For each "Number of entries in table")
4 Bytes - Number of the Vertex chunk containing data 
4 Bytes - Number of the Face chunk containing data
4 Bytes - Number of the Positional chunk containing data
4 Bytes - Number of vertices to skip in vertex chunk
4 Bytes - Number of faces to skip in face chunk
4 Bytes - Size of faces in object
2 Bytes - Unknown
2 Bytes - Number of vertices in object
2 Bytes - Unknown
2 Bytes - Which object from object list does this table belong to
2 Bytes - Number of faces in object
2 Bytes - Unknown
2 Bytes - Unknown
2 Bytes - Unknown

Vertex Chunk:(For each "Number of Vertex Chunks")
Header:(60 Bytes)
4 Bytes - Type of Vertex Chunk, 0 Normal, 4 Split, or 8 Transformational
4 Bytes - Count of vertex entries
4 Bytes - Flags, 80 = only vertices, A0 = Vertices and UV's, A8 = Vertices, normals and UV's
4 Bytes - Unknown
2 Bytes - Unknown
2 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
2 Bytes - Unknown
2 Bytes - Unknown
4 Bytes - Unknown Float
4 Bytes - Unknown Float
4 Bytes - Unknown Float
2 Bytes - Unknown
2 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Size of vertex entries
Data:(For length of "Size of vertex entries*Count of vertex entries" either 12,16,20,24,32,36, or 40)
12 Bytes (3 Floats)= Contains only vertices  * Count of vertex entries
16 Bytes (4 Floats)= Contains vertices and extra unknown float  * Count of vertex entries
20 Bytes (5 Floats)= Contains vertices and UVs  * Count of vertex entries
24 Bytes (6 Floats)= Contains vertices and extra unknown float and UVs  * Count of vertex entries
32 Bytes (8 Floats)= Contains vertices and normals and UVs  * Count of vertex entries
36 Bytes (9 Floats)= Contains vertices and extra unknown float and normals and UVs  * Count of vertex entries
40 Bytes (10 Floats)= Contains vertices and two extra unknown floats and normals and UVs  * Count of vertex entries

Face Chunk:(For each "Number of Face Chunks")
Header:(8 Bytes)
4 Bytes - unknown
4 Bytes - Size of face information
Data:(For length of "Size of face information")
6 Bytes - face 1, 2 and 3 each 2 Bytes * ("Size of face information"/3)

Positional Chunk:(For each "Number of Positional Chunks") - Unlike other chunks, all headers are together, no data is between them.
Header:(72 Bytes)
4 Bytes - Count of entries
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Size of entries
4 Bytes - Unknown
Data:(Size =  "Size of entries*Count of entries" Always starts at "Last Chunk Pointer")
12 Bytes (3 Floats)= Contains only vertices * Count of entries

If the vertex chunk has a 4 or 8 in first 4 Bytes then a large portion of the file is dedicated to "strange" face and transformational data, called "Unique" herein, there is never more than 1.
Sometimes with 4 you can extract the data from the usuall route through the tables, but 8 never has table data accompanying the header.

If the "Unique Face data" or "Unique Transformation Data" has a value other than 0 then it is referring to this chunks data.
The data starts at "Last Chunk Pointer", for the length of "Size of "Unique Face data"", what is odd is that it contains duplicates.
Face Data(Count = "Size of "Unique Face data"/16):
4 Bytes - Face 1
4 Bytes - Face 2
4 Bytes - Face 3
2 Bytes - Unknown(Seemed like a index number from object list, but that is untrue for some/most files, unless some weird logix applys)
1 Bytes - Unknown
1 Bytes - Unknown

After the face data is "Unique Transformation Data", it is used for normal data in the script, but this is wrong as some of the chunks that start with 4 have normal data in the vertex chunk.
Transformation Data:
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Unknown float
4 Bytes - Index number for face data
4 Bytes - Unknown 
4 Bytes - Index number for other transformation data or 0xFFFFFFFF
4 Bytes - Index number for next transformation data or 0xFFFFFFFF

 

Filling in any unknown data would be awesome, and very much appreciated, though I am pretty much finished with this project. I am attaching all the files I have hereto.

WGG Sample.zip

WGG_convert - Final.py

Edited October 21 by Anti6
Attached script and fixed spelling mistake.