Author Topic: Pigg files / geo files / bin files?  (Read 10654 times)

bearpounder

  • Minion
  • **
  • Posts: 22
Pigg files / geo files / bin files?
« on: January 17, 2017, 12:27:42 AM »
Hello, I am looking for some assistance with this. I have been working on modifying a program that can read the geo files packed in the piggs. I have successfully been able to add functionality to dump the geometry to the OBJ file format. The issue I am having now has to do with the way the normals and UV's are stored / processes. As you can see in this image, the program is correctly reading and applying the textures (sort of) in an open GL environment. However to get this to work I had to hunt down what seemed like the most likely texture and change the name (for example I cannot find "X_AP_CityHall_Concrete_01". I had to add the preceding "X_".

So In regards to the GEO files does anyone know how it is supposed to find the correct texture files within the Pigg directory structure?

I am also curious about the Bin files. I know someone has written a python script to extract them. My theory is that they contain data on how to build the actual maps with the geofiles (location, rotation scale etc). Has anyone discovered any further info as to what all that information means?




Floride

  • Elite Boss
  • *****
  • Posts: 863
  • Badgehunter Extraordinaire
Re: Pigg files / geo files / bin files?
« Reply #1 on: January 17, 2017, 09:30:28 PM »
https://www.cohtitan.com/forum/index.php?topic=12222.0

This thread has the links for the Piggviewer which you can use to extract .geo and .bin files, and links to CW's GeoDraw.exe and Bindump.exe. Although I just open the extracted .bins in notepad.
History shows again and again
How nature points out the folly of men

bearpounder

  • Minion
  • **
  • Posts: 22
Re: Pigg files / geo files / bin files?
« Reply #2 on: March 13, 2017, 03:06:44 PM »
What is this Bindump.exe you speak of? I couldn't find a link on that thread.

I already have geodraw... in fact I just created my own geo reader / exporter.

If bindump extracts the schema's of the bin files I am looking for it.

CW sent me the schema for tricks.bin and I am still waiting patiently for a response
as to how he got that info. It would be very useful for several other BIN files if the
schema's are available for them.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Pigg files / geo files / bin files?
« Reply #3 on: March 17, 2017, 07:24:34 PM »
Bindump is a program I wrote back in early 2011 as a universal command-line tool to dump bin files into a text format. Back when the game was live, its primary purpose was to allow for data comparisons across different game versions so that I could create "unofficial patch notes" on our internal forums to show what really changed with each patch, whether the devs documented it or not. Not exactly an easy feat, since the fields inside bin files - and thus the schema needed to read them - often changed from patch to patch, and things like enum values did not always remain constant.

After I joined up with the Titan crew, other variants of the same code base generated the power data exports to load into City of Data, and later Mids, which I took over after TopDoc who had been doing the powers.bin exports previously with his own csv-based tool was MIA for a while.

It wasn't publicly available until around 2014. The link to it is kind of buried, it's in the middle of a discussion about minimaps:

https://www.cohtitan.com/forum/index.php?topic=9546.msg137651#msg137651

The dropbox link appears to still work.

https://www.dropbox.com/s/zn9x1k3181ox58r/bindump.zip?dl=1

Here's some very basic documentation I wrote about it:


-- snip conversation about C vs. C++ for brevity --

So anyway... Here's a toy to play with until I can figure out the best way to present all these file formats. I finally managed to get a windows cross-compile working well enough. The core code is cross-platform, just some of the boost library dependencies (that I'll be avoiding in future rewrites) are a pain to cross compile.

[old google docs link removed, used the dropbox one above]

Just point it at a bin file and it'll dump out a textual representation. It's not quite as slick on Windows when you don't have tools like grep and less to pipe it into (or maybe you do 8)), but it's functional if you just want to get an idea of field names.

You can either point it at a bin file directly, in which case it'll look for bin/clientmessages-en.bin or clientmessages-en.bin in the current directory, or you can point it at a bin file inside of a pigg by doing something like

bindump \path\to\bin.pigg:bin/powers.bin

and it'll use the clientmesssages inside that pigg. You can also specify a path to an alternate clientmessages file for other languages, which can also be either standalone, or inside a pigg (not necessarily the same one) using that same syntax.

There's some options, use -h to get the full list. Ones I use a lot are -v to show normally hidden fields like the attribute tables in classes.bin or the palettes in costumes.bin. -x does a full hex dump and -d does a partial hex dump of numeric fields, useful if you suspect alignment problems are causing a string to end up inside of one.
« Last Edit: March 17, 2017, 08:05:23 PM by Codewalker »

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Pigg files / geo files / bin files?
« Reply #4 on: March 17, 2017, 08:14:36 PM »
I would post the def files from bindump that document the various bin files, however they're a bit byzantine and difficult to decipher. Not only are they in a special format that a Python script has to convert to C++ classes and a bunch of generated code during the build process, but they cover many different variations of the bins -- every iteration from Issue 18 through Issue 24. That's a product of bindump's design goal to support every version of every file at once and do cross-comparisons even when the format changed.

So I'll do one better. The real breakthrough in bin file parsing came sometime in early-to-mid 2012. Before that it was mostly a manual process, helped out by more-or-less reliable field names extracted from the executable's string table. I spent a lot of quality time with the client in OllyDbg reverse engineering exactly how the client loads bin files and identified the relevant functions within textparser -- as well as the binary tables it uses to know how to load each file. The same table is used both for parsing from text format and for persistance in bin format. That gave us not only a 100% accurate and authoritative reference for field names, but also in many cases symbolic names for many integer enums and binary flags. I called it the "Rosetta Stone" of COH data files because it blew the doors wide open. Of course we didn't get to use it for very long before the shutdown announcement came...

Shortly after that breakthrough I wrote a small program to extract a parse table from the executable and render it in an XML document. That's what I sent you for tricks.bin earlier. The program itself takes two parameters -- the path to the cityofheroes.exe file, and the memory address of the table to decode. I did some digging after our last PM exchange and managed to find that program:

schema.exe and C source:
https://www.dropbox.com/s/2s6a50hvloal6dg/schema.zip?dl=1

Since you probably don't want to go through with a debugger and a fine-toothed comb to find all of the parse tables, the source above is mostly for reference purposes. I also found an archive of XML schemas for the parse tables of every Issue 24 bin file format. IIRC, this was obtained by doing a global search in the debugger for function calls to the main textparser entry points, seeing what parse table addresses they were called with, then using the schema extractor tool on each one. Here it is:

https://www.dropbox.com/s/g87dthcx6o2ppo1/i24xml.zip?dl=1

That should definitely help with bin file viewing. :)

bearpounder

  • Minion
  • **
  • Posts: 22
Re: Pigg files / geo files / bin files?
« Reply #5 on: March 19, 2017, 11:03:57 PM »
man that is awesome Info and will help tremendously!

bearpounder

  • Minion
  • **
  • Posts: 22
Re: Pigg files / geo files / bin files?
« Reply #6 on: March 26, 2017, 05:13:38 PM »
Ran into a snag. Seems like most but not all the info is in tricks.bin.

For example: Ouroboros_structure.geo has a reference to several "tga" files (stone_pool.tga for example) but I cannot seem to find these materials or tga files anywhere.

I am continuing to look around but if anyone has any clues on this would be much appreciated.

bearpounder

  • Minion
  • **
  • Posts: 22
Re: Pigg files / geo files / bin files?
« Reply #7 on: March 26, 2017, 05:28:32 PM »
Never mind.... *sigh*

bearpounder

  • Minion
  • **
  • Posts: 22
Re: Pigg files / geo files / bin files?
« Reply #8 on: March 27, 2017, 03:27:43 PM »
so I am noticing a few issues when it comes to Geo files. (this is using the un-modified original geo viewer btw)

As you can see in the image below, the normals for a few vertices seem to be messed up. Also when exporting the OBJ
there are literally thousands of duplicate vertices.  All the OBJ exporter does is go through each face of the model in question
and write out the vertex, vertex normals, and uv cords based on what is read into the GEO.MODEL structures.


Is it possible we are doing something wrong?




MajorDisaster

  • Underling
  • *
  • Posts: 3
Re: Pigg files / geo files / bin files?
« Reply #9 on: June 12, 2017, 05:12:19 AM »
This looks amazing! Any chance I can get a copy?

With the poly's with messed up normal, is it possible you're always reading the files in a particular order, when maybe reading by the vertex listing order is different?

How are you outputting to the .obj file? Most of them that I've seen list the vertices, then list the faces. I haven't dove into the format hard lately, although I did a few years ago. Do you need some help? I'd be willing to pitch in.

Regarding the normals, when you said messed up I thought you meant inversed - that's something a modeler could fix really easily of course, but I think you meant off.  I *think* that's in an area under a building?

== John ==

P.S. I messed up my forum name - fairly new on here, so I just created a 2nd account, which I may need to do some Machinima with anyway. I'd signed up as jgwinner, then changed it to my normal online handle of "CodeMaster" (like Webmaster, only for code), then dove into the forum and I've seen what an unbelievable contribution CodeWalker has done ... and I don't want any confusion! So I joined as "MajorDisaster" my primary CoH Hero.

Fighting Blaze

  • Prom King 2017
  • Underling
  • *
  • Posts: 13
Re: Pigg files / geo files / bin files?
« Reply #10 on: June 12, 2017, 10:52:07 PM »
Any chance one of you could walk me through how to turn my character's .pigg files into obj or something usable with blender?

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Pigg files / geo files / bin files?
« Reply #11 on: September 03, 2018, 01:09:47 PM »
Sorry to necropost, but how did you figure out the .geo file format? I can't find a schema for it anywhere

PhiloticKnight

  • Minion
  • **
  • Posts: 40
Re: Pigg files / geo files / bin files?
« Reply #12 on: February 03, 2020, 08:40:33 PM »
I would post the def files from bindump that document the various bin files, however they're a bit byzantine and difficult to decipher. Not only are they in a special format that a Python script has to convert to C++ classes and a bunch of generated code during the build process, but they cover many different variations of the bins -- every iteration from Issue 18 through Issue 24. That's a product of bindump's design goal to support every version of every file at once and do cross-comparisons even when the format changed.

So I'll do one better. The real breakthrough in bin file parsing came sometime in early-to-mid 2012. Before that it was mostly a manual process, helped out by more-or-less reliable field names extracted from the executable's string table. I spent a lot of quality time with the client in OllyDbg reverse engineering exactly how the client loads bin files and identified the relevant functions within textparser -- as well as the binary tables it uses to know how to load each file. The same table is used both for parsing from text format and for persistance in bin format. That gave us not only a 100% accurate and authoritative reference for field names, but also in many cases symbolic names for many integer enums and binary flags. I called it the "Rosetta Stone" of COH data files because it blew the doors wide open. Of course we didn't get to use it for very long before the shutdown announcement came...

Shortly after that breakthrough I wrote a small program to extract a parse table from the executable and render it in an XML document. That's what I sent you for tricks.bin earlier. The program itself takes two parameters -- the path to the cityofheroes.exe file, and the memory address of the table to decode. I did some digging after our last PM exchange and managed to find that program:

schema.exe and C source:
https://www.dropbox.com/s/2s6a50hvloal6dg/schema.zip?dl=1

Since you probably don't want to go through with a debugger and a fine-toothed comb to find all of the parse tables, the source above is mostly for reference purposes. I also found an archive of XML schemas for the parse tables of every Issue 24 bin file format. IIRC, this was obtained by doing a global search in the debugger for function calls to the main textparser entry points, seeing what parse table addresses they were called with, then using the schema extractor tool on each one. Here it is:

https://www.dropbox.com/s/g87dthcx6o2ppo1/i24xml.zip?dl=1

That should definitely help with bin file viewing. :)

I just want to say, this single post is an amazing example of your skill levels codewalker. I bow down before you as not being worthy. Did you or anyone else that you're aware of ever think about making tools (either command-line or perhaps a dll) that could be used in an automated way to debin/bin and de-pigg/pigg files? Kind of like how WinRar and 7Zip's command line process works? I started down this path, but very quickly got my eyes crossed with the complexity that seemed far above my ability.