Author Topic: Technical side discussion  (Read 54134 times)

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #480 on: October 14, 2018, 04:05:27 PM »
That is some damned cool beans, Codewalker. Thanks for sharing!

Nice to see that some of my guesses about things like Properties syntax were close to the mark. This will be a big help for checking that my parser is doing what it should.

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #481 on: October 14, 2018, 06:13:33 PM »
I have some ideas for an Icon 2.0 built on top of the Paragon Chat engine that is kind of a COH development swiss army knife. For that I want to try to go a step further and get the actual editor working, but that's purely hypothetical at this point.

Given what I've seen of the editor UI now that I've had a taste of it, I think that would be a pretty cool addition to the PChat toolset. I assume, though, that it means actually running some sort of proxy map server; moreso, that is, than PChat already pretends to be.

Well, if there's anything I can do to help make it less hypothetical, let me know. I won't pretend that I'm "skilled" in this stuff yet, but I think I'm at the level that I could do some basic grunt work for you if you need it. (The day I get my own geo parser going and get a home-brew map viewer online, I'll give myself my first gold star, heh.)


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,716
  • Moar Dots!
Re: Technical side discussion
« Reply #482 on: October 15, 2018, 12:49:13 AM »
I assume, though, that it means actually running some sort of proxy map server; moreso, that is, than PChat already pretends to be.

PChat is a lot closer to a real mapserver than many people assume. It kind of has to be -- the mapserver protocol is so tightly bound to implementation details shared by both client and server that it's almost impossible to speak the protocol in any useful way without actually doing things the same way that the server did.

It's for that reason that if I was going to try to build a map editor, or something that does anything beyond what Icon is capable of, I'd probably use PChat as a starting point for it. The core is modular enough that it doesn't have to use XMPP as a sync protocol (see also: offline mode).

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #483 on: October 15, 2018, 04:16:21 PM »
PChat is a lot closer to a real mapserver than many people assume.

I think it's less that people don't realize and more that we all wear self-imposed blinders because otherwise we'd all by asking "Why aren't Codewalker/Leandro/anybody-at-all taking the next step and making an emulator?" ;-)

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #484 on: October 16, 2018, 03:17:09 PM »
Codewalker, upstream when we talked about GuyPerfect's Atlas Park Echo mod, you said:

In order to make it work, you'll need to put a copy of the geobin for the map into the appropriate path under ParagonChat\data as well as the client's data folder.

This doesn't work for me. I copy Guy's mod verbatim into ParagonChat\data, identical to how it's installed in ParagonChat\client\data. If you've got a few minutes sometime, could you verify that it works for you?

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #485 on: October 19, 2018, 04:51:28 PM »
Apropos to nothing - today's Girl Genius comic (the last few panels; the context of the comic is an attempt to cure a lead character of mind control embedded in his brain) illustrates how I frequently feel when I imagine all of the moving parts involved with just wanting to tell my bot "Run away from me and don't get stuck on the terrain."


spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #486 on: December 06, 2018, 12:00:25 AM »
I'm still chugging along with my map viewer, progress has been slow as I don't have a great deal of time at the moment. I got to the point of loading a map geobin, reading the defs, then attempting to traverse the scene graph via the refs. When a group name contains a slash then it's either a def in another geobin, or a model in a geo. The first problem there is which one is it? I just go looking for a geobin, and if there is one see if it has a def which matches the name. Then I ran into the problem that the group name containing the slash doesn't give the name of the actual geobin file, just a folder, and some folders contain multiple geobins. Now going back through this thread it looks like defnames.bin may be what I'm looking for, I will look at that next.

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #487 on: December 06, 2018, 01:22:53 PM »
Where's the schema for defnames.bin? defs.xml doesn't look right for what I'm seeing in the file...

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #488 on: December 06, 2018, 02:05:19 PM »
I think I figured it out by looking at the file, and some of the stuff you guys said before. It looks like this:

Code: [Select]
int32 file_count;
string file_names[file_count];

int32 def_count;
Def defs[def_count];

struct Def
{
   string name;
   uint16 is_geo;
   uint16 file_index;
}

A couple of things I don't understand - even with a def which is NOT a geo, but is a geobin, the file in the file name array still has the .geo extension, but in some cases at least no such geo exists, only the bin in the geobin folder.

When the def is a model from a geo, there's also a bin in the geobin folder, which will contain a def with the same name. What's that for? I'm unsure at what point a group in a def is actually referencing a model from a geo.

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #489 on: December 06, 2018, 04:41:04 PM »
With the holidays, my workload has increased and my free time has decreased, so I've been taking a break; long enough that I'm starting to forget some things and am going to need a refresher soon. :-/ That's one of the reasons that I so frequently detail things I learn here, so that I can find them in the correct context three months down the road.

Anyway - I think you're finding the same kind of thing that initially confused me, when I ended up writing this:

Quote
What I finally figured out is that it works like this:

idx1 = Uint16
idx2 = Uint16
idx1 = idx1 & 0X7F (Mask off low bit of idx1 where idx1 is little-endian)
IndexVal = idx1 + idx2

My experience is that what you've labeled "is_geo" isn't JUST a boolean flag. There's offset data in there also, even though in many cases it evaluates to zero and makes "is_geo" APPEAR to be just a boolean flag. If you set the flag bit to zero and then take what remains in "is_geo" and add it "file_index", then you should get the ACTUAL file_index for the thing you're trying to find. I think this is another example data being stored as a C language union structure, but I don't pretend to be the reverse engineer that Codewalker is.

Anyway - there seems to be a lot of "obsolete" data in the object library, and some of it also seems to be incomplete, which is probably not unexpected considering that we're working with a beta build of an issue that was never officially completed and released.

Also, things will get more interesting once you start to try correlating tricks to geos. :-o

« Last Edit: December 06, 2018, 04:54:26 PM by slickriptide »

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #490 on: December 06, 2018, 04:48:33 PM »
I couldn't find any "Def"s in that array where the "is_geo" uint16 was greater than 1 though?

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #491 on: December 06, 2018, 05:22:05 PM »
Have you dumped defnames.bin with bindump?

It has two sections. The first is a table of geo files. The second is the table of defnames that index into that table of geo files. Anything that ends in .geo is NOT a defname. It's the thing that a defname in the second table resolves to. Your code snippet looks like you're treating the whole defnames.bin file as a single homogenous table.


spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #492 on: December 06, 2018, 05:30:03 PM »
So the first "table" is just a list of file names
Code: [Select]
int32 file_count;
string file_names[file_count];

And the second table is the defnames.
Code: [Select]
int32 def_count;
Def defs[def_count];

struct Def
{
   string name;
   uint16 is_geo;
   uint16 file_index;
}

Which row are you seeing where the first uint16 is > 1?

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #493 on: December 06, 2018, 06:09:54 PM »
/headsmack.

Never mind me, this is what happens when you work from bindump output instead of from the actual file data and make assumptions based on that. I'm the one who's confused here, not you. (The erroneous original assumption was that the data in defnames.bin was all uint32, just like in the geobins. That's not the case, though. Then I compounded it by assuming that your intial file_names array was the array of source file names in the bin file header.)

So, given that, maybe it will help if you give some examples of the entries that don't appear to be working out correctly for you. It sounds like you're talking about objects that are made up of multiple geos but after the above headsmacking, it would be better to know exactly what you're seeing before assuming anything.
« Last Edit: December 06, 2018, 08:16:42 PM by slickriptide »

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #494 on: December 06, 2018, 08:01:29 PM »
A couple of things I don't understand - even with a def which is NOT a geo, but is a geobin, the file in the file name array still has the .geo extension, but in some cases at least no such geo exists, only the bin in the geobin folder.

I'm assuming here that we're talking about something like this: object_library/Omni/EncounterSpawns/P_City_00_03/P_City_00_03.geo. Where, if you follow it into the actual object_library folder instead of the geobin folder, there isn't any P_City_00_03 folder and thus no geos below that non-existent folder.

This goes back to the discussion of textures upstream, about how there are all these definitions for .TGA files when there aren't many, or any, actual Targa files in the data. The textparser that compiles the raw specification files into the bin files plays fast and loose with file extensions and frequently ignores them entirely.

The important thing from the game client's standpoint is that there IS a P_City_00_03 folder in the geobins, with an associated P_City_00_03.bin and P_City_00_03.bounds underneath it. It doesn't care that P_City_00_03.geo doesn't exist because it doesn't actually use that file index result to tell it what model to use because the is_geo flag is false. Instead, it opens P_City_00_03.bin and adds it to the scene graph as an object, and then adds all of the appropriate models below that object. In this particular case, it ends up loading spawn markers of the sort you see when you've got developer mode turned on in Icon or Pchat. Essentially, they're always there in the zone but usually they are invisible and are marked as no-collision.

In short - it's marked with a .geo because everything in the defnames filename index is marked as a .geo; even things that don't need to be. The "is_geo" flag differentiates real models from objects made up of other models.

When the def is a model from a geo, there's also a bin in the geobin folder, which will contain a def with the same name. What's that for? I'm unsure at what point a group in a def is actually referencing a model from a geo.

As near as I can tell, when a leaf def has its "object" attribute set to the name of the object, that means it's referencing a model inside of the geo that corresponds to the same name and path as the bin file that holds the leaf def. That's why, for instance, there's no P_City_00_03/P_City_00_03.geo. All of the models that might conceivably appear in that geo file already exist in the base city file geos. The bin files "point" to the scene graph segments that ultimately contain them. The client doesn't give a hang about how the label reads in the defnames index as long as it can find whatever it needs to find.
« Last Edit: December 06, 2018, 08:09:56 PM by slickriptide »

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #495 on: December 07, 2018, 09:35:20 AM »
This goes back to the discussion of textures upstream, about how there are all these definitions for .TGA files when there aren't many, or any, actual Targa files in the data. The textparser that compiles the raw specification files into the bin files plays fast and loose with file extensions and frequently ignores them entirely.
Ah I see, so the file extensions are largely useless, ok I can cope with that.

As near as I can tell, when a leaf def has its "object" attribute set to the name of the object, that means it's referencing a model inside of the geo that corresponds to the same name and path as the bin file that holds the leaf def. That's why, for instance, there's no P_City_00_03/P_City_00_03.geo. All of the models that might conceivably appear in that geo file already exist in the base city file geos. The bin files "point" to the scene graph segments that ultimately contain them. The client doesn't give a hang about how the label reads in the defnames index as long as it can find whatever it needs to find.
Interesting, I hope Codewalker can confirm/deny that this is what the "object" field is for.

If the "object" field means "model inside geo corresponding to same name/path as this geobin", I wonder if that means that all instances of models MUST come from def instances from the geo's corresponding geobin, or whether that's not necessarily the case.

E.g. if I have my map city.bin, scenery.geo and corresponding geobin scenery.bin. I wonder if city.bin can directly include models from scenery.geo, or whether it would have to have instances of defs from scenery.bin which then contain the models from scenery.geo.

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #496 on: December 07, 2018, 01:32:47 PM »
It looks like you're right. I knocked up a bin->json converter to help me follow one def through this crazy russian doll situation all the way to a model in a geo.

I picked a random geobin line from my defnames text dump:
Code: [Select]
geobin:1578:_ES_Des_Loiter_D9_P_City_00_02
// that row is...
1578:object_library/Omni/EncounterSpawns/P_City_00_02/P_City_00_02.geo

So then looking at object_library/Omni/EncounterSpawns/P_City_00_02/P_City_00_02.bin, I look for a def called "_ES_Des_Loiter_D9_P_City_00_02", I find this:
Code: [Select]
"name": "_ES_Des_Loiter_D9_P_City_00_02",
"type": "",
"obj": "",
"groups": [
{
  "name": "object_library\/Omni\/EncounterSpawns\/BaseTypes\/ES_BrawlShipRec_BaseType",
  "position": {
    "x": 0,
    "y": 0,
    "z": 0
  },
  "rotation": {
    "x": 0,
    "y": 0,
    "z": 0
  }
}
]

(Don't mind the screwed paths) So take the end of the path "ES_BrawlShipRec_BaseType" and search in defnames, I find this:
Code: [Select]
geobin:1486:ES_BrawlShipRec_BaseType
// that row is
1486:object_library/Omni/EncounterSpawns/BaseTypes/BaseTypes.geo

This is another geobin, so I look in object_library/Omni/EncounterSpawns/BaseTypes/BaseTypes.bin for "ES_BrawlShipRec_BaseType":
Code: [Select]
"name": "ES_BrawlShipRec_BaseType",
"type": "",
"obj": "",
"groups": [
{
  "name": "Omni\/EncounterSpawns\/Encounter_V_40",
  "position": {
    "x": 0,
    "y": 0,
    "z": 0
  },
  "rotation": {
    "x": 0,
    "y": 0.872665,
    "z": 0
  }
},
{
  "name": "Omni\/EncounterSpawns\/Encounter_E_01",
  "position": {
    "x": 11.5,
    "y": 0,
    "z": -8
  },
  "rotation": {
    "x": 0,
    "y": 1.919862,
    "z": 0
  }
},
// etc

I thought I'd look at "Omni\/EncounterSpawns\/Encounter_E_01", so I search defnames for Encounter_E_01:
Code: [Select]
geobin:1537:Encounter_E_01
// which is this row
1537:object_library/Omni/EncounterSpawns/EncounterSpawns.geo

So I look in object_library/Omni/EncounterSpawns/EncounterSpawns.bin for Encounter_E_01:
Code: [Select]
"name": "Encounter_E_01",
"type": "",
"obj": "",
"groups": [
{
  "name": "_encounter_E_01",
  "position": {
    "x": 0,
    "y": 0,
    "z": 0
  },
  "rotation": {
    "x": 0,
    "y": 0,
    "z": 0
  }
}
]

This doesn't have a path, so I look in the same geobin for a def with that name:
Code: [Select]
"name": "_encounter_E_01",
"type": "",
"obj": "_encounter_E_01",
"groups": [ ]

The obj field is set, so look in defnames for "_encounter_E_01":
Code: [Select]
geo:1537:_encounter_E_01
// that row is
1537:object_library/Omni/EncounterSpawns/EncounterSpawns.geo

So finally this is one which is a geo rather than geobin, load up "object_library/Omni/EncounterSpawns/EncounterSpawns.geo" in geodraw, and I find this:
Code: [Select]
13  _encounter_E_01__ENEMY
Hooray!

Now the thing I'm wondering is - can a def have the "obj" field set (therefore having a model), as well as having child groups which also have models etc? Should be easy enough to find out.

slickriptide

  • Elite Boss
  • *****
  • Posts: 317
Re: Technical side discussion
« Reply #497 on: December 07, 2018, 03:53:33 PM »
E.g. if I have my map city.bin, scenery.geo and corresponding geobin scenery.bin. I wonder if city.bin can directly include models from scenery.geo, or whether it would have to have instances of defs from scenery.bin which then contain the models from scenery.geo.

Defnames is the ultimate arbiter.  You should be able to reference any model "directly" as long as you have a parent group def that contains its attributes (loc, pyr, etc...) and the model def has an entry in defnames that indexes to the geo file that contains the model. The point of the object library is to avoid doing that sort of thing, of course, but it doesn't confine you to only doing things that way. It's more that the scene graph organization of the object library naturally conforms to the structure of the object library itself.

Now the thing I'm wondering is - can a def have the "obj" field set (therefore having a model), as well as having child groups which also have models etc? Should be easy enough to find out.

Probably? There's nothing preventing it, but you'd be breaking the spec, so to speak.

I've played a bit with the built-in scene editor and the concept behind it seems to be that a "group" is a container and everything else represents "things" that are being contained and defined. You don't typically find "things" with attributes added or with children depending from them. "Groups" provide the structure and the instructions for placing or otherwise modifying the "things". I think the behavior of the client if you fed it an "object" that also pretended to be a "group" with children would be considered "undefined".
« Last Edit: December 07, 2018, 04:05:27 PM by slickriptide »

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #498 on: December 07, 2018, 08:40:45 PM »
Probably? There's nothing preventing it, but you'd be breaking the spec, so to speak.
I don't mean it's something I want to do, I wonder if any of the game geobins do it. I think I'm going to have to write a few tests to load every geobin in the game and test our theories and assumptions. I'll report back any findings :)

spectre1989

  • Minion
  • **
  • Posts: 31
Re: Technical side discussion
« Reply #499 on: December 10, 2018, 12:53:59 AM »
Ok so as there turns out, there's a fair few defs which have both an obj (so therefore a model?) as well as groups. There were quite a few, so if you're interested there's a pastebin here.

On the question of whether a geobin can directly include a model from a geo which also has a corresponding geobin (rather than including a def from the corresponding geobin) - yes, all the time.

One last interesting finding was that there's a single def in all of the geobins which has an obj field which doesn't actually exist in defnames.