Author Topic: Technical side discussion  (Read 164696 times)

Paragon Avenger

  • Circles and Triangles
  • Elite Boss
  • *
  • Posts: 6,246
Re: Technical side discussion
« Reply #420 on: August 25, 2018, 04:25:12 AM »
div {
    color: #FFFFDD;
    background-color: white;
    font-weight: 900;
}

You're welcome.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #421 on: August 27, 2018, 04:05:47 PM »
Why is it that when you grab a tool on the internet, even a tool that is doing conceptually simple things and that has a track record of working with much more esoteric sorts of jobs, that inevitably a bug or an unexpected issue arises to throw a monkey-wrench into the works?

I may end up having to roll my own geobin parser. Tedious, but not particularly difficult thanks to knowing the spec ahead of time. I was kind of hoping to avoid the tedium and have a tool that I could use to likewise quickly prototype costume and entity parsers. If the Kaitai people confirm that I found a bug and that they aren't in a hurry to fix it, then my little project here will once again put me in the position of re-inventing the wheel - in this case, a version of bindump, albeit one that only speaks Parse6 and that has an API that a bot can hook into if it needs to. I'm hoping it doesn't come to that, though.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #422 on: August 27, 2018, 08:08:54 PM »
Why is it that when you grab a tool on the internet, even a tool that is doing conceptually simple things and that has a track record of working with much more esoteric sorts of jobs, that inevitably a bug or an unexpected issue arises to throw a monkey-wrench into the works?

If I had a nickel for every time I said that, I would have collapsed into a black hole by now.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #423 on: August 28, 2018, 10:33:50 PM »
Step One: Create a Kaitai grammar for geobin format.
Step Two: ????
Step Three: Profit!!!

Step One is complete. Since they say up-front that their Python support is dodgy and I've seen one bug myself before I even really got started, I'm being cautiously optimistic until I've managed to actually do something with their generated Python code that spits out the same data that bindump spits out. The web IDE definitely is able to parse the map file successfully. Worst case is that I'd end up making a stand-alone C program and using "geodump" to dump the geobin data I care about into XML that can be transferred into a database that the bot can deal with.



For the curious, here's the Kaitai grammar for geobin.xml:

Code: [Select]
meta:
  id: geobin
  file-extension: bin
  endian: le

seq:
  - id: header
    type: root_header
  - id: bin_data
    type: root_root
   
types:
  root_header:
    seq:
      - id: signature
        type: str
        encoding: ASCII
        size: 8
      - id: crc
        type: u4
      - id: parse_version_size
        type: u2
      - id: parse_version
        type: str
        encoding: ASCII
        size: parse_version_size
      - id: file_string_size
        type: u2
      - id: file_string
        type: str
        encoding: UTF-8
        size: file_string_size
      - id: table_size
        type: u4
      - id: file_count
        type: u4
      - id: table_data
        size: table_size
 
  files_rec:
    seq:
      - id: filename
        type: pad_string
      - id: filedate
        type: u4
 
  root_root:
    seq:
      - id: version
        type: u4
      - id: scenefile
        type: pad_string
      - id: loadscreen
        type: pad_string
      - id: def_count
        type: u4
      - id: def_array
        type: root_def
        repeat: expr
        repeat-expr: def_count
      - id: ref_count
        type: u4
      - id: ref_array
        type: root_ref
        repeat: expr
        repeat-expr: ref_count
      - id: map_imports
        size-eos: true
       
  pad_string:
    seq:
      - id: strlen
        type: u2
      - id: strval
        type: str
        size: strlen
        encoding: ASCII
      - id: padding
        size: (4 - _io.pos) % 4
        doc: This adds padding to the next full four-byte word. Also, generated Python code may have a bug (duplicate /) that needs to be checked in the output code here.

  root_ref:
    seq:
      - id: ref_size
        type: u4
      - id: ref_body
        type: root_ref_body
        size: ref_size
 
  root_ref_body:
    seq:
      - id: ref_str1
        type: pad_string
      - id: ref_pos_x
        type: f4
      - id: ref_pos_y
        type: f4
      - id: ref_pos_z
        type: f4
      - id: ref_pyr_pitch
        type: f4
      - id: ref_pyr_yaw
        type: f4
      - id: ref_pyr_roll
        type: f4
       
  root_def:
    seq:
      - id: def_size
        type: u4
      - id: def_body
        type: root_def_body
        size: def_size
     
  root_def_body:
    seq:
      - id: def_name
        type: pad_string
      - id: group_count
        type: u4
      - id: group_array
        type: root_def_group
        if: group_count > 0
        repeat: expr
        repeat-expr: group_count
      - id: property_count
        type: u4
      - id: property_array
        type: root_def_property
        if: property_count > 0
        repeat: expr
        repeat-expr: property_count
      - id: tintcolor_count
        type: u4
      - id: tintcolor_array
        type: root_def_tintcolor
        if: tintcolor_count > 0
        repeat: expr
        repeat-expr: tintcolor_count
      - id: ambient_count
        type: u4
      - id: ambient_array
        type: root_def_ambient
        if: ambient_count > 0
        repeat: expr
        repeat-expr: ambient_count
      - id: omni_count
        type: u4
      - id: omni_array
        type: root_def_omni
        if: omni_count > 0
        repeat: expr
        repeat-expr: omni_count
      - id: cubemap_count
        type: u4
      - id: cubemap_array
        type: root_def_cubemap
        if: cubemap_count > 0
        repeat: expr
        repeat-expr: cubemap_count
      - id: volume_count
        type: u4
      - id: volume_array
        type: root_def_volume
        if: volume_count > 0
        repeat: expr
        repeat-expr: volume_count
      - id: sound_count
        type: u4
      - id: sound_array
        type: root_def_sound
        if: sound_count > 0
        repeat: expr
        repeat-expr: sound_count
      - id: replacetex_count
        type: u4
      - id: replacetex_array
        type: root_def_replacetex
        if: replacetex_count > 0
        repeat: expr
        repeat-expr: replacetex_count
      - id: beacon_count
        type: u4
      - id: beacon_array
        type: root_def_beacon
        if: beacon_count > 0
        repeat: expr
        repeat-expr: beacon_count
      - id: fog_count
        type: u4
      - id: fog_array
        type: root_def_fog
        if: fog_count > 0
        repeat: expr
        repeat-expr: fog_count
      - id: lod_count
        type: u4
      - id: lod_array
        type: root_def_lod
        if: lod_count > 0
        repeat: expr
        repeat-expr: lod_count
      - id: def_type
        type: pad_string
      - id: def_flags
        type: u4
        enum: def_enum1
      - id: def_alpha
        type: f4
      - id: def_obj
        type: pad_string
      - id: texswap_count
        type: u4
      - id: texswap_array
        type: root_def_texswap
        if: texswap_count > 0
        repeat: expr
        repeat-expr: texswap_count
      - id: soundscript
        type: pad_string
      - id: leftovers
        size-eos: true

    enums:
      def_enum1:
        0x00000001: ungroupable
        0x00000002: fadenode
        0x00000004: fadenode2
        0x00000008: nofogclip
        0x00000010: nocoll
        0x00000020: noocclusion
        0: none
       
  root_def_group:
    seq:
      - id: group_size
        type: u4
      - id: group_name
        type: pad_string
      - id: pos_x
        type: f4
      - id: pos_y
        type: f4
      - id: pos_z
        type: f4
      - id: pyr_pitch
        type: f4
      - id: pyr_yaw
        type: f4
      - id: pyr_roll
        type: f4
      - id: group_flags
        type: u4
        enum: group_enum1
   
    enums:
      group_enum1:
        0x00000001: disableplanarreflections
        0: enableplanarreflections

       
  root_def_property:
    seq:
      - id: prop_size
        type: u4
      - id: prop_str1
        type: pad_string
      - id: prop_str2
        type: pad_string
      - id: prop_int1
        type: u4
       
  root_def_tintcolor:
    seq:
      - id: tint_size
        type: u4
      - id: tint_uarray1_0
        type: u4
      - id: tint_uarray1_1
        type: u4
      - id: tint_uarray1_2
        type: u4
      - id: tint_uarray2_0
        type: u4
      - id: tint_uarray2_1
        type: u4
      - id: tint_uarray2_2
        type: u4
       
  root_def_ambient:
    seq:
      - id: ambient_size
        type: u4
      - id: ambient_uarray1_0
        type: u4
      - id: ambient_uarray1_1
        type: u4
      - id: ambient_uarray1_2
        type: u4

  root_def_omni:
    seq:
      - id: omni_size
        type: u4
      - id: omni_uarray1_0
        type: u4
      - id: omni_uarray1_1
        type: u4
      - id: omni_uarray1_2
        type: u4
      - id: omni_f1
        type: f4
      - id: omni_flags
        type: u4
        enum: omni_enum1
   
    enums:
      omni_enum1:
        0x00000001: negative
        0: positive

  root_def_cubemap:
    seq:
      - id: cm_size
        type: u4
      - id: cm_int1
        type: s4
      - id: cm_int2
        type: s4
      - id: cm_f1
        type: f4
      - id: cm_f2
        type: f4

  root_def_volume:
    seq:
      - id: vol_size
        type: u4
      - id: vol_f1
        type: f4
      - id: vol_f2
        type: f4
      - id: vol_f3
        type: f4

  root_def_sound:
    seq:
      - id: sound_size
        type: u4
      - id: sound_str1
        type: pad_string
      - id: sound_f1
        type: f4 
      - id: sound_f2
        type: f4 
      - id: sound_f3
        type: f4
      - id: sound_flags
        type: u4
        enum: defsoundenum1
   
    enums:
      defsoundenum1:
        0x00000001: exclude
        0: include

  root_def_replacetex:
    seq:
      - id: rt_size
        type: u4
      - id: rt_int1
        type: s4
      - id: rt_str1
        type: pad_string

  root_def_beacon:
    seq:
      - id: beacon_size
        type: u4
      - id: beacon_str1
        type: pad_string
      - id: beacon_f1
        type: f4
   
  root_def_fog:
    seq:
      - id: fog_size
        type: u4
      - id: fog_f1
        type: f4
      - id: fog_f2
        type: f4
      - id: fog_f3
        type: f4
      - id: fog_uarray1_0
        type: u4
      - id: fog_uarray1_1
        type: u4
      - id: fog_uarray1_2
        type: u4
      - id: fog_uarray2_0
        type: u4
      - id: fog_uarray2_1
        type: u4
      - id: fog_uarray2_2
        type: u4
      - id: fog_f4
        type: f4

  root_def_lod:
    seq:
      - id: lod_size
        type: u4
      - id: far
        type: f4
      - id: far_fade
        type: f4
      - id: scale
        type: f4

  root_def_texswap:
    seq:
      - id: ts_size
        type: u4
      - id: ts_str1
        type: pad_string
      - id: ts_str2
        type: pad_string
      - id: ts_int1
        type: s4


slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #424 on: August 29, 2018, 05:18:55 PM »
If you want just a very rough idea of the terrain and are willing to build the whole scene graph (including the object library), one thing you could do is load the .bounds files instead of trying to parse geometry.

Sorry for "thinking out loud" but it helps me think. If someone had told me from the beginning that something as conceptually simple as "run away from me until I catch you or you reach a target destination" was going to involve learning how to solve real game development problems, I would probably have just said, "I guess I'll wait until Codewalker solves it."

In a lot of ways, I should just "settle" for something that I can develop and run with right now. Right now, today, I could make a system where I lay out a bunch of waypoints on a map and script a bot to travel between those waypoints. Doing that would give me some concrete results.

It would also be completely mechanical. What I desire is to instruct a bot to "go to City Hall" and have it find its own path to that waypoint, or to instruct it "run away from me" and have it find its own flee path without a specific destination in mind, and without following a predictable path on rails. In short, I want a solution that works for any map in the game, and not something that is specific to a particular map, and I want to avoid trying to "render" geometry where it's at all possible.

Currently, and the reason I'm working to be able to read a city zone geobin file, I've been planning to use enemy spawn points and existing navigation beacons as a network of pre-existing waypoints where the ground level is known and the waypoints are guaranteed to be "outside". (For the moment, I'm ignoring the fact that areas like "indoors" City Hall or Freedom Corp exist. I can eventually remove those nodes from the network.) If the bot occasionally falls through the ground on a hillside, or gets momentarily hung up on a fence, I'm okay with that for the time being because the distance between most waypoints is short enough that Rover's "ground level" will be corrected quickly.

Outside of waypoints, I've been reading about navigation meshes and that got me thinking about the bounding boxes again. If a bot can compute the bounding boxes in the local area, then invert that model, it would essentially have a sort of dynamically generated navigation mesh where any point within that "inverted" area is a point it can travel through. The waypoint network would still be useful as a sanity check for "where's the ground" but the bot would be free to plot a path anywhere in the general area of the waypoints instead of strictly on rails between them.

However, first things first. If a navigation mesh, of sorts, is to work as I've outlined above then I still need to first have a working network of waypoints, so that remains the current goal. Create a network of waypoints. Create a pathing algorithm to traverse the network given a start and a destination. Generalize it to work with any existing zone or mission map.


slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #425 on: August 30, 2018, 04:40:26 PM »
Okay, time to admit that I've reached the limits of my current skillset and that I am just not that interested in learning how to parse the geometry of a city zone and then ray cast around in it to find collisions.

My original idea/theory was that I could use the pre-existing spawn points and beacons as a kind of neural network. If I plotted all of the edges between all of the points and then did an A* search or something similar, the final path would, in most cases, go around large obstacles and get hung up only briefly on small obstacles that it didn't also go around. I've lost confidence in this idea recently as I've been learning about just what a pain path-finding is in real gaming situations, but I might still try pursuing it just to see where it leads. This might hold some promise if I ever got a good enough understanding of the geometry/bounds situation to be able to test each possible edge and eliminate any that intersected the bounds box of an object, then I hand-added any "missing" paths for special cases like, say, walking around underneath Atlas. Since this is basically ray-casting, it's a question of whether I want to learn yet another thing (or more likely a half-dozen other things related to OpenGL) that I never intended to learn in order to accomplish it.

My alternate idea/theory was that I could do a movement-based method of path-finding instead of graph-based, where the bot would "look" in the direction it was facing towards the nearest known waypoint, and be "pulled" toward it if it was a beacon/spawn or "pushed" away from it if it was an object. The problem with this idea is that a whole bunch of the "objects" in the Atlas Park map, for instance, are buildings with walkable plazas attached. Doing things this way, the bot would completely avoid walking near the train station ramp, let alone up it OR under it, and likewise it would never even attempt to walk under the statue of Atlas. I have no clue at all how it would handle something like the Croatoa forest paths or Perez Park.

Yeah, and the idea of somehow rendering all of the bounds boxes into some kind of graph and turning that into a navigation mesh is making my head spin.

So, I guess I'm just going to have to deal for now with the fact that my bot is going to have to have its paths on any given map graphed out ahead of time. For the case where I want the bot to "run away" or "run to $DESTINATION" I'll have to just brute force trace all of the paths specifically for that particular map and try to make enough vertices that a bot starting from point A and running to point B will at least feel different from a bot starting from point A and running to point C or point B'.

Besides - it's going to have to do it that way anyway, if it's inside a base instead of on a zone or mission map. Bases don't have a map file with all of the geometry laid out ahead of time.

And people wonder why nobody just "makes an emulator". It's really a shame that nobody at Paragon Studios smuggled out a dvd of the server source code.

On the plus side, if you'd tried talking to me about "A* searching" a couple of weeks ago, I probably would have given you a blank look.
« Last Edit: August 30, 2018, 05:03:23 PM by slickriptide »

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #426 on: September 04, 2018, 10:14:25 PM »
At the risk of appearing to be using this thread as my personal blog, I'm going to toss out a few more ideas I'm thinking about; both to codify them in my own mind and to put them out there where smarter people who already know about these things can say, "No; that's a terrible idea." (Though, I'm keeping in mind Arcana's admonition that paraphrases to, "If people only ever waited for perfect solutions, most of the time they end up doing nothing.") OTOH, if I'm turning into Joshex, then someone needs to slap me upside the head and say, "Hey! Wake up!"

The biggest problem I have with all of this is that I'm having to embrace the idea of Rover as a pseudo-game-client and I have zero experience with 3D environmental programming. Despite that, and despite the fact that anything I do currently is likely to be rendered obsolete the first day that Codewalker  posts, "Hey, I've added NPC's and LUA scripting to Paragon Chat!",  I seem to still be stuck with a desire to see Rover "run away" in a "life-like" fashion instead of in a "train on rails" fashion.

So, I have two semi-related problems to solve: One is pathfinding, and the other is terrain navigation.

"Semi-related" because you can't begin to understand where the paths are without first understanding where the terrain is. This is the bit that I've been having trouble wrapping my head around. Scene graphs, I get, mostly. Collision math, I get, mostly. Determining how you decide what things to test for collision without having to test EVERYTHING, every clock tick,  is the bit that's been eluding me.

The first step was to read a geobin file. I've got a parser that can do that now, even though I haven't put it to use yet. As an exercise of sorts, I DID make a clientmessages parser and generate a strings.cfg file from it. I can now modify all the strings in my personal Paragon Chat instance. Whee! I also learned, for instance, that the game client pre-generated instance names for every city zone up to around fifty instances, which is interesting, if not immediately useful. (Though it did make me wonder if I could ADD strings to strings.cfg that don't exist in clientmessages. I'm not sure what Paragon Chat would do with that. Probably just ignore it as a presumed typo.)

The point being that if I could use my Kaitai-generated parser to load up the whole clientmessages file without running out of memory, access the "P-strings" array and use that info to access the associated strings in the strings array, and write the whole thing out as a strings.cfg file then I should, in principle, be able to do the same process with the geobin parser. That is, I assume at this point that the parsers I'm generating out of Kaiti (I made a .bounds parser as well) will work as expected with minor touchups to insure that text decoding happens correctly. (Python gets touchy about insisting that things conform to your current code page if you're working in a Windows environment.)

Scene Graph
So, now I can theoretically read a zone map geobin, which is basically a serialized scene graph. So, the question is, "How do you deserialize it?"

Based on the output of bindump, it looks a lot like the original scene graph is being traversed and written out "leaves-first". The Refs array represents the first-level children of the scene Root. I'm not really sure why TextParser felt the need to write the "eldest" children seperately from the others, but there you go.

In any case - the first Defs encountered in the geobin file are the outer "leaves" of the grp_Atmosphere_NPC "branch", which is the first entry in the Refs array. Each group represents its own "branch" that can contain many child groups. The children refer back to their parent, and the parents eventually lead back to one of the root Refs. A parent group can contain multiple instances of a single child group. Defnames that contain file paths lead into the object library, where the geometry lives along with bin files describing each individual model in the .geo file as well as groups of models, and bounds data for the models, represented as both rectangular and spherical bounds.

Once all of the grp_Atmosphere_NPC nodes are read, the next node is a grand-child of Ref grp_audio, which is the next Ref in the Refs array. It appears safe to expect each of the following Refs to be "untraversed" in turn in order to completely deserialize the data in the bin file and arrive at something that resembles the original scene graph.


The issue of geometry rears its head at this point, since the ultimate termination of any particular path through the scene graph should be a piece of geometry that can be drawn or manipulated. Here is where I intend to cheat, using the advice Codewalker recommended last year - My bot can't actually "see" the world and it never intends to render anything visually. The contents of the actual .geo file is irrelevant. All that the bot cares about is the collision bounds. Which means that as far as the bot is concerned, it is living in either Boxville or Spheretown. Either way, the "geometry" of the zone is faked at this point using the data from the .bounds file for the individual objects in the zone. As far as I can tell, EVERYTHING is an object of some sort. Even the streets and sidewalks and plazas are all objects with a bounding box defined as well as the radius of a sphere that presumably likewise encompasses the object.

In short - there is no "ground", as such. Rather, the "ground" is whatever "object" an entities feet happen to be colliding with due to gravity. (I haven't actually followed this line of logic through for a forested/outdoor zone like Croatoa, though I EXPECT it to apply the same way.)

I should emphasize at this point that my tossing around terms like "scene graph" is not some indication that I'm at some high level understanding of the term. When it comes to most of this 3D visualization and rendering stuff, I can say that I'm past the level where the terms represent some sort of mystical incantation that causes pretty pictures to appear on a computer screen. Beyond that, I'm still pretty much Harry Potter in his first year at Hogwarts, and I'm definitely no Hermione.

So, now the question is: What do I do with this scene graph?

For starters, if Rover is going to be acting like a pseudo-client anyway then I figure I should use software suited to the task. That means loading up Openscenegraph and using it to manage the de-serialized scene graph for me. The minimal benefits are that I'll be using software designed for the task I'm trying to accomplish and if I want to actually visualize Boxtown or Sphereville for some reason, it should be possible to come up with some coding examples that illustrate how to do that.

It isn't entirely clear to me yet just how much collision detection is available in Openscenegraph (OSG, hereafter). There are different versions and the documentation ranges from recent to over a decade old. I THINK that I might be able to insert Rover's position and bounding box into the scene graph, and have it automagically incorporated into the scene, such that casting a ray in any given direction will return a list of objects that are potentially things Rover would collide with.

If that doesn't work, I've also found a fallback - Open Dynamic Engine (hereafter, ODE). This is more or less a physics engine sort of like Phys X but intended for doing rigid body simulations, with the "dynamic motion" component separated from the "collision detection" component, so you can use either of them standalone. Plus, ODE has a reasonably simple and easy-to-understand idea of the World and of Spaces. I THINK it would be relatively simple to load Boxville into a ODE World instance and have it treat all of the boxes as motionless/massless bits of terrain and then ODE can, given the position of Rover, deliver a list of collision points that he's likely to hit at his current position and velocity.

Either of these solutions results in Rover sort of "feeling" his way around the geometry of the world, without actually knowing the precise geometry of the world. Practically speaking, I think it would work like I mentioned back when I first brought this up last year, where I'd use the collision points to create an avoidance force that "repelled" Rover away from the obstacle but hopefully still in the general direction he "wants" to go in.

Pathfinding

The other benefit of the fully realized scene graph is that I've got the list of beacons and spawn points that I've mentioned several times previously. These are suitable for use as path waypoints, since each of them is guaranteed to be at "ground level" (with a few exceptions, like the blimp's path around Atlas Park). However - Waypoints without any paths between them aren't really useful.

The solution, as I see it, is to build a "neural net" graph that connects every waypoint with every other waypoint. I then feed that graph into a scanner whose job is to brute-force test each of the resulting line segments for collisions using the scene graph/terrain visualization. Segments that come up with a positive hit are dropped from the database. POSSIBLY, the scanner could check the height of the bounding box for the offending object and if it was shorter than some arbitrary jump height, that segment could be broken into two new segments that both terminate at the object and that both are marked as "jump over" at the endpoints near the obstacle. I think that would fall into "stretch goal" territory.

The end result would be a database of waypoints that each connect to at least one other waypoint, forming a graph of the "walkable" space in the zone. Such a graph could then be searched with a A* search or some similar pathfinding algorithm to dynamically find a path from any "point A" to any "point B" that lies within the graph. Such a database could be generated once and then used by any bot, even if that bot knows nothing at all about the geometry of the zone. As long as it was located somewhere on the "pathnet", it could find a way to every other part of it.

I'm only just at the barely beginning stages of realizing any of the above vision, but I believe that the above actually represents an achievable "spec" for a bot that can wander freely around any zone map or mission map in the game, given appropriate preparation.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #427 on: September 06, 2018, 10:31:57 PM »
Some really good info in this thread, I'd wondered what the files section of the bin files was for!

I've been working on (hopefully eventually) a map viewer for a bit of fun, but I'm also struggling with geometry. I've found enough code out there for version 0 .geo files, but I'm having trouble with later versions. GeoDraw can open all .geo files that I throw at it, but I can't find the source anywhere - I think it was made by CW and GuyPerfect so hopefully, they'll see this and possibly shed some light on those.

It would be really great to get some comprehensive documentation somewhere, people with reverse-engineering skills are few and far between, so unless we write this stuff down I worry it might die one day. I know there are certain legal issues around reverse-engineering, but I don't believe there are any legal issues with sharing RE'd info that you've learned without actually doing the RE. I think, anyway.

Anywho if anyone's interested in what I've got so far (just doing file parsing tests so far) GitHub

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #428 on: September 08, 2018, 07:10:30 PM »
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.

Let me see what I can dig up.

EDIT: I do remember seeing something about the SEGS folks having a mostly working map viewer, so they might have some useful info there. Though since they're using a really old client it may only have to deal with version 0 geos.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #429 on: September 08, 2018, 09:03:56 PM »
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.

Let me see what I can dig up.

EDIT: I do remember seeing something about the SEGS folks having a mostly working map viewer, so they might have some useful info there. Though since they're using a really old client it may only have to deal with version 0 geos.
That'd be awesome if you could - I've been using segs as my main source of information so far, but I found for pigg/bin loading that having multiple sources to draw on helped fill in the gaps missed by one or the other. Oh, and as you say, segs expects version 0 I think, it certainly doesn't handle the version 7/8 I threw at it.

There are other files I have yet to look at:
.bounds - well, I had a brief look and figured out the min/max/radius fields, not sure what any of the others mean.
.texture - segs should help me here, but I'm not sure if there's any versiony business to watch out for
.anim - has anyone reversed these yet?

I plan to write up some file format docs in the github repo at some point

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #430 on: September 10, 2018, 06:27:29 PM »
Hey, Codewalker - I've got a question about the schema.xml files:

When there's something like this:
Code: [Select]
    <entry name="ObjFlags" type="FLAGS" enum="static">
        <enumval value="0x00000001" name="ForceAlphaSort" />
        <enumval value="0x00000400" name="ForceOpaque" />
        <enumval value="0x00000100" name="TreeDraw" />
        <enumval value="0x00000008" name="SunFlare" />
        <enumval value="0x00001000" name="WorldFx" />
        <enumval value="0x00008000" name="StaticFx" />
        <enumval value="0x00000004" name="FullBright" />
        <enumval value="0x00000010" name="NoLightAngle" />
        <enumval value="0x00000002" name="FancyWater" />
        <enumval value="0x00080000" name="DontCastShadowMap" />
        <enumval value="0x00100000" name="DontReceiveShadowMap" />
    </entry>

An enumeration, to me, says that the field in question holds exactly one of those values. The enumvals are just names for various potential values. So, ObjFlags, for instance, can be  set to StaticFX or FullBright, but not both.

However, I can't help noticing that all of these enumerations are bit-wise values and the type of ObjFlags is "FLAGS", plural. Is the above paragraph correct, or is the schema actually using "enum" to describe all of the possible flags that could be OR'ed together to make the composite value of ObjFlags (or whatever enum variable is being described)?


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #431 on: September 10, 2018, 07:54:45 PM »
If the type of the field is FLAGS is generally indicates a bitmapped flags field, so you can probably OR them together.

The same type of internal structure is used for the parser to recognize text names of both enums and flags, that's why it gets called <enumval/> in the schema dump.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #432 on: September 10, 2018, 08:08:32 PM »
Groovy. I know it was kinda obvious but I hadn't really paid attention in the past to how some enums had type "INT" and some had type "FLAGS" so I figured better to clarify than assume the "obvious" thing.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #433 on: September 10, 2018, 10:09:25 PM »
Some of the fields aren't named in the schema, but I've had some success in filling in the blanks by dumping symbols from a PDB and finding the struct/class definitions. Some fields are missing due to the symbols being outdated but gets a bit closer.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #434 on: September 11, 2018, 03:40:42 PM »
.anim - has anyone reversed these yet?

I'm not sure how much it helps, given that it was released back around I-9 or I-10ish timeframe, but the Cryptic Animation Rig is still available with a bit of Wayback Machine-fu. A bit more searching turned up some efforts by a guy in the past to reproduce it in Blender, though I didn't manage to find any downloadable data files at the time I was looking for them.

If nothing else, just being able to see the rig makes it easier to conceptualize how the costume bits fit together. I don't think it necessarily helps with the .geo or .anim files directly, in that they must have had some custom import plugins to 3DS Max to handle their custom data files, but maybe the data points in the rig can help identify data points in the .geo/.anim files.

In fact, here, I'll save you the searching of the Wayback - Cryptic was giving this stuff away before they let the domain lapse so I don't see any harm from sharing what they already shared: https://www.dropbox.com/sh/h3h3fjxhaces2n2/AACfmQbJHK-h5G2al-hXwzD_a?dl=0

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #435 on: September 11, 2018, 09:27:03 PM »
Nice one, thanks :)

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #436 on: September 11, 2018, 09:30:44 PM »
anim files are simultaneously really simple and deceptively complex.

The format itself is very barebones (no pun intended), just some basic headers. Each bone in the rig has a flat list of rotations and positions for each frame (30 frames in 1 second).

Where it gets complicated is that the animation data is compressed using one of several different methods. *Most* of the rotation data is compressed down to 5-byte packed quaternions, where 3 of the components are stored as 12-bit fractions that map to the range [-1/sqrt(2), 1/sqrt(2)]. The largest component is omitted and reconstructed from the other 3 when loading, which gets a little mathy.

If that weren't enough, rotations can also be stored as 8-byte quaternions where each component is a 16-bit fraction, or simply uncompressed as 32-bit floats. There's also an alternate 5-byte format that doesn't use a linear mapping but instead represents the components as the output of a sine function, but while the code can read it, I haven't encountered any anim files that actually use that one.

Similarly, position data can either be stored as floats, or with a 16-bit half precision variant that is used if the data falls into the (-1,1) range.

Since anims aren't really needed serverside except for the skeleton hierarchy and to get animation lengths in some cases, I haven't done much code work on them. If you're looking at client modding / expansion, I'd recommend getting a solid handle on geos first and some usable tools for them written before going down the anim rabbit hole.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #437 on: September 11, 2018, 09:37:44 PM »
anim files are simultaneously really simple and deceptively complex.

The format itself is very barebones (no pun intended), just some basic headers. Each bone in the rig has a flat list of rotations and positions for each frame (30 frames in 1 second).

Where it gets complicated is that the animation data is compressed using one of several different methods. *Most* of the rotation data is compressed down to 5-byte packed quaternions, where 3 of the components are stored as 12-bit fractions that map to the range [-1/sqrt(2), 1/sqrt(2)]. The largest component is omitted and reconstructed from the other 3 when loading, which gets a little mathy.
Oh cool, smallest 3 compression - we use that in netcode often for sending rotations over the network.

If that weren't enough, rotations can also be stored as 8-byte quaternions where each component is a 16-bit fraction, or simply uncompressed as 32-bit floats. There's also an alternate 5-byte format that doesn't use a linear mapping but instead represents the components as the output of a sine function, but while the code can read it, I haven't encountered any anim files that actually use that one.

Similarly, position data can either be stored as floats, or with a 16-bit half precision variant that is used if the data falls into the (-1,1) range.

Since anims aren't really needed serverside except for the skeleton hierarchy and to get animation lengths in some cases, I haven't done much code work on them. If you're looking at client modding / expansion, I'd recommend getting a solid handle on geos first and some usable tools for them written before going down the anim rabbit hole.
Yeah I'm doing clienty stuff, as I said before, starting with a map viewer, and if I ever finish that it'd be cool to spawn some characters in there. As you say though, start with static stuff first and worry about anims if I actually make that far!

Thanks for the info :)

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #438 on: September 13, 2018, 04:04:33 PM »
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.

Let me see what I can dig up.

Just wanted to add my vote here as interested in seeing the geodraw source code if it's available. Trust me, I'm not judging anyone on how "bad" their code is. Once I made the decision that Rover has to load a scene graph and act like a client, I've come more and more around to the idea that I originally had when I started this whole thing, which is that what I should be writing is a bot controller that can manage several bots rather than launching one process per bot.

In short; if I have to get intimately involved with the scene graph and the geometry, then what makes the most sense is a central "Director" that knows the geometry of a zone and tracks all of the moving parts, and that spawns individual "Actors" that are capable of following their individual scripts but can query the "Director" for world-related information when they need it, so that only the "Director" has to carry the whole zone around in its head. That ends up sounding a lot like a map server instead of like a "bot". If I'm going that far, then I might as well get down to the actual geometry of the world or at least have the option to do that.

I've added Panda3D to the list of 3D engines to get educated about. I'm starting to think it's the tool I need for this: It has the scenegraph support, it has native collision detection and it also supports Bullet and ODE, it's a professional-level tool (though mostly supplanted by Unity in recent years), it actually has decent documentation, and it's written in C++ but primarily aimed at python so the python bindings are up-to-date, maintained and documented.

« Last Edit: September 13, 2018, 06:32:31 PM by slickriptide »

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #439 on: September 13, 2018, 09:26:53 PM »
Yeah, we've all written messy code - you can find some of my embarrassments on GitHub :D