Author Topic: Technical side discussion  (Read 164695 times)

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #440 on: September 14, 2018, 03:42:21 PM »
Btw slickriptide, what is it from the geos that you actually need? If you just need meshes I think I'm close to having that much figured out, it's just a lot of the other data in the files that I'm not sure about.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #441 on: September 14, 2018, 09:12:24 PM »
Btw slickriptide, what is it from the geos that you actually need? If you just need meshes I think I'm close to having that much figured out, it's just a lot of the other data in the files that I'm not sure about.

For the moment, I'm still figuring out the scene graph deserialization, so I don't need geo data yet, and I'll probably fake it at first by just using boxes based on the bounds in the .bounds files. Assuming that all that works and Boxtown becomes a reality, then I might see about incorporating actual geometry for collision purposes. I THINK I'd just need mesh data for that, since a bot wouldn't care what the texture looked like. OTOH, I seem to be heading down the road of using Panda3d to handle my scenegraph, and that means that once I had the scenegraph fully loaded and ready to use that I'd be halfway to having a map viewer, so THAT might be a good reason to be able to load textures and geometry together.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #442 on: September 14, 2018, 10:33:14 PM »
Speaking of scenegraphs, I think I'm almost ready to have a go at building a scenegraph in Panda3D. The initial levels, anyway.

The Kaitai parser is working fine. It's pretty cool, really, to write a few lines of code and have an entire geobin file load up, ready to be used.

My first little geobin parser looks like this:

Code: [Select]
from geobin import *
import psutil
import gc

print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
print("Loading geobin...")
gb = Geobin.from_file("C:/Users/Scott/Downloads/tequila/Extracted_Full/geobin/maps/City_Zones/City_01_01/City_01_01.bin")
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))


defs = gb.bin_data.def_array
num_defs = gb.bin_data.def_count
refs = gb.bin_data.ref_array
num_refs = gb.bin_data.ref_count

print("Num Defs: %d / Num Refs: %d" % (num_defs, num_refs))
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))

print("Listing Refs")

for ref_index in range(0, num_refs, 1):
    rb = refs[ref_index].ref_body
    ref_name = rb.ref_str1.strval
    ref_pos = (rb.ref_pos_x,rb.ref_pos_y,rb.ref_pos_z)
    ref_o = (rb.ref_pyr_yaw, rb.ref_pyr_pitch, rb.ref_pyr_roll)
    print("Refname:%s -- Pos%s Hpr%s" % (ref_name, ref_pos, ref_o))

print("Freeing geobin...")

del defs
del num_defs
del refs
del num_refs
del gb

print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))

and produced this:

Code: [Select]
PS C:\errbot.bogart\kaitai> python gb_test.py
Mem footprint: 16.636719mb
Loading geobin...
Mem footprint: 52.238281mb
Num Defs: 7239 / Num Refs: 33
Mem footprint: 52.238281mb
Listing Refs
Refname:grp_GC_Shutdown_Atmo -- Pos(-71.0, 15.931458473205566, -474.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_audiogroup -- Pos(186.380859375, 67.989501953125, -938.831298828125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Badge_tourism -- Pos(721.3763427734375, 164.83200073242188, -699.31494140625) Hpr(0.07853981852531433, 0.0, -0.0)
Refname:grp_beacons -- Pos(522.87060546875, 262.0234375, -574.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_blackmap -- Pos(512.0, -128.0000457763672, -763.474609375) Hpr(0.0, 0.0, 0.0)
Refname:grp_blimp -- Pos(176.0, 707.0, -1040.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Day_Job_Volumes -- Pos(-59.5, -867.0, -914.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Geometry -- Pos(511.38037109375, 569.0, -703.999755859375) Hpr(0.0, 0.0, 0.0)
Refname:grp_Halloween_Event_06 -- Pos(-68.01048278808594, 4.472891807556152, -10.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_Holiday_Event_06 -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HollowsBeacons -- Pos(189.818359375, 245.71826171875, -2069.03076171875) Hpr(0.0, 0.0, 0.0)
Refname:grp_Makeover_Spawndefs -- Pos(642.7833251953125, 7.882775783538818, -797.4718017578125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Manholes -- Pos(1235.75, 156.25277709960938, -242.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_mission_doors -- Pos(120.030517578125, 465.505859375, -748.54931640625) Hpr(0.0, 0.0, 0.0)
Refname:grp_neighborhoods -- Pos(-32.0, 0.0, -351.9999694824219) Hpr(0.0, 0.0, 0.0)
Refname:grp_patrol_box -- Pos(-5.161986827850342, 18.90635108947754, -706.25) Hpr(0.0, 0.0, 0.0)
Refname:grp_PersistentNPC -- Pos(-25.9998779296875, -159.36572265625, -899.901611328125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Plaques -- Pos(206.2913818359375, 24.575931549072266, -414.63629150390625) Hpr(0.0, 0.0, 0.0)
Refname:grp_PVP_Scripts -- Pos(140.43020629882812, 20.40468406677246, -241.53944396972656) Hpr(0.0, 0.0, 0.0)
Refname:grp_Rikti_Invasion_Event -- Pos(2260.0, 0.0, 3388.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_RiktiBetaEvent -- Pos(-83.82254028320312, 841.516357421875, -643.6749877929688) Hpr(0.0, 0.0, 0.0)
Refname:grp_ShadowShardInvasion -- Pos(511.4013671875, 409.1253356933594, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Truck_Door -- Pos(340.6702880859375, 135.29078674316406, -1199.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_warpvolumes -- Pos(-1664.0, -896.0, 1408.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Winter_Event -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HalloweenEvent_09 -- Pos(1199.0, 41.59211349487305, -1189.7000732421875) Hpr(0.0, 0.0, 0.0)
Refname:grp_ParagonCityTeleport -- Pos(-33.21428680419922, 0.0, -171.35714721679688) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Phase -- Pos(637.5, -0.4998469948768616, 67.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Trays -- Pos(2450.0, 56.017494201660156, -1372.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_spawndefs -- Pos(772.814208984375, -16.52880859375, -699.10009765625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Zowies -- Pos(2442.5, 57.2479248046875, -1374.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_GalCty_PerstntNPC -- Pos(391.30340576171875, 68.48408508300781, -864.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Kraken_Event -- Pos(1354.25, -32.00006103515625, -736.5) Hpr(0.0, 0.0, 0.0)
Freeing geobin...
Mem footprint: 52.238281mb
PS C:\errbot.bogart\kaitai>

My concern at the moment is this bit: "Freeing geobin...Mem footprint: 52.238281mb". Freeing the geobin parser didn't actually free the memory it used, probably because the underlying Kaitai structures are still referencing the data. There's not a lot I can do about that, unless I want to roll my own parser and handle the memory management myself, which I don't. By itself, 52mb isn't any big deal. I'm more concerned with what happens when I start going into the object library and opening up dozens of geobins associated with object library pieces. I guess I'll cross that bridge when I come to it. Most of the low level Kaitai stuff is undocumented, and there may be some method it has to free up its memory that I just don't currently know about.

That aside, it's handy that the Kaitai parser loads up the entire geobin file at once. I've pretty much verified that the "right" way to deserialize the scenegraph is to do it in reverse, with respect to the way the data is stored in the geobin file. I could build it up leaf-to-branch-to-root but it should be easier and less error-prone to just start at the end of the refs array and the defs array and move backwards through them, building the tree naturally from the root to the leaves.

The refs are the first-level children of the root. Everything else depends from there. Bindump confirms that moving backwards from array-end to array-start goes like this:

Code: [Select]
    -------- Defs 6055 --------
    Name = grp_worldlink_beacons
    Group =
        -------- Group 0 --------
        Name = grp6676
        Pos =
            X = -1283.57
            Y = -31.8641
            Z = 311.098
        PYR =
            Pitch = 0
            Yaw = 0
            Roll = 0
        Flags = 0
        -------- Group 1 --------
        Name = grp1745
        Pos =
            X = -309.435
            Y = -33.1197
            Z = 1652.77
        PYR =
            Pitch = 0
            Yaw = 0
            Roll = 0
        Flags = 0
    Property =
        -------- Property 0 --------
        u034 = Layer
        u035 = worldlink_beacons
        u036 = 0
    TintColor = []
    Ambient = []
    Omni = []
    Cubemap = []
    Volume = []
    Sound = []
    ReplaceTex = []
    Beacon = []
    Fog = []
    Lod = []
    Type =
    Flags = 0
    Alpha = 0
    Obj =
    TexSwap = []
    SoundScript =


"Name" is the name of the parent node. The Group array represents the children to add to that parent. The other arrays likewise are attributes that either apply directly to the parent or that are attached as children to the parent.

Here's our bindump excerpt for the next two defs, moving backwards:
Code: [Select]
    -------- Defs 6053 --------
    Name = grp1744
    Group =
        -------- Group 0 --------
        Name = Omni/MissionBeacons/_PlayerSpawn
        Pos =
            X = 0
            Y = 0
            Z = 0
        PYR =
            Pitch = 0
            Yaw = -3.14159
            Roll = 0
        Flags = 0
    Property =
        -------- Property 0 --------
        u034 = SpawnLocation
        u035 = LinkFrom_City_01_02
        u036 = 0
    TintColor = []
    Ambient = []
    Omni = []
    Cubemap = []
    Volume = []
    Sound = []
    ReplaceTex = []
    Beacon = []
    Fog = []
    Lod = []
    Type =
    Flags = Ungroupable (1)
    Alpha = 0
    Obj =
    TexSwap = []
    SoundScript =
    -------- Defs 6054 --------
    Name = grp1745
    Group =
        -------- Group 0 --------
        Name = grp1384
        Pos =
            X = 43.5209
            Y = -421.523
            Z = -24.6548
        PYR =
            Pitch = 0
            Yaw = -2.79253
            Roll = 0
        Flags = 0
        -------- Group 1 --------
        Name = grp1388
        Pos =
            X = 43.0209
            Y = -421.523
            Z = -9.65479
        PYR =
            Pitch = 0
            Yaw = -3.14159
            Roll = 0
        Flags = 0
        -------- Group 2 --------
        Name = grp1403
        Pos =
            X = -52.4791
            Y = -421.523
            Z = -9.15479
        PYR =
            Pitch = 0
            Yaw = 2.96706
            Roll = 0
        Flags = 0
        -------- Group 3 --------
        Name = grp1617
        Pos =
            X = -38.9791
            Y = -421.523
            Z = -25.1548
        PYR =
            Pitch = 0
            Yaw = 2.61799
            Roll = 0
        Flags = 0
        -------- Group 4 --------
        Name = grp1618
        Pos =
            X = -39.9791
            Y = -421.523
            Z = -9.65479
        PYR =
            Pitch = 0
            Yaw = -3.14159
            Roll = 0
        Flags = 0
        -------- Group 5 --------
        Name = grp1619
        Pos =
            X = -4.47913
            Y = -422.023
            Z = -1.15479
        PYR =
            Pitch = 0
            Yaw = -3.14159
            Roll = 0
        Flags = 0
        -------- Group 6 --------
        Name = grp1744
        Pos =
            X = 43.0209
            Y = -421.523
            Z = -33.1548
        PYR =
            Pitch = 0
            Yaw = -2.61799
            Roll = 0
        Flags = 0
    Property = []
    TintColor = []
    Ambient = []
    Omni = []
    Cubemap = []
    Volume = []
    Sound = []
    ReplaceTex = []
    Beacon = []
    Fog = []
    Lod = []
    Type =
    Flags = 0
    Alpha = 0
    Obj =
    TexSwap = []
    SoundScript =

Sure enough - Def 6054 is the last child of Def 6055, and Def 6053 is the last child of Def 6054. The pattern repeats until we hit a leaf; typically an object library piece. Then the "de-traversal" moves back up the tree and follows the next branch to its end and so on.

So, as long as I've got a handle on creating appropriate Panda3D NodePaths and attaching them together correctly, the scenegraph pretty much writes itself, at least up to the point where the object library and the actual geometry come into play.

The object library geobins are technically identical in format to the map geobins, except that they don't have refs, and they have a corresponding .bounds file that needs to be attached to the scene as part of the leaf data. The one unusual thing is that "leaf" nodes in an object library have TWO terminating records. One is a record with the object name and an Object attribute as a child. The last is an empty node with the object's name as the parent. Presumably, the former is meant to indicate that it contains geometry and the latter is intended to have the geometry attached to it. At least, that's my best guess at the moment.

Despite what I said about the scenegraph "writing itself", there are a lot of attributes that I'm not sure what to do with or that I'm considering disposing of because I DON'T care about them. Sounds, or Level of Detail (LOD) nodes, for instance. The real game client cares, of course, but a bot doesn't care; at least not in any way I'm currently envisioning. If I end up having to define a bunch of custom nodes to handle that stuff, or the memory footprint becomes unwieldy, then it might be easier to dispose of it. I guess, like other things, that I cross that bridge when I come to it. I'm trying to keep in mind that if I do this thing "right", that some other person may find a use for these details even if they aren't particularly useful to me at the present time.



« Last Edit: September 14, 2018, 10:41:54 PM by slickriptide »

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #443 on: September 16, 2018, 05:45:21 PM »
While "writing itself" was a bit optimistic, I'm getting the hang of the Panda3d scenegraph.



I need to add instancing, so that multiple copies of a node are indirect references to a single node rather than a hundred individual copies of that node. Then I need to work on getting down into the object library.

That's the bare minimum. Then I have to decide what to do with attributes like Properties and Omni and whatever. It turns out it's not as simple as just sub-classing the basic node because of the whole "Python on top of C++" thing). The simplest solution might simply be to program in C++ instead of Python but we'll see how it goes as I go along here.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #444 on: September 18, 2018, 12:06:09 AM »
I've written up some of what I've found out or figured about the geo file format, vertex/triangle/normal/texcoord data for all geo versions present in I24. I haven't figured out how the textures are applied yet, but if/when you get to the point of needing to read geos it might save you some time.

https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #445 on: September 18, 2018, 03:18:06 PM »
If you're serious about doing correct collision, you'll probably need to load texture info at some point. Texture names are effectively material names once you tie them to tricks. Aside from editor-only stuff that can probably be excluded at the object level, there are some objects that have faces which are visible but do not block collision. Spider webs and other environmental details come to mind.

Here's the geodraw code. It's ugly, but the logic is almost exactly what the game itself does to load geos (we disassembled the geo loader and single stepped through it).

https://github.com/cwtitan/geodraw

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #446 on: September 18, 2018, 04:20:32 PM »
https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md

Well done, Spectre! You must have been a tech writer in a previous life. That's some of the most complete and readable docs I've read in a while.

https://github.com/cwtitan/geodraw

Thanks, Codewalker! This will help a lot as far as understanding what's happening under the hood.

We'll see about correct collision, but it's certainly something I'd like to succeed at implementing. Whether it works or whether the bots ultimately live in Boxtown will depend a lot on whether I can get a grasp on how to convert a geo model into something that a non-Cryptic engine can use. Panda, in this case, but it could be Ogre or Godot or Openscenegraph or one of the dozens of others out there. I keep hoping that one of the ex-Paragon modelers would "accidentally" let their 3DS import/export plugins escape into the wild.



Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #447 on: September 18, 2018, 04:34:24 PM »
spectre1989: Your "unknown section" that exists from version 2-6 is LOD (level of detail) info. Version 2-5 include one of these for each model in the file, in the same order as the models themselves:

Code: [Select]
int32 num; // (can be 6 at most)
{
    f32 allowedError;
    f32 near;
    f32 far;
    f32 nearfade;
    f32 farfade;
    uint32 flags;
} // repeated 6 times

Version 6 is similar, but instead of having a static array of 6 structures, it only repeats "num" times, and after each flags field there are two null-terminated strings: model name and file name to substitute at that distance.

Version 7 and newer geos omit this information. Some of the info was moved to lods.bin instead, indexed by model name. This version and higher can do automatic mesh reduction instead of substituting other models -- the information for this is a little complex to describe here but is one of the later sections in the model header.

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #448 on: September 18, 2018, 09:45:16 PM »
Here's the geodraw code. It's ugly, but the logic is almost exactly what the game itself does to load geos (we disassembled the geo loader and single stepped through it).

https://github.com/cwtitan/geodraw
Thanks very much for sharing that :D

Well done, Spectre! You must have been a tech writer in a previous life. That's some of the most complete and readable docs I've read in a while.
Thanks mate :)

spectre1989: Your "unknown section" that exists from version 2-6 is LOD (level of detail) info. Version 2-5 include one of these for each model in the file, in the same order as the models themselves:

Code: [Select]
int32 num; // (can be 6 at most)
{
    f32 allowedError;
    f32 near;
    f32 far;
    f32 nearfade;
    f32 farfade;
    uint32 flags;
} // repeated 6 times

Version 6 is similar, but instead of having a static array of 6 structures, it only repeats "num" times, and after each flags field there are two null-terminated strings: model name and file name to substitute at that distance.

Version 7 and newer geos omit this information. Some of the info was moved to lods.bin instead, indexed by model name. This version and higher can do automatic mesh reduction instead of substituting other models -- the information for this is a little complex to describe here but is one of the later sections in the model header.
Oh awesome - some questions:
* do you happen to know how nearfade/farfade work off hand? Saves some brain ache if you do, no worries if not
* do you know what the various flag values mean?
* are there any version 6 geos in I24? I couldn't find any myself, unless there are files which use this same file structure which don't use .geo as the extension
* while we're here, there's something I've always wanted to know about piggs - what's the table of compressed blobs of data which comes after the strings table? I've heard it referred to as "meta"/"headers", but no idea what it's actually for?

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #449 on: September 18, 2018, 11:58:22 PM »
Fun factoid about geobin files - you have to deserialize them in the order they're written, leaves-first. Otherwise, you decode a lot of instances before you decode the thing they're instancing from.

I discovered this by doing the exact opposite, of course. Back to the drawing board.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #450 on: September 19, 2018, 02:20:19 AM »
Fun factoid about geobin files - you have to deserialize them in the order they're written, leaves-first. Otherwise, you decode a lot of instances before you decode the thing they're instancing from.

That's correct. The proper way to read a geobin is to build up a library of Defs by reading them and creating an in-memory Def structure for each one as you go. By doing them in order, it's guaranteed that any child defs of the group are already in memory -- child defs should just be a pointer to one that's already been loaded.

Then Refs, which are the roots of the scene graph, are simply pointers to Defs that have already been loaded.

The object library is a bunch of Defs that are assumed to always be available (preloaded) regardless of which geobin you're reading. Technically they're demand loaded when first referenced, but there's an index so the names are effectively already defined, and they can hang around in the cache when changing maps. There's nothing preventing someone from making a map with a Ref directly to something out of the object library that isn't even defined in the geobin.

Refs are also special in that top-level refs are part of the scene packet and explicitly sent by the server to the client. There is a protocol for tracking the visibility of top-level Refs -- this is used for things like making the Ski Chalet visible, or for making the War Walls invisible during an invasion.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #451 on: September 19, 2018, 02:51:37 AM »
Oh awesome - some questions:
* do you happen to know how nearfade/farfade work off hand? Saves some brain ache if you do, no worries if not

Those are easy, it's just the distance ("inside", relative to near/far) that the object starts to fade out.

So if near=50, nearfade=25, far=200, farfade=10,
the object would be invisible if you're closer than 50', would gradually fade to full opacity between 50' and 75', then at 190' would start fading out, and invisible again beyond 200'.

Most full-quality 'master' models don't use near or nearfade, but lower-quality models use those so that they are hidden until you get to a certain distance, which is the old way of doing things.

Objects with AutoLOD (most newer models) just specify far/farfade as the mesh reduction info handles the distances for deleting triangles.

* do you know what the various flag values mean?

According to the textparser table there's only two:
ErrorTriCount: 0x02
UseFallbackMaterial: 0x80

I have no idea what ErrorTriCount does. UseFallbackMaterial I would *guess* from the name might force the use of the fallback textures for the non-multi9 shader (used for graphics cards that can't handle ultra mode features), but I haven't tested that.

* are there any version 6 geos in I24? I couldn't find any myself, unless there are files which use this same file structure which don't use .geo as the extension

Honestly I'm not sure. That info is based on how a version 6 geo would be loaded by the game, but I haven't run a full sweep to see if any of those actually exist. All of the assets created at that point in time might have been re-exported later for some reason, wiping them out.

* while we're here, there's something I've always wanted to know about piggs - what's the table of compressed blobs of data which comes after the strings table? I've heard it referred to as "meta"/"headers", but no idea what it's actually for?

My old C++ libpigg called it "extra", but "headers" is probably the best term.

It's an optimization trick. Certain types of files, specifically geo, texture, and anim, all start with a 4-byte header size field. When pigg files are generated, those types of files have the header extracted and saved uncompressed into a special section of the pigg.

That section is loaded as a single chunk with the pigg file and always kept resident in memory. When the pigg library handles a read request for one of those files, if the offset+size is entirely within the cached header, it can be served without ever having to perform disk I/O.

This mostly affects game startup, when all of the in-memory indexes are built and every texture and geo header is scanned.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #452 on: September 19, 2018, 05:14:03 PM »
***edit***
See below
*********
« Last Edit: September 25, 2018, 05:26:25 PM by slickriptide »

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #453 on: September 19, 2018, 07:35:57 PM »
That's correct. The proper way to read a geobin is to build up a library of Defs by reading them and creating an in-memory Def structure for each one as you go. By doing them in order, it's guaranteed that any child defs of the group are already in memory -- child defs should just be a pointer to one that's already been loaded.

Well, that worked for Galaxy City. Generalizing from a single example, I'm sure it will work for every other geobin. ;-)

While I need to work on loading the object library before I make any pronouncements, it was also pretty quick to load and not nearly as memory intensive as I was afraid it would be. Of course, it's not holding any actual geometry at the moment, so it's too early to decide that "hey, this ain't so bad after all".

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #454 on: September 19, 2018, 08:15:52 PM »
It's an optimization trick. Certain types of files, specifically geo, texture, and anim, all start with a 4-byte header size field. When pigg files are generated, those types of files have the header extracted and saved uncompressed into a special section of the pigg.

That section is loaded as a single chunk with the pigg file and always kept resident in memory. When the pigg library handles a read request for one of those files, if the offset+size is entirely within the cached header, it can be served without ever having to perform disk I/O.

This mostly affects game startup, when all of the in-memory indexes are built and every texture and geo header is scanned.
Ah right! Yeah, just looked at the meta for a file along with the actual file in a hex editor, makes sense now. Thanks! :)

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #455 on: September 19, 2018, 09:57:06 PM »
Hmm whats allowedError for then, if the fading takes care of pop-in?

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #456 on: September 23, 2018, 10:41:22 PM »
***edit***
I'm going to make this simple.

Codewalker, Back in the thread Technical Discussion of Bases from 2016, you said this:
Quote
When I say things like "the server just sends a map filename for the client to load", that's actually a gross simplification to keep conversations manageable. What actually happens is that the server first sends a list of files to load defs from - first all of the various object library files it depends on*, then the map itself. Then it sends the list of top-level refs that point to some of the defs just loaded.

What I need to know - How is the Paragon Chat "server" building the list of object library bin files that it sends to the client? I'm asking because processing the map geobin is leading me to all kinds of loose ends that APPEAR to be irreconcilable with either the virtual filesystem in the .piggs or the index in defnames.bin. It looks like an object path in the map geobin is a reference to a scene graph branch rather than some kind of absolute file path. How are you finding the absolute file paths to load from?
*********
« Last Edit: September 25, 2018, 05:38:31 PM by slickriptide »

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #457 on: September 27, 2018, 05:46:49 PM »
Well, while we wait for Codewalker to return from whatever dimension he's currently exploring...

A small victory - Here's the shuttle model in a Panda3d window:


I don't want to oversell it as an "accomplishment". I pretty much just turned geodraw into an exporter that creates Panda .EGG files from CoH .GEO models. What I really need to do is create an API that can skip the intermediate step and load into an in-memory Panda model node straight from the .GEO file. I figure that first I'll see if I can get tricks.bin and the model textures figured out, then go for the "generating geometry on the fly part".

Though, as I've observed in the past, there's never going to be an Issue 24+ so there may be something to be said for just exporting ALL of the models and ditching the .PIGG and .GEO files.

***edit***

So, now it's textured, but I'm doing this all by hand right now. What I still don't see is how "x_H_Shuttle" in the .GEO file turns into "/texture_library/World/Space/Shuttle/H_Shuttle.dds". I expect this goes along with all of the defnames.bin entries that are numbers that don't have a corresponding path they index to. There's some part of the index that is either "missing" or that needs to be built up by the client.
« Last Edit: September 27, 2018, 08:47:29 PM by slickriptide »

spectre1989

  • Minion
  • **
  • Posts: 38
Re: Technical side discussion
« Reply #458 on: September 27, 2018, 09:38:54 PM »
Great work dude!

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #459 on: September 28, 2018, 03:42:35 PM »
Great work dude!

Thanks. It's a way's away still, but I'm starting to think that approximating the city geometry for a collision/pathfinding model is a doable thing.

Back to questions for Codewalker - I've been using TonyV's Pigg Viewer Pro as my pigg viewer/extractor. It's pretty much the best of the bunch and the source code is still available in the Google Code archive, which means I can crib from his code if I ever decide to the "right" thing and implement a pigg-based virtual file system. (Panda3D has its own concept of virtual file system, so there may be some mileage in writing a glue lib that lets Panda "mount" the piggs onto its own VFS.)

PVPro tells me that all of those textures in the texture library are stored in the piggs as .texture files. However, all of the references in the databases refer to Targa .tga files and that's the one option that PVPro does NOT export them as. PVPro lists "original" as .dds. It's easy enough to work around that, but I want to make sure that I'm not missing something or losing some functionality by dealing with the textures as .dds or some other format compared to whatever raw format a .texture file is. Of course, it would make some sense if the Textparser turned the original .tga files into .dds files in the process of compiling them into the .pigg files.