-
Well, as long as we're posting speculation, this quote by Leandro "You can make the COH client connect directly to a dbserver, skipping the authserver entirely, and have full functionality." on this reddit page: https://www.reddit.com/r/Cityofheroes/comments/39fpnj/coh_server_emulator_project/ makes me think of combat functionality in a single player mode, assuming that the dbserver does what I think it does, i.e., holds data related to power numbers, enemy spawn placement, NPC health, and such. I could be completely wrong, I'm just speculating here :P
-
The dbserver does none of that. Its most important function is to handle loading and saving characters from a database, presenting them to the client to show the character list at login, and transferring them between maps. Powers, spawn points, etc. in COH are not part of the database.
The dbserver would also normally handle cross-map chat, but in this project that functionality is not needed for what are hopefully obvious reasons.
-
I see. My mistake, then. What DOES handle the number side of things, then, if you don't mind my asking? Also, I had a question about the way enemy spawns worked. In Icon, I can open the dev/debug mode and see all the markers, and see what I think are enemy spawn points marked as E1, E2, etc. However, and it may just be poor memory on my part, but I don't recall street spawns being in clumps of 8 or so. Are the E# markers just potential spawns, and the game would randomly pick which ones to use in any given spawn group?
-
That's all part of the mapserver, which is responsible for player input, chat, movement, physics, AI, character stats, powers, emotes, animation, network communication, combat mechanics, missions, story arcs, rewards tables, NPC pathing, auctions, inventory management, the kitchen sink, and probably about 200 more things that I'm forgetting to mention.
The powers data itself is in structured binary files, the format of which has been long understood and are what powers things like City of Data and Mids.
-
Interesting, quite interesting. Not a programmer, simply very curious about all sorts of things. Though I did post an additional question question as an edit before seeing your response, sorry.
-
Those markers are potential spawn points. Each of those clusters is an entry from the "object library" (shared components that can be pasted into a map) which contains a bunch of spawn points that get dropped into place on a map. Based on the names of the object library entries, "encounter group" is the right terminology for a cluster of spawn points.
The maps themselves contain references to spawndef files alongside those encounter groups, which presumably tell the server which of those spawn points to use (depending on team size and difficulty), and what to spawn there. The spawndefs were server-side only and do not exist in the client data.
The groups seem to be overlapped quite often, so it's likely that not all of the groups get spawned, but the server decides according to some algorithm which ones it's going to use.
-
I see. Thank you for assuaging my curiosity.
-
Could you build these as stand-alone "Hive" servers?
I am curious as DayZ servers often have persistant hives. You can have a different character server-toserver but your bank and other benefits are usuable on other servers.
I am just thinking if you could hive servers to provide different functions. One server is the chat and other functions - one is combat - one is the market and so on. I am curious due to the benefits this would have for development.
-
Could you build these as stand-alone "Hive" servers?
I am curious as DayZ servers often have persistant hives. You can have a different character server-toserver but your bank and other benefits are usuable on other servers.
I am just thinking if you could hive servers to provide different functions. One server is the chat and other functions - one is combat - one is the market and so on. I am curious due to the benefits this would have for development.
The short answer is obviously: yes.
But your question has some implied facets to it that are rather complex. For one thing, at the moment there's no such thing as server persistence, because as described the state of the system is likely being stored locally. So currently, the only persistence that can exist is what each individual player's client remembers (like the state of character slots, for example, which they seem to have hacked into the client-side software).
More generally, XMPP servers can be clustered with known techniques, and most seem to have the ability to have modular components added to them to add functionality. Stating for the record that this is as much a developer-preference thing as anything else, I believe that if the project continues along and Codewalker continues to add both functionality and scalability, the most likely architecture he'd pursue would be one where the primary communications hub was clustered if necessary (XMPP servers can handle a lot more connected clients than City of Heroes mapservers and whole game servers could ever handle, so clustering might not be necessary from a performance perspective), and a separate, modular system for adding enhanced functionality that could plug into the base XMPP server's message processing.
However, I have a feeling that most of the work Codewalker will be focused on in the short term is enabling client-side features that require limited or no central processing to make work, which means server-side concerns will probably be theoretical for a long while. It depends on how much functionality is already there, but knowing Codewalker I believe he's releasing this as soon as he believes he has sufficient functionality to be workable with minimal feature holes between the functionality. That almost certainly means there's tons more things he can do: he wouldn't wait to release until every last thing was working.
Put it this way: before we see "real" combat, I suspect we'll see "fake" combat first. From what I can deduce from what's being described, I'll bet Codewalker is (if he hasn't already, of course) dying to make powers work at least visually, and perhaps even physically (as in: using the local client-side physics engine). Its a lot easier, albeit not easy, for Codewalker to add functionality that would allow us to activate a power, like "powexec_name Power_Blast target_name" and have the software send that message to the XMPP server, so that both you and everyone else sees the animation, FX, and sound effect played for that power when its directed at "target_name." Nothing would happen, because there's no central server keeping track of health bars, so you can't defeat the target or even hurt it, but you could see a power "work" in that sense. You might even be able to trigger knockback, although there are security/griefing issues to resolve when allowing that player to player (player to NPC, on the other hand, seems at least plausible).
That could happen without a central server, and therefore without having to solve the problem of central mapservers or even a combat engine. But since Codewalker has already stated he doesn't have movement mode bits working yet for Fly, its probable this is a City of Jabber 2.0 or even 3.0 feature. But I know Codewalker can do it with the system he's not-describing, and its the logical outgrowth of the work he's done with Icon and the XMPP system, and its a very likely waypoint between here and eventual combat.
-
I just had a thought pop into my head, I know you guys are not ready to detail the full technical info on this project but I am curious if its possible to have hooks in place for character information beyond costumes like xp inf power choices etc. or even just awareness of what powersets were chosen at character creation? The reason I ask is because this could open up the game to new neat possibilities like a character api or some kind of app to connect and interact with the game world/characters outside the client.
-
So I can trigger X attack on target Y and get Z effect even though no damage is being scored. It would make role playing more visual as long as we had stagger/drunk/faint emotes lol
-
So I can trigger X attack on target Y and get Z effect even though no damage is being scored. It would make role playing more visual as long as we had stagger/drunk/faint emotes lol
Hypothetically. Codewalker is nowhere near that yet, or even close enough for anyone else to add directly yet I believe. Its theoretically within the limits of the technology based on the outer bounds of what he's accomplished so far. I just want to be clear my theoretical musings shouldn't get conflated with what Codewalker is prepared to release at any time.
-
Hypothetically. Codewalker is nowhere near that yet, or even close enough for anyone else to add directly yet I believe. Its theoretically within the limits of the technology based on the outer bounds of what he's accomplished so far.
Unless I'm missing the point, the client side of SOON is based on reverse engineering the client/server protocol sufficiently to send arbitrary information to the client.
Just the parts we know about look like a strong foundation for a fully functional community server at some point down the line.
Hmm... if either the source for SOON or the documentation for the API used by it were released at some point, the potential implications would be... interesting.
-
Hypothetically. Codewalker is nowhere near that yet, or even close enough for anyone else to add directly yet I believe. Its theoretically within the limits of the technology based on the outer bounds of what he's accomplished so far. I just want to be clear my theoretical musings shouldn't get conflated with what Codewalker is prepared to release at any time.
You see my question on the "Hive" server idea is that if say some interested people wanted to make an "approved" server to join the group then you have a distributed server group by geography and little to no server costs to any one entity. You could join the server that gave you the best ping/least lag.
I guess what I am thinking is you eliminate the ability to shutdown the game easily and also make the running of the game fairly cheap as I know you can rent a decent server for under $50 a month. Once you have console permissions and if you can make an approved Client install - this would seem to be an ideal way to proceed.
-
You see my question on the "Hive" server idea is that if say some interested people wanted to make an "approved" server to join the group then you have a distributed server group by geography and little to no server costs to any one entity. You could join the server that gave you the best ping/least lag.
I guess what I am thinking is you eliminate the ability to shutdown the game easily and also make the running of the game fairly cheap as I know you can rent a decent server for under $50 a month. Once you have console permissions and if you can make an approved Client install - this would seem to be an ideal way to proceed.
There would be technical hurdles to overcome, like interserver lag and bandwidth, in the unlimited generic case. But that might not be completely necessary because the game itself placed some limitations on us in the first place. We had to log into a specific server. And on that server, we had to stand in a specific zone. There's no reason why someone couldn't set up a "private server" that in effect was the mapserver for a private zone, sort of like a base or mission instance, that they could invite friends into, and if desired "zone out" to a public zone when they were willing and able to connect to a public server.
From what I've heard so far, it seems the actual "server" is a fairly generic XMPP server with a specific configuration, but not special code. Its the client-side element of this system that has the special software to translate client actions into XML chunks. They are probably exercising some level of control in the embryonic stages of this project which might prevent this sort of thing from happening easily, but I believe the hive-distributed architecture you're thinking about already implicitly exists in what they've already built. Its just a question of figuring out the best way to make those features work in a way that keeps the community together rather than promoting fragmentation, at least while its still spinning up.
-
That is just what I am thinking other than street sweeping everything is instanced - so if you let local servers handle the instances only requiring linking and updates as you zone back to the main server or in regularly scheduled pings to keep things sync'd. We have similar things here where I work to refresh IP addresses across the enterprise. It polls every few minutes to keep from duplicating addresses and if we change our connection across switches quickly - we occasionally need to run an ip config /renew so it updates.
I am not sure other than as with the DayZ servers they have to link to Steam - in this case to your version of Steam as it were. You will need some version of security handshaking to protect your central server from rogues. That is why I was saying the local servers would be required to be registered and updated.
-
I just had a thought pop into my head, I know you guys are not ready to detail the full technical info on this project but I am curious if its possible to have hooks in place for character information beyond costumes like xp inf power choices etc. or even just awareness of what powersets were chosen at character creation? The reason I ask is because this could open up the game to new neat possibilities like a character api or some kind of app to connect and interact with the game world/characters outside the client.
That would be pretty cool. Although until someone builds a web browser compatible costume creator, I'm not sure what use it would be before the game returned, since the only relevant info is the costume (as opposed to badges, level, power sets and such, which would be relevant to players of an actual game).
Although I suspect you could get a website that took the costume piece text strings and displayed pictures of them, to get an idea of what a character is wearing when you view it on a mobile device.
As for interacting with the "game world" outside the client, you can already do that; using a regular XMPP chat program, you log in like it's a chat room. Can't see anyone, but you can talk.
-
Simple question - Is the protocol documented someplace such that a bot could simulate a client when talking to other clients?
If it's not obvious, what I want is to make a chatbot that has an avatar in the game and the ability to "speak" or emote as if it was a player. In other words, a kind of XMPP-based NPC.
As I understand things, at the current time a "player" is a collection of costume bits and a set of map coordinates.
Is that about the size of things?
-
Yes, it should be possible for a bot to log into XMPP and pretend to be a player. The protocol doesn't have any formal documentation yet, but if someone wants to write some I can definitely help with that. Wiki article maybe? Just need somebody with an understanding of XML structures and technical writing.
-
Yes, it should be possible for a bot to log into XMPP and pretend to be a player. The protocol doesn't have any formal documentation yet, but if someone wants to write some I can definitely help with that. Wiki article maybe? Just need somebody with an understanding of XML structures and technical writing.
I'm probably about 30% there. If you have a schema or something, I could take a swing at it. I think have the basics of pc:presence and pc:character Presence stanzas and the pc:u Message stanzas. I don't have any of the Iq stuff yet. I'm probably 75% of the way to a MirrorBot, where the bot can clone character appearances (since I don't know how to send costume data yet, but I'm close to being able to send hashes). I thought MirrorBot would be plenty spooky enough.
-
Simple question - Is the protocol documented someplace such that a bot could simulate a client when talking to other clients?
If it's not obvious, what I want is to make a chatbot that has an avatar in the game and the ability to "speak" or emote as if it was a player. In other words, a kind of XMPP-based NPC.
As I understand things, at the current time a "player" is a collection of costume bits and a set of map coordinates.
Is that about the size of things?
Well, your chat bot would need to understand XMPP and be able to do XMPP things. I would start there, in the programming language of your choice.
Then, to the best extent I've figured out the process, you have to:
Login into the Titan XMPP server
Find and join a room or rooms. You would probably want to join a room that represents a City of Heroes zone, like "atlaspark"
You would need to join its related meta data channel, in this case atlaspark_meta
If your bot wants to send zone broadcast messages, you would then send zone broadcast chat as a muc (multi user chat message) in the zone room, aka atlaspark
To actually appear to other players, you would need to announce yourself with a Presence message in the meta data room. This would have among other things your character data - name, origin, costume hash ID, etc.
You would also need to send position data in another different XMPP message.
When Paragon Chat players got within visual range of your bot (based on position) they will send you an inquiry for what you look like. You have to respond with a proper message with your costume data.
When your bot moves, or performs an animation, or both, you have to send that as a message with some combination of position, orientation, movement velocity, and animation sequence name. Paragon Chat clients will see that message and animate your bot's visual avatar on their screens appropriately.
Rinse and Repeat.
Bingo: you're visible.
Edit: I posted the code I was playing around with for a simple invisible bot; a bot that doesn't actually appear to Paragon Chat players (because I didn't know how to do that yet, and I'm still working on it) in this post: http://www.cohtitan.com/forum/index.php/topic,10947.msg185173.html#msg185173. That should give you an idea, however, of how to get off the ground. It can log into an XMPP server, talk, reply to other players, and look up stuff in City of Data (because, of course it does). Becoming visible, however, is an order of magnitude harder. Still not hard-hard, but it takes knowledge of the protocol to get it right and I've been trying to figure it out without bothering Codewalker too much. But if he's willing to give up the stanza schemas, I'll write them up in something less byzantine and then go back to banging on bots.
-
If it's not obvious, what I want is to make a chatbot that has an avatar in the game and the ability to "speak" or emote as if it was a player. In other words, a kind of XMPP-based NPC.
I immediately think of Eliza (https://en.wikipedia.org/wiki/ELIZA). This could actually be pretty cool.
-
If you have a schema or something, I could take a swing at it.
I don't have a DTD or anything, but can sketch out the structure:
Presence schema:
<presence>
(normal XMPP stuff)
<pc xmlns="pc:presence" protocol="5" jid="user@domain/Resource" />
<character xmlns="pc:character" name="Somebody" class="Class_Blaster" origin="Science" map="maps/City_Zones/City_01_01/City_01_01.txt" instance="2" costume="{hash}" />
</presence>
The "pc" stanza indicates to other Paragon Chat clients that you are running PC. The protocol attribute is the protocol version, and is incremented when incompatible changes are made, or at least changes that other clients need to be aware of to maintain compatibility. The current protocol version is 5. The jid attribute is just a copy of your JID, needed to make things work in anonymous chatrooms where only you room alias would otherwise be visible. I suppose I should probably disclose this comment:
// CW NOTE: We are explicitly disclosing the full JID in the presence data.
// This is a big no-no according to the MUC spec, as anonymous chatrooms go
// out of their way to hide the real JID of the participant. However, we
// really do need it to keep identities straight and associated with the
// right character in the game world. So we assume that anyone using this
// program will not be joining anonymous chatrooms with it, or if they do,
// they consent to making their real JID be known to the others in it...
xmpp_stanza_set_attribute(pc, "jid", makeJid(xl->node, xl->domain, xl->resource));
The character stanza includes character information, used both to create the entities of other players in the same zone, as well as to show info in the global friends list. It really should only be sent to your roster as well as the meta channel, but currently is sent to all channels you join. I'll fix it to not disclose your location to global channels at some point.
The instance attribute is normally omitted, but contains the instance number if you're not in the first instance. It's used only to show the correct zone name in the global friends list for other people. The costume attribute is a Base64 encoded SHA1 hash of your costume data; I'll post the exact algorithm for generating it later when I have some more time (and when I'm sure it's not still causing people to be invisible, heh). When you switch costumes, PC broadcasts a new presence stanza with a different hash.
The description protocol is used to fetch the character description (bio) when someone clicks info. It is an IQ request.
<iq to="user@domain/resource" id="id12345" type="get"><description xmlns="pc:description" /></iq>
Upon receiving this iq, Paragon Chat will respond with an iq that differs only in that type="result", and the description element contains the text of the description, properly escaped (or in a CDATA) of course.
Metadata updates:
These are groupchat messages sent to the metadata channel, i.e. kingsrow_meta. They are sent when you are moving according to an algorithm designed to minimize updates where possible and batch them together. When someone joins the metadata channel, Paragon Chat starts an internal timer with a random value from 1 to 9 seconds in the future. After the timer expires, they send a full update with all of the attributes filled out so that the joining player knows where you are standing. Otherwise they might not see stationary players for quite some time. If someone else joins before the timer expires, a second update will not be sent; it assumes that a single update will cover them both.
<message type="groupchat" to="zone_meta@conference.xmpp.server">
<u xmlns="pc:u" v="1.0 2.0 3.0" p="1.0 2.0 3.0" o="1.0 2.0 3.0" m="mov" />
</message>
p is the absolute world position, 3 floating point numbers separated by a space. PC rounds to the nearest tenth to keep the message short.
v is the velocity. This is in world units per tick, where a tick is 1/30th of a second. PC rounds this to the nearest hundredth.
o is the orientation. This is pitch, yaw, roll, in radians. Note that Paragon Chat, like COH, applies the transformations in the order Yaw, Pitch, Roll despite calling it a "PYR" and the numbers being in PYR order. PC rounds to the nearest hundredth of a radian.
m is the current animation that is playing. This is the same MOV names that are used in demorecords. Paragon chat optimizes this by only sending a new MOV if the client couldn't reasonably predict it by using the "next move" and cycle info from the sequencers.
Standard messages.
There are a few special XML tags that might show up in direct and groupchat messages.
<message type="groupchat">
<body>(standard XMPP body)</body>
<channel xmlns="pc:message" id="request" />
</message>
The <channel> element is sometimes inserted into groupchat messages in the zone broadcast room. Currently the only valid attribute is id="request", which causes Paragon Chat to send the message using the Request channel rather than Broadcast.
For direct messages, the message type is chat. Normally these are handled as private tells. However, if they include a <channel> element with the id of "local", the message is handled as local chat instead. The id can also be "emote" for the emote text channel. Local and emote chat do not go to the zone channel, but are sent as direct messages to everyone who is in range. They use the XMPP multicast extension if possible (XEP-0033), otherwise they fall back to sending a bunch of individual messages.
if (xaddr) {
xmpp_send(xl->conn, message);
} else {
int i;
// have to do this the hard way...
for (i = 0; i < listsize(to); ++i) {
XMPPResource *res = to[i];
xmpp_stanza_set_attribute(message, "to", xmppJidFromResource(res));
xmpp_send(xl->conn, message); // hope we don't get banned
}
}
The color, bgcolor, and bordercolor pseudo-html tags that are allowed in COH chat messages for the chat bubble colors are converted to XML elements in the "pc:attr" namespace and inserted into the message alongside the body, so that they don't show up as ugly inline tags to people using standard XMPP clients.
Costume Protocol
I'm out of time so I'll have to do this later. It's easily the most complex thing PC does with XMPP.
Didn't have time to check for typos, so please quote anything that looks suspicious and I'll correct if needed.
-
Extremely quick summary for validation purposes:
Message stanzas/namespaces you need to support to write a visible Paragon Chat bot:
pc:presence
pc:character
pc:u
pc:description
pc:message
pc:attr
A visible bot MUST announce Presence using <pc /> and <character /> stanzas in the pc:presence and pc:character namespaces respectively, when you login, when you zone, and when you change costumes. Presence will be automatically sent to new Paragon Chat users when they first login to Paragon Chat: you do not need to periodically announce Presence (pro forma XMPP, but mentioned here for completeness sake). You should do this on the zone meta data channel and the bot's friends roster if any. Paragon Chat currently broadcasts to all channels a user is a member of but this is considered a bug: bots do not need to do this. A bot that fails to implement this protocol will be effectively invisible.
A visible bot MUST send the bot's position with groupchat messages using <u /> stanzas in the pc:u namespace to the metadata channel of the zone you're in. You can send these as often as desired, but Paragon Chat uses throttles to send this only periodically when moving, and only at periodic intervals when someone new joins the zone room if not moving. If you are not moving, not playing an animation, and no one has joined the zone room recently, Paragon Chat doesn't send positional updates. This is important if the bot has to track where players are. A bot that fails to implement this protocol will not visibly move or animate.
If a visible bot wants to talk, it can use standard XMPP <message /> message stanzas with type "groupchat" or "chat." If the type is "groupchat" and the target is a room corresponding to a City of Heroes zone, that message plays in the broadcast channel of that zone unless there is a <channel /> substanza. If so, it plays on the City of Heroes channel in that zone, although the only currently supported channel id is "Request." If the message type is "chat" and the target is a Paragon Chat user that plays as a private tell if there is no <channel /> substanza. If there is a channel substanza then if it has an id of "local" or "emote" that plays as a local chat message or an emote message to its target. Paragon Chat calculates how far every player is from its own (player's) location, and sends this message to all players within range. A bot would have to do this calculation itself: a bot cannot send a "local" message and have it automatically play to everyone closeby, because in Paragon Chat protocol local messages are send to individual users that the Paragon Chat client itself deems needs to hear it. <Channel /> substanzas are in the pc:message namespace.
A visible bot MAY implement color tags in the pc:attr namespace and use them in chat messages. The supported tags are <color />, <bgcolor />, and <bordercolor />. Using these tags in the pc:attr namespace will prevent them from showing up in chat clients that do not understand them: Paragon Chat will interpret them and color the chat bubbles on screen appropriately. Using them in the default namespace will still work, but chat clients will display the raw tags in chat messages cluttering text messages for non-Paragon Chat users.
A visible bot MAY implement handlers to read and process XMPP messages similar to the above, to see what Paragon Chat players are saying.
Note: a bot that implements the chat message protocols but none of the other ones will be invisible, but able to communicate with other Paragon Chat users.
A visible bot SHOULD implement a reply to handle Iq requests for description. If a player tries to view the description field of the bot, Paragon Chat will send a request to fetch that description using an XMPP Iq get message for <description /> in the pc:description namespace. A bot SHOULD choose to implement a response. A bot that fails to implement this protocol will not have a readable description in character information.
A visible bot MUST implement a reply to handle the Iq request to fetch costume data. When a Paragon Chat client detects that the bot is within visual range, it will send an Iq get request to the bot for <costume /> in the pc:costume namespace. The bot must respond with the appropriate data, or the Paragon Chat client will be unable to render the bot's appearance (note: costume data is cached and indexes with a hash key; if the Paragon Chat client already has a costume with that key cached, it will not request the same costume data again). The details of costume encoding are not currently published. If the bot fails to respond or responds with invalid data, the bot will be invisible.
-
That is way more detail than I expected to get without having to dig for it somewhere. Thanks loads, @Codewalker and @Arcana.
-
That is way more detail than I expected to get without having to dig for it somewhere. Thanks loads, @Codewalker and @Arcana.
Technically speaking, I think its enough to make a bot, period. Until Codewalker gives us the costume computation protocol, we can't create random bots on the fly. But a technically savvy person wouldn't need to. You set up your own Openfire server**, log Paragon Chat into it, and log your bot into it. Create a character that looks like what you want the bot to look like. Send the Iq character inquiry from the bot. Paragon Chat will respond with the correct character stanza. Have your bot record that. Now add code to the bot to play that literally when it receives Iq character requests, with the appropriate to/from/other stuff. You've now created something I earlier called a "mirror bot***." You could make a bot that "steals" appearances from other players, and then plays them back. It would be visible without needing to know the precise details of the character stanza structure. It would be limited, but functional.
With the above information and a dev server and a bot that can stanza-trace, I think its now possible. Technically, it was possible before Codewalker posted his info, which is why I was already working on it. Its just that I couldn't write a guide with any sharpness without a lot more work on determining the technical requirements, as opposed to just what happens to work.
** This step is not strictly speaking necessary, but I'm going to keep harping on any prospective bot writers to test their creations on their own servers first, rather than make a radical coding error on a live but still early beta server that real people are using. Be responsible.
*** I was going to wait until I could compute costume data before continuing Bot development, but I realized yesterday I could vampirically steal costumes from other players, including ones I create. At that moment, ArcanaBot0.1 became MirrorBot0.2. MirrorBot is probably a few dozen lines of python away from working.
Edit: although what I've described is enough to make a visible bot, its not the end of the road for bot makers by any means. There are issues like pathing and how velocity interacts with client side prediction to create smooth motion that would make bots better. Clocking, threading, zoning, object interaction, map/geometry interaction, sequence following - the sky's the limit for making a bot behave like you want it to.
Edit2: what I wrote above is *not* the documentation I intend to write to document the technical information necessary to understand how Paragon Chat works the way it works and how you might use that information to write a bot. Its a very technical summary. I will write a more detailed set of guides after I actually confirm my understanding is good enough to actually make a bot work. Codewalker's post and my post above should allow anyone with sufficient programming skill and knowledge of XML/XMPP to get started. But the more reasonable documentation will have to wait a tiny bit. I would hate to write a guide that doesn't actually work. Hopefully, I'm in striking distance of having a functional bot that provides proof of concept for the technical information above within a day or so, real life permitting.
-
Sidebar: XEP-0033
I briefly mentioned this in the description of local and text emote chat. XEP-0033, or Extended Stanza Addressing allows you to send a single message to multiple recipients. If the server supports XEP-0033 natively, it will include <feature var='http://jabber.org/protocol/address'/> in its feature discovery response (disco#info). In that case, you can format the outgoing message like this:
<message type="chat" to="domain">
<addresses xmlns="http://jabber.org/protocol/address">
<address type="bcc" jid="user1@domain/Resource"/>
<address type="bcc" jid="user2@domain/Resource"/>
<address type="bcc" jid="user3@domain/Resource"/>
</addresses>
<body>Hello, people!</body>
<channel xmlns="pc:message" id="local"/>
</message>
Paragon Chat uses bcc as the address type, as it is not necessary nor desirable for each recipient of a local chat message to see the entire list of who it was sent to.
Some servers do not support multicast natively, but instead as a separate service under a subdomain. Paragon Chat does not yet implement that style of XEP-0033, and will fall back to sending to each recipient individually if the server does not support it internally.
-
It's a shame the XMPP message format is in XML... So much wasted space spent on the message schema itself instead of the more important data. It'd be better in JSON format!
-
It's a shame the XMPP message format is in XML... So much wasted space spent on the message schema itself instead of the more important data. It'd be better in JSON format!
Perhaps, but it *is* a chat server. They're specifically designed to promote communication rather than efficiency, particularly between live people and automated responders. If you were building a game server from the ground up, you'd probably end up with something like the CoH server, compressing as much information as possible.
In any case, if part of your goal is to make it so easy to run a server that anyone can do it then you are going to prefer something comprehendable over something obfuscated in the name of efficiency.
I'll be setting up an Openfire server this weekend and doing some initial botting experiments. I really appreciate all of the info being shared here about the workings of Paragon Chat.
I had a thought out of the blue this morning, that sounds completely impractical but that sort of illustrates where things could go, eventually. For no particular reason (other than having been thinking about bots and such) I began musing about the MOO I used to help administer many, many years ago. That's when some part of my brain asked, "What if each of those NPC's in a MOO 'room' was a chat client? What if the UI to a MOO or MUSH or MUD was not a text parser but a chat server gateway?"
Mind you, I'm not suggesting that you would literally take an existing MUD and tack a XMPP gateway onto it. You'd pretty much have to build something new from scratch to make the whole thing work, but existing MUD-style game engines could be used as models for how to build a simple Architect Entertainment style of scripted mission processor.
-
Perhaps, but it *is* a chat server. They're specifically designed to promote communication rather than efficiency, particularly between live people and automated responders. If you were building a game server from the ground up, you'd probably end up with something like the CoH server, compressing as much information as possible.
Sure, but PC needs speed above all else. Personally, and this is just me and not a suggestion to anyone at all, I'd use the minimum possible XML message stanza and for the actual CoH related data itself, I'd use JSON or something even smaller (but not compressed or you'd have to decompress it), to get the data across as fast as possible. Infact, JSON's also easier on the dev to process than clunky old XML is. :p
Mind you, for all we know, CW's doing something along those lines for the costume data...
-
When working in non-managed languages, binary data is an order of magnitude more efficient and easier on the programmer than JSON is. In C, JSON is no easier to deal with than XML is.
If maximum performance was a design goal of Paragon Chat, it would certainly be using a handrolled binary protocol rather than XMPP.
But of course that wouldn't be easy for the community to plug their own stuff into.
-
When working in non-managed languages, binary data is an order of magnitude more efficient and easier on the programmer than JSON is. In C, JSON is no easier to deal with than XML is.
If maximum performance was a design goal of Paragon Chat, it would certainly be using a handrolled binary protocol rather than XMPP.
But of course that wouldn't be easy for the community to plug their own stuff into.
TBH, I was primarily thinking of the unwieldy verbosity of XML over JSON. Obviously, binary is better, but not human readable. JSON is! (Not that I'm evangelising JSON. I'd be inclined to just go with comma separated data if I was that worried about brevity + readability).
-
And if there were a bunch of premade servers for routing JSON-based messages between multiple peers, then it might be an option. :P
Embedding JSON inside XMPP seems even sillier as then you'd need two separate parsers.
In practice, the difference in verbosity is really a non-issue. XMPP very strongly pushes TLS for connection privacy; and TLS, at least in the configuration we're using it in, uses DEFLATE compression. So all of those redundant end tags and repeated attributes are replaced by tiny binary backreference tokens, taking up no more space than JSON's "}".
Yes, there is additional CPU time wasted on compression, but far less than the amount you're spending to encrypt the data.
-
Embedding JSON inside XMPP seems even sillier as then you'd need two separate parsers.
Technically speaking, I had to write an attribute parser to yank values out of <u /> blocks, since those aren't technically "XMLified" (i.e. <pitch>0.1</pitch>, or even <u xmlns="pc:u" pitch="0.1" roll="0.0" yaw="0.3" ... etc). Bad, bad monkey. :P
But no, I would not want to embed JSON inside of XML sent as XMPP. If you were going to go JSON, you'd go all the way and make Paragon Chat push JSON to a central web server, and fetch JSON from the same web server using state pages to echo dynamic state to the clients. Its possible (JSON push "I'm here"; JSON fetch "who's here"'; JSON push "send message to zone: 'yo!'"; JSON fetch "latest messages from zone X") but you'd have to write that central server. And the latency would probably be horrible. Instead of City of Heroes as chat system, it would effectively be City of Heroes as message board.
-
Okay, here it is, the costume exchange protocol.
First, some definitions.
COLOR: For the purposes of this document, COLOR is an 8-digit hexadecimal representation of a color. The order of the digits is RRGGBBAA. The numbers a-f must be represented in lowercase (important for hashing). The alpha channel MUST be ff in all cases; Paragon Chat enforces this by ORing the alpha with 0xff before hashing or sending the costume.
The alpha value is unused by the game, but the client sometimes sticks random numbers there that have no meaning. In hindsight, using 6 digits for the protocol would have been better, but it's too late now to change it without breaking things.
COSTUME HASH: A costume hash is a short (~28 byte) string that uniquely identifies a costume. These hashes are sent in presence stanzas. When a client needs a costume, it can use the hash to request a peer who owns that costume to send a copy. Once a client has received a costume, it should cache the costume locally for a reasonable length of time and associate it with the corresponding hash, so that it does not need to be requested again.
To compute the costume hash, first concatenate together the following into a single byte array or string*:
BodyType + SkinColor + Scale + BoneScale + HeadScale + ShoulderScale + ChestScale + WaistScale + HipScale + LegScale + Head3DScale + Brow3DScale + Cheek3DScale + Chin3DScale + Cranium3DScale + Jaw3DScale + Nose3DScale + Parts[]
BodyType is a string that is "0" for male, "1" for female, or "4" for huge.
SkinColor is a COLOR.
Each *Scale is a string representation of a floating point number for each of the scale sliders; as found in a costume file. They should be truncated to at most 4 decimal places before hashing and trailing zeroes removed. Scales which have a value of 0 are skipped and not concatenated.
Each *3DScale is an 8-digit hexadecimal number representing the encoded version of a 3-dimensional floating point vector. How to calculate this will be covered below. Encoded scales which have a value of 00000000 are skipped and not concatenated.
For each costume part, numbered 0 to 27 in the same order they appear in a saved costume file or demorecord, examine the Geo and Fx of the part. If they are both empty or "None", skip this costume part and do not hash it. Otherwise, concatenate the decimal string representation of the number (0-27) of the part to the string to be hashed. Then concatenate the Geometry, Texture1, Texture2, and Fx strings, in that order, if they are not empty or 'None'.
For each Color1, Color2, Color3, or Color4 of a costume part that is not skipped, and where the color is not black (RGB = 000000), first concatenate either a "1", "2", "3", or "4" depending on which color it is, then the hex COLOR representation.
Once all of the valid parts are concatenated, take the SHA-1 of the combined string. Base64 encode the resulting hash. The Base64 result is the costume hash.
* Or use incremental SHA-1 updates, which is what PC does, but saying to concatenate it all is easier to understand for documentation purposes and has the same end result.
Costume request protocol:
Upon encountering an entity whose costume hash is known, but no valid costume is cached for, Paragon Chat sends a costume request to the peer in this form:
<iq to="user@domain/resource" id="id12345" type="get"><costume xmlns="pc:costume" hash="ZAqyuuB77cTBY/Z5p0b3q3+10fo="/></iq>
When such a request is received, if the hash corresponds to a known costume, Paragon Chat will reply with a copy of the costume in the following format:
<iq to="user@domain/resource" id="id12345" type="result">
<costume xmlns="pc:costume">
<appearance bodytype="1" skincolor="123456ff"
scale="2.1" bone="0.5" head="0.5"
shoulder="0.5" chest="0.5" waist="0.5" hip="0.5" leg="0.5"
head3d="123456ab" brow3d="123456ab" cheek3d="123456ab"
chin3d="123456ab" cranium3d="123456ab" jaw3d="123456ab"
nose3d="123456ab" />
<part n="0" geom="Leather_02"
tex1="Leather_03a" tex2="Leather_03b"
fx="Auras/Female/Gaseous/GasEyes.fx"
color1="123456ff" color2="123456ff"
color3="123456ff" color4="123456ff" />
<part n="1" ... />
...
</costume>
The rules for composing the stanza are identical to the rules for computing the hash. Scales with a value of 0.0 may be omitted from the appearance element. Encoded 3D scales with a value of 00000000 may also be omitted.
Costume parts that have neither a geom nor an fx (or where they are 'None') may be omitted. The "n" attribute is the decimal index of the costume part, from 0 to 27, so that the order can be reconstructed even if parts are omitted. geom, tex1, tex2, and fx attributes should be omitted if they are empty or 'None'. color* attributes should only be included if they have a value other than 000000ff.
3D scale encoding:
The three dimensional face scales are encoded in an optimized form comprised of a single 32-bit integer value rather that represents a 3-element floating point vector. This form comes from the COH network protocol and is how costumes are transmitted over the wire. It is only applicable to three-element vectors with each component having a value between -1.0 and 1.0.
The 32-bit integer is divided into 3 8-bit sections:
31 23 15 7 0
| | | | |
00000000 Szzzzzzz Syyyyyyyy Sxxxxxxxx
First, the value of each component (X,Y,Z) of the vector should be clamped to the range -1.0 to 1.0. It is multiplied by 100 and truncated, then encoded as an 8-bit integer with the 'S' acting as a sign bit. Bits 24-31 are unused and must be 0.
This encoding ensures that a vector of 0,0,0 is represented by an integer 0 so that it can be optimized out of transmission.
-
(https://38.media.tumblr.com/9cb88303a62b31d8665922a60e7b69ca/tumblr_mqr4a4EXkN1rpd22bo10_r1_400.gif)
-
(https://38.media.tumblr.com/9cb88303a62b31d8665922a60e7b69ca/tumblr_mqr4a4EXkN1rpd22bo10_r1_400.gif)
Believe it or not, there are very good reasons for most of that... For example the skipping of empty or 'None' parts and geos is because the client treats them as interchangeable, so a costume you send to the COH client with a NULL Geometry might very well come back as "None" later and hash to something different.
Technically speaking, I had to write an attribute parser to yank values out of <u /> blocks, since those aren't technically "XMLified" (i.e. <pitch>0.1</pitch>, or even <u xmlns="pc:u" pitch="0.1" roll="0.0" yaw="0.3" ... etc). Bad, bad monkey. :P
There are two schools of thought with XML. One is to make everything into an element and character data inside the element. I find that makes things annoying to parse more often than it doesn't, and leads to ambiguities that are difficult to resolve.
I'm a follower of the second, which is to use elements only for things that should either be able to contain other elements, or that it makes sense to have more than one of (because elements can be repeated). Attributes should be used for simple and unique data. It wouldn't make much sense for a single update message to have multiple orientations, after all.
Or are you complaining about shoving the X/Y/Z and P/Y/R values into a single attribute? I suppose a valid argument can be made there that they should be separate. I was just thinking along the lines of it being a single value of type Vector rather than 3 individual floats.
-
Or are you complaining about shoving the X/Y/Z and P/Y/R values into a single attribute? I suppose a valid argument can be made there that they should be separate. I was just thinking along the lines of it being a single value of type Vector rather than 3 individual floats.
That was one of the two examples I gave for better "XMLization" - either tag everything, or if using attributes break up the values into separate attributes instead of structured data that required separate parsing.
I was joking more than complaining: since I'm writing in python I can pretend those are actually string tokenized vectors. I can almost treat them as actual vectors with something like:
def message_insert_position(self, pmsg):
pmsg['u']['p'] = ' '.join([str(x) for x in self.position])
pmsg['u']['o'] = ' '.join([str(x) for x in self.orientation])
pmsg['u']['v'] = ' '.join([str(x) for x in self.velocity])
and
def bot_update_position(self, pmsg):
self.position = [float(x) for x in pmsg['u']['p'].split(' ')]
self.orientation = [float(x) for x in pmsg['u']['o'].split(' ')]
self.velocity = [float(x) for x in pmsg['u']['v'].split(' ')]
Its *barely* parsing. The question is when I get to motion tracking, will I treat those as vectors with vector calculations, or will I loop through Cartesian directions? That'll have to wait until I can troubleshoot an odd problem with some of my presence stanzas apparently going out as quoted escaped bodies of message stanzas. I didn't have a lot of time last night to work on it.
-
I was joking more than complaining
In case it wasn't obvious in text form, my word choice itself was intended as a joke.
-
My dreams of adapting an existing bot with a PC plugin are starting to look like so much star dust, lol.
I guess I'm going to have to dive into the nuts and bolts of XMPP to do this right.
-
My dreams of adapting an existing bot with a PC plugin are starting to look like so much star dust, lol.
I guess I'm going to have to dive into the nuts and bolts of XMPP to do this right.
I believe it is highly unlikely you'll be able to repurpose existing bot technology here. Most bot technology is designed to manipulate game clients or computer input (i.e. pushing buttons, typing keys, and using pattern recognition to locate buttons and other screen elements to drive the bot actions). Unless you want to write a bot that literally takes over the City of Heroes game client, and can only do what a player can do operating that client, you'll have to implement the actual XMPP communications channel instead.
This is something few if any bots do, because its ordinarily so complicated and difficult of a task. However, in this case, as complex as Codewalker's information appears, it is infinitely easier to generate XMPP to pretend to be a Paragon Chat bot than it would be to try to do this directly with a game's client under normal circumstances. You might be able to make a client-input-controlling World of Warcraft bot in an afternoon, but a native WoW bot that talks directly to the WoW servers? That's a task and a half. I'm all the way to an invisible bot and most of the way to a visible Paragon Chat bot (that can do things no player can do) in about ten cumulative hours of work. That's ten hours to learn XMPP, understand how stanzas work, learn a python XMPP framework (sleekxmpp), write practice code, test it, and also install and configure a dedicated Openfire test server to test my code without interfering with live Paragon Chat users.
Starting from scratch ten hours probably doesn't get you past the WoW authentication servers.
However, if all you want to do is literally run around like a player might, you could use someone's bot framework and write a bot that basically takes over keyboard and mouse from you after you start up Paragon Chat. What I want to do can't be done that way. But if what you want to do can be, it might be easier than implementing a native bot from scratch.
-
I got the impression slickriptide was talking about something more akin to adapting an IRC chat bot or response bot, which is a lot closer in principle to how a Paragon Chat bot would work.
I could probably whip up some reference code of the costume hash at some point. Maybe in Python since it's easy to do quick prototypes in, and would plug in to your existing bot framework.
-
I got the impression slickriptide was talking about something more akin to adapting an IRC chat bot or response bot, which is a lot closer in principle to how a Paragon Chat bot would work.
On the surface yes, but when slickriptide specified that he/she wanted an avatar and the ability to both speak and emote, to me that immediately left the realm of text chat botting, at least based on my knowledge of the limits of how those tend to work. The IRC part would be doing so little compared to how much you would have to build to make work as to make the contribution of the IRC bot itself seem not worth the effort to me. Perhaps there are better chat bot frameworks that I'm not familiar with that you could adapt. That's outside my wheelhouse.
-
Also sounds vaguely like a MUD bot or plugin running an NPC script, but those usually don't have to worry about costumes and 3D positioning...
-
Also sounds vaguely like a MUD bot or plugin running an NPC script, but those usually don't have to worry about costumes and 3D positioning...
If you would be so kind as to add /move_left, /move_right, /move_forward, /move_backward, /move_leap, /turn_clockwise, /turn_counterclockwise, /display_costume_file, and /use_item slash commands in version 0.97, we could turn City of Heroes into Zork and bot writing would become a heck of a lot easier.
-
If you would be so kind as to add /move_left, /move_right, /move_forward, /move_backward, /move_leap, /turn_clockwise, /turn_counterclockwise, /display_costume_file, and /use_item slash commands in version 0.97, we could turn City of Heroes into Zork and bot writing would become a heck of a lot easier.
That's actually why one of the future goals is to add a scripting API using Lua or something similar to Paragon Chat eventually, so that you can leverage its knowledge of the game world to move around without having to load and parse maps and geometry, etc.
That's a ways off though, so for now XMPP bots with static locations or fixed paths are probably the way to go.
-
I'm actually working with Sleekxmpp but it was only a couple of hours ago that I dug up the developer docs on Github that talked about how to create custom stanzas. Now I'm feeling more confident that it actually can do the sort of processing required to deal with the special Paragon Chat <pc:> and <u:> stanzas and whatever else there is or might potentially be.
I am not imagining anything terribly complicated for a first outing. For a chatbot-based NPC, I envision something like old-school Everquest NPC's. Imagine the following:
A little girl is standing near a bus stop. Periodically, she does a "worry" emote and says, "Oh, my poor kitty!" or "I'm so worried about my kitty!"
If someone local says, "What about your kitty?" or really, anything with the word "kitty" in it, the girl replies that her kitty is stuck in a tree and can't get down.
If the erstwhile hero asks where is the tree, she says, "Follow me" and leads the hero around the zone to a place with a tree and an invisible bot in that location.
If the hero chooses any of a number of verbs (shake tree, kick tree, punch tree, etc...) the invisible bot gives itself a costume hash, becomes visible,and a catgirl drops out of the sky, meows threateningly, and runs away leaving the hero to accept the thanks of the little girl and wonder what the hell just happened, heh.
All of that ought to be doable, though the moving around bit would present some challenges.
Speaking of stanzas, I've been wondering - Are there dtd's for the namespaces of those custom stanzas or is it enough to simply define the name spaces and leave the structure of them ambiguous? If Paragon Chat has DTD's defined for <pc:> and <u:> and what not, do bots that define the same name spaces require those DTD's?
I think what I'm really going to be interested in whether a bot can induce a change in the state of a player. For instance, I've edited my character record in the SQLite database to put my character next to the flagpole atop city hall. It would be very interesting to be able to send a message to a player that says, "Change to these x,y,z coordinates". You could make a taxibot that offers a series of destinations and a player could say, "Taxibot, flagpole" to be "teleported" to the flagpole.
-
I'm actually working with Sleekxmpp but it was only a couple of hours ago that I dug up the developer docs on Github that talked about how to create custom stanzas. Now I'm feeling more confident that it actually can do the sort of processing required to deal with the special Paragon Chat <pc:> and <u:> stanzas and whatever else there is or might potentially be.
It most definitely can. Its a little frustrating figuring out the language, but the capabilities are all there. Just don't ask me send/send_X semantics questions yet.
Speaking of stanzas, I've been wondering - Are there dtd's for the namespaces of those custom stanzas or is it enough to simply define the name spaces and leave the structure of them ambiguous? If Paragon Chat has DTD's defined for <pc:> and <u:> and what not, do bots that define the same name spaces require those DTD's?
Codewalker described all of them above. However, since I already worked out sleekxmpp's kinks:
from sleekxmpp import ClientXMPP
from sleekxmpp.xmlstream.stanzabase import ElementBase
class pcCharacterStanza(ElementBase):
namespace = 'pc:character'
name = 'character'
plugin_attrib = 'character'
interfaces = set(('origin','map','class','name','costume'))
class pcPresenceStanza(ElementBase):
namespace = 'pc:presence'
name = 'pc'
plugin_attrib = 'pc'
interfaces = set(('jid','protocol'))
class pcuStanza(ElementBase):
namespace = 'pc:u'
name = 'u'
plugin_attrib = "u"
interfaces = set(('p','m','v','o'))
class pcChannelStanza(ElementBase):
namespace = 'pc:message'
name = 'channel'
plugin_attrib = 'channel'
interfaces = set(('id'))
class pcDescriptionStanza(ElementBase):
namespace = 'pc:description'
name = 'description'
plugin_attr = 'description'
(warning: do not list subinterfaces like a lot of sleekxmpp demo code does. That will convert stanza attributes into substanzas, i.e. it will turn <u xmlns="pc:u" o="0 0 0" /> into <u xmlns="pc:u"><o>"0 0 0"</ o></ u>. Demo writers like to teach highly normalized XML/XMPP.)
You have to register those as plugins to the appropriate parent stanza:
registerStanzaPlugin(Presence,pcPresenceStanza)
registerStanzaPlugin(Presence,pcCharacterStanza)
registerStanzaPlugin(Iq,pcDescriptionStanza)
registerStanzaPlugin(Message,pcuStanza)
registerStanzaPlugin(Message,pcChannelStanza)
You then make handlers that access the contents of a stanza like so:
def handle_pcPresenceStanza(self,msg):
print "Debug: pcPresence jid:" + str(msg.values['pc']['jid'])
print "Debug: pcPresence character costume: " + str(msg.values['character']['costume'])
and register them as a callback in the appropriate xmpp object, with the appropriate XPath matching pattern:
self.registerHandler(Callback('pcPresenceStanza Handler', MatchXPath('{%s}presence/{%s}pc' % (self.default_ns, pcPresenceStanza.namespace)), self.handle_pcPresenceStanza))
Technically speaking, you don't need to implement stanza plugins because you can just register handlers to process all Iq, Presence, and Message stanzas and write your own parser to parse the raw stanza information (ala str(stanza_msg)), but that's crazy.
I haven't written the costume code yet, so those stanza definitions aren't above yet. But you should be able to figure out how to do that by following the examples.
I think what I'm really going to be interested in whether a bot can induce a change in the state of a player. For instance, I've edited my character record in the SQLite database to put my character next to the flagpole atop city hall. It would be very interesting to be able to send a message to a player that says, "Change to these x,y,z coordinates". You could make a taxibot that offers a series of destinations and a player could say, "Taxibot, flagpole" to be "teleported" to the flagpole.
At the moment I do not believe that is possible, because I don't think Paragon Chat itself supports that. Which is why I'm investigating bots. Paragon Chat doesn't "react" to events, but nothing stops a bot from doing so. To put it another way, at the moment a bot cannot knock a player over. But hypothetically speaking, a player can knock a bot over, because bots can take dives.
Just a note: your bot won't have any idea where flagpoles and buildings are, and for that matter where the ground is, unless you write special code that can read City of Heroes mapfiles. In the absence of such an ability, you will be forced to "hardcode" where the bot goes so it doesn't pass through walls or move downward through the ground. Your bot doesn't have the game client's collision detection logic, so you have to write your own or figure out another way for the bot to go where you want it to go with the right movement path that "looks right" to everyone else in the game.
That's what the whole LUA-thing exchange was between Codewalker and myself just now. Theoretically speaking, Paragon Chat (the software) itself knows what players can and cannot do in large part, including knowing how to read maps and detect collisions. Its hypothetically possible for Codewalker to add features to Paragon Chat that allow bot writers and other scripters like machinima authors to tell Paragon Chat what they want the bot to do, and have Paragon Chat tell the bot if those actions are legal, or alternatively calculate their result and push that back to the bot. That would basically put me out of the bot-writing business (and put me in the LUA scripting business).
-
Thanks for the code @Arcana. That is substantially what I was composing in my brain after reading the docs, though I was still scratching my head over the way Sleekxmpp tends to treat attributes and elements as somewhat interchangable from a data manipulation standpoint.
My question about namespaces is a non-issue. It's so ingrained in me that a xml namespace is a URI that it never occurred to me that the URI was simply a convention without inherent meaning. I learned something new. Thanks wikipedia! ;-) (Due, no doubt, to Googles's web page schema data initiative being my first exposure to name spaces.)
Personally, I look forward to the day that I get out of the bot scripting biz and into the LUA scripting biz. ;-)
I think maybe you're right about the teleporting. Given what we know about the way movement used to work, it seems more likely that "teleport" was really a case of the client saying to the server, "I'm jumping over fo point X,Y,Z now using power Blah" and the server replying, "Okay, that's legit. I won't rubber band you back where you started."
OTOH - Team teleport illustrates that some sort of triggered moving mechanism had to exist in the client for that to work.
The server had the final say about the character's position in the world. With Paragon Chat acting as a kind of stub server, it might be possible for Codewalker to make a "move character #NAME to point X, Y, Z" primitive that a bot or script could invoke.
-
I've have gotten a whole bunch of power animations working. Yay me! ;)
However, they have no FX, therefore only the basic fighting ones are of much use. Is there a way of spawning power fx? If not... Is that in the TODO list that CW hasn't written up yet? ;)
Also, many animations are made of multiple seqbits, yet chaining with mutliple Bits lines in emote.cfg does nada. Is it possible to chain movs?
-
There is no support for Fx yet, as Fx are more complex in that they need source and target entities, and have to be handled differently for persistent fx versus one-shot fx. There are also some anti-griefing protections that need to be dealt with as there are some screen space Fx that do things like blur, shake, or black out everybody's screen.
Also, many animations are made of multiple seqbits, yet chaining with mutliple Bits lines in emote.cfg does nada. Is it possible to chain movs?
No, emotes are emotes and briefly assert a specific set of bits when used.
I'm not sure why you would need that, though. Not even powers do that, they all just set the bits once and let the sequencers handle chaining between moves.
-
Ah, no worries, I was misreading how the sequencer ran the bits. I understand better now. :)
For the FX part, should not the targeting aspect work automatically, as long as I have someone actually targeted? Also, I quite understand about the griefing part, but as the ignore mode utterly removes an offender from your view etc, I presumed you wouldn't see that either...
Anyway, no rush on any FX stuff.. That's a "would be nice" thing way down the road of lots of more important stuff! ;)
-
The selection (target) has nothing to do with fx playing. Fx have distinct source and target entities that don't necessarily have any relation to your actual target. For example, Fx designed to be played as power hit fx operate backwards, where the source is the entity that is being hit, and the target is the entity that launched the attack.
Ignore is good, but the problem with screen space Fx is that in a crowded zone, there wouldn't be any way to tell just who was causing the offending Fx to play.
-
OK, gotcha. So my next question is. What about player based fx such as heal aura's and what not? No target other than the actual player to worry about, then.
-
Yeah, that would be nice to have, though as I mentioned about the toggle fx in costumes, Paragon Chat doesn't load the fx info so it has no way of knowing what kind of fx a given name is.
I'm not sure if loading the fxinfo is the best way to go in the long run, or if parsing the powers data and gleaning it from what kind of powers it's used by would be the safer bet. Either way means taking a big hit on Paragon Chat's startup time and memory usage, though I suspect if the powers data could be used for other things people may not mind so much.
-
The selection (target) has nothing to do with fx playing. Fx have distinct source and target entities that don't necessarily have any relation to your actual target. For example, Fx designed to be played as power hit fx operate backwards, where the source is the entity that is being hit, and the target is the entity that launched the attack.
Ignore is good, but the problem with screen space Fx is that in a crowded zone, there wouldn't be any way to tell just who was causing the offending Fx to play.
One possibility is to use an opt in mechanism whereby Paragon Chat will let fx play on its own client, and send messages to everyone else with a special <u /> that has to be approved by the destination with some mechanism. If it is, you will see the results of that person's fx spawning. If you do not, you will not see. Basically a reverse ignore. By default you ignore everyone, opt in allows you to see. Then this can be used in things like private maps or RP zones where everyone agrees to play nice. Something like /enable_fx and /disable_fx, with parameters to enable and disable a specific person's fx commands. For the technically minded, you could send combat spam to a CoH channel we could monitor that would explicitly state what fx we were seeing and how the spawner was (i.e. ArcanaBot initiated SomeKindOfFX from source to destination). Sufficiently technically adept players could use that channel to identify and report griefers.
Its not perfect, but what it does is let people experiment with it even though most people may not see it, while eliminating all the griefing possibilities until a more global way to deal with fx griefing is established.
-
Yeah, that would be nice to have, though as I mentioned about the toggle fx in costumes, Paragon Chat doesn't load the fx info so it has no way of knowing what kind of fx a given name is.
I'm not sure if loading the fxinfo is the best way to go in the long run, or if parsing the powers data and gleaning it from what kind of powers it's used by would be the safer bet. Either way means taking a big hit on Paragon Chat's startup time and memory usage, though I suspect if the powers data could be used for other things people may not mind so much.
Similar to emote.cfg, I would recommend exporting the powers database to a more efficient searchable form (most obvious solution would be to put the data into a sqlite database) as a one-time thing when the database does not exist, and eat the start up cost only once when you first run Paragon Chat. Not sure if sqlite is fast enough for our purposes here, but maybe some kind of in-memory hash table could speed up search enough to make it credible.
-
Then this can be used in things like private maps or RP zones where everyone agrees to play nice.
That's kind of along the lines I was thinking for private map support once that is implemented. The creator of the map is effectively the GM while you're on it, and everyone's clients trust them fully. They can designate others as helper GMs (probably by making them admins of the XMPP room) who also gain those abilities.
Stuff like Fx could either be enabled by default on private rooms, trusting the 'GM' to enforce rules while there, or start out disabled and can be enabled for specific people or everyone with a command.
-
I take the Golden AN, and I put it in the TAN VAN, yeah, and then I take it to Horace...
I think I'm getting close, but I'm probably missing some small critical XMPP protocol thing. So I connect and announce presence in the atlaspark_meta room:
<presence from="arcanabot@core3770/ArcanaBot" xml:lang="en"><pc xmlns="pc:presence" jid="arcanabot@core3770/ArcanaBot" protocol="5" /><character xmlns="pc:character" origin="Magic" map="maps/City_Zones/City_01_01/City_01_01.txt" costume="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" name="ArcanaBot" class="Class_Blaster" /></presence>
And then I groupchat my location to the room name:
<message to="atlaspark_meta@conference.core3770" type="groupchat" xml:lang="en" from="arcanabot@core3770/ArcanaBot"><u xmlns="pc:u" p="146.4 0.3 -98.4" m="READY" o="0.0 -2.83 0.0" v="0.0 0.0 0.0" /></message>
And then I wait. I'm pretty sure that is sending in the right way, because the bot sees that Presence boomerang back (and ignores it, but it sees it) as well as the <u />. And when Paragon Chat logs into the same room, I see its Presence:
<presence to="arcanabot@core3770/74096bbe" from="atlaspark_meta@conference.core3770/Arcana"><pc xmlns="pc:presence" jid="arcanacoh@core3770/Arcana" protocol="5" /><character xmlns="pc:character" origin="Magic" map="maps/City_Zones/City_01_01/City_01_01.txt" costume="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" class="Class_Blaster" name="Arcana" /><x xmlns="http://jabber.org/protocol/muc#user"><item affiliation="owner" jid="arcanacoh@core3770/Arcana" role="moderator" /></x></presence>
When I see a new Presence, I retransmit <u />:
<message to="atlaspark_meta@conference.core3770" type="groupchat" xml:lang="en" from="arcanabot@core3770/ArcanaBot"><u xmlns="pc:u" p="146.4 0.3 -98.4" m="READY" o="0.0 -2.83 0.0" v="0.0 0.0 0.0" /></message>
I see Paragon Chat do the same:
<message to="arcanabot@core3770/74096bbe" type="groupchat" from="atlaspark_meta@conference.core3770/Arcana"><u xmlns="pc:u" p="116.4 -0.5 -89.3" m="READY2" o="0 -1.55 0" /></message>
And then... nothing. I keep sending, PC keeps sending, but I don't see the bot in Paragon Chat. Not sure why. I engineered it so the bot has the same costume hash as the Paragon Chat client, so I do not need to support Iq costume, but even if that was wrong I should have then seen an Iq request for costume, which I don't. Paragon Chat doesn't think I'm "there" yet, and I'm still trying to hunt down why.
I hope its not one of a hundred miscellaneous XMPP protocol thingies that I presumed XMPP would take care of but isn't for some reason. That might be difficult to track down an hour at a time. Separate from the atlaspark_meta channel, the bot also joins the atlaspark channel, and that works fine. Chat clients can see the bot's presence, the bot can chat with them, the bot can hear what they say. Its not totally broken, so the problem is likely in the pc: namespace stanzas somewhere.
On the plus side, I do have wasd moving me around. In a theoretical sort of way. I'm working on a radar for the bot, so it can show the locations of every Paragon Chat character in the zone that it receives position data for. No map info yet, just radar in space. But actually becoming visible is my high priority figure-out at the moment. If I can get the bot visible, I can then clean up all the debug code and probably start writing documentation. At the moment, I can only document how to make a bot that nobody can see, although it tries really hard in a sad "I'm here, can anyone see me?" sort of way.
-
Do you have the latest update, Arcana?
Versions prior to 0.97c had a bug in the order of processing where the metadata channel presence callback would be called before the main presence handler (that parses <character>). So if the first presence received was for the meta channel, it would fail to create the proxy entity, making position updates meaningless.
This was masked by the separate issue that PC sends <character> in every presence stanza, including those to the broadcast channel and globals (i.e. Paragon Chat). So in most cases it had already seen <character> first. Once I fixed that issue the processing order bug became readily apparent. But your bot might very well be joining the metadata room first and bumping into that.
-
Do you have the latest update, Arcana?
Versions prior to 0.97c had a bug in the order of processing where the metadata channel presence callback would be called before the main presence handler (that parses <character>). So if the first presence received was for the meta channel, it would fail to create the proxy entity, making position updates meaningless.
This was masked by the separate issue that PC sends <character> in every presence stanza, including those to the broadcast channel and globals (i.e. Paragon Chat). So in most cases it had already seen <character> first. Once I fixed that issue the processing order bug became readily apparent. But your bot might very well be joining the metadata room first and bumping into that.
I tested with 0.97c today, although amusingly enough I was going to PM you to ask if it was possible that Paragon Chat needed to see me in the normal room first before it would accept metadata updates, because *I* originally had the bot signing into the two rooms with different names. Switching to identical handles didn't fix the problem, so I assumed the bug probably wasn't that.
-
It did used to need that because of the bug, but it should be fixed now. The design intent is that a bot should be able to join the metadata channel and not have to join the broadcast channel unless it wants to participate in zone broadcast messages.
-
I had the notion that there was a possibility there was a subtle Paragon Chat bug where two people with exactly identical costumes might not see each other, but then I realized I could easily test for that myself by just running Paragon Chat twice, and as it turns out you can have as many exact clones of yourself as you want. So identical costume hashing is not the issue. I think the next step is to trace two Paragon Chat clients talking to each other, and stanza for stanza compare that to a Paragon Chat bot to see what's different, if anything.
-
One point of confusion that I don't think is central to the problem, but has turned up in my stanza by stanza tracing. Your spec says Paragon Chat advertises jid in the <pc /> stanza, explicitly stated to be user@domain/resource. However, what I see coming from Paragon Chat is user@domain/CharacterName, which if I'm understanding XMPP correctly (I might not be) isn't the assigned Resource from the server. The user@domain/resource JID I get assigned to me from Openfire is more like arcanabot@core3770/f57daa0e. arcanabot@core3770/ArcanaBot is the bot, and direct messages sent to that address work, as do arcanabot@core3770/f57daa0e. But does Paragon Chat require something specific, like user@domain/CharacterName to work correctly with <pc /> presence substanzas? Or does it not matter. Maybe I'm just confused on terminology: I've only been speaking XMPP for two weeks. But I thought "f57daa0e" was the resource, and "ArcanaBot" (character name) was the nickname.
Honestly, I've tried it both ways and neither fix invisibility, but resolving this ambiguity would mean one less thing to have to check for.
-
The resource is only auto-assigned by the server if you don't specify one when you log in. The resource is normally up to the client to decide how it wants to identify itself. For example, you might set Pidgin on your work laptop to use a resource of 'work', so your full JID would be user@domain/work
Paragon Chat uses the character name you log into as your resource. It does not have any specific requirements or even care what the resources of other XMPP users are, other than remembering them in order to distinguish separate clients using the same base JID. The only place it gets character name from is the character stanza in the presence.
-
I think the next step is to trace two Paragon Chat clients talking to each other, and stanza for stanza compare that to a Paragon Chat bot to see what's different, if anything.
If it helps, the version of 0.97d uploaded a few minutes ago (link (http://www.cohtitan.com/paragonchat/ParagonChat.exe)) has a -debug command line option that causes it to show all XMPP sent and received in a console window.
It's newer than the 0.97d in Tequila and that was uploaded yesterday, but didn't see a reason to bump the version number just for that.
-
And then... nothing. I keep sending, PC keeps sending, but I don't see the bot in Paragon Chat. Not sure why. I engineered it so the bot has the same costume hash as the Paragon Chat client, so I do not need to support Iq costume, but even if that was wrong I should have then seen an Iq request for costume, which I don't. Paragon Chat doesn't think I'm "there" yet, and I'm still trying to hunt down why.
Does Paragon Chat store the costume of its own avatar in its cache?
Is it possible that PC is recognizing the hash as "known", preventing it from requesting a costume stanza from the bot? If that was the case and PC did NOT have a cached copy of its own avatar, it would have nothing to display for the bot.
On a different note: Are possible values of the "map=" attribute documented someplace, or discoverable by pigg diving?
-
Does Paragon Chat store the costume of its own avatar in its cache?
Is it possible that PC is recognizing the hash as "known", preventing it from requesting a costume stanza from the bot? If that was the case and PC did NOT have a cached copy of its own avatar, it would have nothing to display for the bot.
Yes, and it serves costumes requests from peers out of the cache. It periodically refreshes the player's costume in the cache at intervals half that of the cache expiration to ensure that it is always available to serve responses to costume requests with.
On a different note: Are possible values of the "map=" attribute documented someplace, or discoverable by pigg diving?
It could be pretty much any map file (even ones that don't exist), though right now only the ones in zone.cfg are meaningful. The corresponding names for the zone maps can be looked up in clientmessages-en.bin. Mission maps don't have translations there, so if somebody adds a mission map to their zone.cfg, global friends will see the map filename in the list.
-
I have a technical / "has anyone asked this" type of question. ( I have the "has anyone asked this" cause I couldn't find a post on it ).
Naturally this would be a "after costumes and powers and such are working", but has the idea been tossed around about what it would take to get Architect Entertainment up and running? I know there is no server at the moment to back it up, but would it be theoretically possible that a chat server could store the saved maps/meta data to serve up when requested by 1 to n users?
I would think the clients would handle the actual "play" of the scripts and assets transmitted, yes?
-
Does Paragon Chat store the costume of its own avatar in its cache?
Is it possible that PC is recognizing the hash as "known", preventing it from requesting a costume stanza from the bot? If that was the case and PC did NOT have a cached copy of its own avatar, it would have nothing to display for the bot.
It slipped my mind this might not have been discussed in the thread, in a private question I asked Codewalker, he stated that PC caches its own costume, which is what made me realize that someone who was writing a bot and had not yet perfected costume stanzas could still make itself visible by cloning another character if it knew their has (which it would, because it would know everyone's hashes from their Presence stanzas).
-
I have a technical / "has anyone asked this" type of question. ( I have the "has anyone asked this" cause I couldn't find a post on it ).
Naturally this would be a "after costumes and powers and such are working", but has the idea been tossed around about what it would take to get Architect Entertainment up and running?
Discussed? Yes. On a technical level? No. Only Codewalker would really know how much effort it would take to do this, although I suspect its more than people think for several technical reasons, not the least of which would be:
I know there is no server at the moment to back it up, but would it be theoretically possible that a chat server could store the saved maps/meta data to serve up when requested by 1 to n users?
Theoretically, anything is possible. Openfire has a plugin architecture that allows anyone to write special extensions to the chat server. You could write one that created an architect mission database, accepted messages to store them, and served Iq requests to retrieve them. However, I should point out that in practice, what you are asking is comparable to asking whether, in the year 1215, it would be theoretically possible to recreate the spark plug. The answer is yes, it would take a lot of effort to do so, and most importantly alone it would be practically useless. Making an architect mission server would be the last step in a very long list of things necessary to get player generated mission content working. We don't even have a way to get NPCs to move without ghosting through walls or falling through the floor, or really at all yet, because NPC AI was a server-side feature: clients have no capability of doing that on their own (which is part of the whole reason behind thinking about bots, which could pretend to be NPC and have their intelligence within the bot and not require anything special from the player clients).
I would think the clients would handle the actual "play" of the scripts and assets transmitted, yes?
In a perfect world, yes. But that would require adding an enormous amount of logic to Paragon Chat itself, because a lot of things City of Heroes did was only done on the servers and sent to clients. It would be difficult to make a server plugin that became an entire mapserver (it would be easier for Codewalker to literally write a mapserver that talked directly to the game clients). It would be almost as difficult to make Paragon Chat assume those capabilities. And beyond all of that, there's the question of whether doing all of that subverts the original idea behind Paragon Chat, namely to be a community platform.
The way I see Paragon Chat, I don't see it recreating the game, or even wanting to. Codewalker could wake up tomorrow and decide that's what he wants to do, but I think what Paragon Chat is supposed to be is a platform where Codewalker can continue to experiment with how the game worked, and in the process we get a set of tools to make our own things happen in a community environment. Its most important feature is that its an open and (relatively) simple platform. So far, I've spent very little time figuring out how Paragon Chat works, and most of my time figuring out how XMPP and sleekxmpp work, because Paragon Chat is not difficult to work with. It opens the door for everyone from popmenu writers to bot writers to literal dice rollers to play around in the environment. I think adding more support for more things CoH used to be able to do is a good thing, but its best if its done in a way that continues to allow people to play with the environment more than literally replicate the entire game. If Codewalker wants to give players a literal Architect server to run missions in, I have to believe it would be far easier and faster if he implements an actual mapserver than tries to retrofit one into Paragon Chat.
I haven't even touched on the legal sticky wickets involved in going from a server that does nothing to a server that does something.
-
It slipped my mind this might not have been discussed in the thread, in a private question I asked Codewalker, he stated that PC caches its own costume, which is what made me realize that someone who was writing a bot and had not yet perfected costume stanzas could still make itself visible by cloning another character if it knew their has (which it would, because it would know everyone's hashes from their Presence stanzas).
I had expected something of the sort, really, but the tester in me tries to come up with all of the edge cases, heh. I actually think that serving the avatar's costume out of the cache is a rather clever bit of design.
So, as a tester, here's what I'd consider next - To eliminate any potential bugs related to duplicate hashing (or alternatively, narrow it down to that) I'd login as PC-A, then while PC-A is online, login PC-B. Once you can be sure that PC-B ought to have cached PC-A's costume hash (whether by time passed or by sniffing the stanza stream), logout PC-A, then login ArcanaBot with PC-A's costume hash. This should still cause PC-B to pull the costume from its cache but without a "hash collision" between the PC and ArcanaBot.
The worst case is that it all produces identical results, in which case you're no closer to a solution but you're definitively eliminated one of the edge cases. The best case is that now it works and you've narrowed it down to something in the "hash collision" being the culprit. Yes, you've previously proven that two instances of Paragon Chat can withstand a "hash collision" but that's not a definitive statement about ArcanaBot, given that ArcanaBot is what is being tested. (One reason being that both instances of PC-A and PC-A-prime have the costume in their cache while ArcanaBot does not have a cache at all.)
Most likely it will be much ado about nothing but hey, that's unit testing for you. You don't pick and choose based on expected results, you test everything, unless your goal is to only test the things that achieve the results you expected. ;-)
-
I have been reading through the xmpp theory and trying to get an overview.
I have a question, could powers be given a name linked to each character as I saw this in the basic theory and was trying to figure a way to activate powers.
You can login with multiple locations for instance as test@titan.com and then have your powers logged in as test@titan.com/power1 and the conversation your powers have activate the animation? I am trying to get my head around this it would seem the ability of XMPP to allow multiple concurrent logins would help if the powers were run from instanced servers?
If I am way off base a simple sorry won't do that is good :)
-
So, as a tester, here's what I'd consider next - To eliminate any potential bugs related to duplicate hashing (or alternatively, narrow it down to that) I'd login as PC-A, then while PC-A is online, login PC-B. Once you can be sure that PC-B ought to have cached PC-A's costume hash (whether by time passed or by sniffing the stanza stream), logout PC-A, then login ArcanaBot with PC-A's costume hash. This should still cause PC-B to pull the costume from its cache but without a "hash collision" between the PC and ArcanaBot.
My strong suspicion now is that something is subtly wrong with the way I am constructing Presence stanzas, and Codewalker already confirmed that in at least one respect I was probably treating something not quite right, namely conflating resources and nicks. Until now, it was difficult to see without using extremely verbose server logs what Paragon Chat was seeing coming from me (what you say and what they see are not exactly the same thing, because of XMPP semantics). -debug will make that a lot easier, but it will have to wait until I get home from work tonight. If I brought my bot code to work I'd never get anything done.
-
I have been reading through the xmpp theory and trying to get an overview.
I have a question, could powers be given a name linked to each character as I saw this in the basic theory and was trying to figure a way to activate powers.
You can login with multiple locations for instance as test@titan.com and then have your powers logged in as test@titan.com/power1 and the conversation your powers have activate the animation? I am trying to get my head around this it would seem the ability of XMPP to allow multiple concurrent logins would help if the powers were run from instanced servers?
If I am way off base a simple sorry won't do that is good :)
*If* I understand what you're saying, that's not really a good idea. First of all, the problem with activating powers is simply that power activation was a server-side thing, and therefore Codewalker has to program Paragon Chat to understand the fundamentals of how to do that. PC currently has no idea how to spawn power bolt projectiles, for example, so there's no way for any player or bot to do that.
I think what you are thinking is that the problem is literally activation; that the powers would work if we could only figure out how to tell the game to do that. But that's not the case.
In a sense, think of XMPP and more precisely Paragon Chat's implementation of XMPP as a translator. Its a CoH to Kind-of-English translator, but with a very limited dictionary. When something happens in your game client, that is sent to Paragon Chat. Paragon Chat converts the CoH network message into ParagonChat-ese, and that's sent to the XMPP server, which sends that to everyone else connected to that server. Their Paragon Chat programs convert the ParagonChat-ese into CoH network messages and send those to their game clients, and they see what you did. And vice versa.
Because ParagonChat-ese is XMPP and in Kind-of-English, its possible to inject messages into it without even running Paragon Chat, vis-a-vis bots. But the bots can only talk in ParagonChat-ese. ParagonChat-ese has no words for "Please Execute Energy Blast's Power Bolt." You can't say it, so you can't tell Paragon Chat to do it. To make that happen, Codewalker has to add those words into Paragon Chat's dictionary, and have the appropriate conversion into CoH network message protocol.
Now, *IF* Codewalker figures out how to add the appropriate words to Paragon Chat's XMPP message dictionary, *THEN* there are certain advantages bots might have. Separate from being able to do something in theory, for a player to do it you need some kind of control to make it happen. For example, consider power animations (http://www.cohtitan.com/forum/index.php/topic,11110.0.html). Right now, if you wanted your character to look like they were firing invisible Sniper Blasts, you'd need to play two animations: the Sniper Blast Windup, and then the Sniper Blast shot. You'd have to do that yourself, somehow, by typing in the two commands back to back with a pause between, or use macro buttons. Not convenient. Bots don't have that problem, because bots don't have game controls. A bot can simply be programmed to send the first message, pause one second, and then send the second one. Bots can be scripted, in ways players currently cannot be (although Codewalker is thinking about embedding LUA, a separate discussion topic). So *WHEN* Paragon Chat has more "power stuff" in its dictionary, bots might be able to do certain things better than players might be able to do, and that opens the door to bots potentially being player-helpers as well.
-
The plot thickens. I added resources to my XMPP connections, and reran with -debug. According to sleekxmpp, this is what I'm sending out as presence:
<presence from="arcanacoh@core3770" xml:lang="en"><pc xmlns="pc:presence" jid="arcanabot@core3770/ArcanaBot_meta" protocol="5" /><character xmlns="pc:character" origin="Magic" map="maps/City_Zones/City_01_01/City_01_01.txt" costume="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" name="ArcanaBot" class="Class_Blaster" /></presence>
This is what Paragon Chat records at the exact same moment:
xmpp DEBUG RECV: <presence lang="en" to="arcanacoh@core3770/Arcana" from="atlaspark_meta@conference.core3770/ArcanaBot_meta"><x xmlns="http://jabber.org/protocol/muc#user"><item role="participant" jid="arcanabot@core3770/ArcanaBot_meta" affiliation="none"/></x></presence>
xmpp DEBUG RECV: <presence lang="en" to="arcanacoh@core3770/Arcana" from="atlaspark@conference.core3770/ArcanaBot"><x xmlns="http://jabber.org/protocol/muc#user"><item role="participant" affiliation="none"/></x></presence>
xmpp DEBUG RECV: <message lang="en" type="groupchat" to="arcanacoh@core3770/Arcana" from="atlaspark_meta@conference.core3770/ArcanaBot_meta"><u p="146.4 0.3 -98.4" xmlns="pc:u" m="READY" v="0.0 0.0 0.0" o="0.0 -2.83 0.0"/></message>
xmpp DEBUG RECV: <message lang="en" type="groupchat" to="arcanacoh@core3770/Arcana" from="atlaspark@conference.core3770/ArcanaBot"><body>Hello, participant Arcana</body></message>
Somehow, my pc namespace substanzas are not making it to Paragon Chat, so Paragon Chat doesn't know I exist. It must have something to do with Presence stanza handling I'm unaware of yet. Time to RTFM.
-
Okay, I see two different things going on above.
1. You're sending presence to the server, I assume right after login, with the pc:presence and pc:character namespaced attributes. This is good. However, be aware that this presence gets sent only to people who have a presence subscription -- i.e. friends on your roster.
2. I see incoming presence from the two rooms the bot joined. I don't see outgoing directed presence on the bot side, so I assume your xmpp library is handling that for you. That's the important presence, though.
In XMPP MUC, the directed presence that you send to the room in order to join it is the presence that other members of the room receive. They do not get the presence that you sent to the server at the start. It sounds weird at first, but makes sense when you think about it. Hopefully your library has the ability for you to add custom elements to the presence it's sending to the room in order to join it.
-
In XMPP MUC, the directed presence that you send to the room in order to join it is the presence that other members of the room receive.
Feh**. I *originally* sent presence to the room, but when I ran into a separate problem with server initiation, I accidentally deleted that code. Then I realized I did that and put it back. Except in the round trip, I put it back wrong: I accidentally sent presence to atlastpark_meta@core3770. When I correctly send presence to atlaspark_meta@conference.core3770, it works. Thanks for pointing in that direction. Also, I had no idea what the two Presence messages were for, I just copied sample code. That makes a lot of sense now.
Incidentally, why do you apparently send presence to atlaspark_meta@ConferenceServer.FQDN/Resource when sending to atlaspark_meta@ConferenceServer.FQDN seems to work just as well?
So, I'm now visible:
(https://farm1.staticflickr.com/393/19680758735_a06017ec7c_o.jpg)
I can start cleaning things up. At the moment, a lot of stuff is just plain hard coded, just to get things working. The bot only logs into Atlas Park, has hardcoded login information, has a hard coded costume hash, doesn't really do anything except say hi, is currently two pixels embedded into the sidewalk, but it does work. Proof of concept. But for that one stupid typo, I've been banging on this for a couple days.
I need to add local chat support, multiuser chat support (for local chat), and some better means of sending the bot commands besides hard coded functions. Maybe I'll have it accept JSON just to mess with FFM's head. And once the costume algorithm settles down, some other way of setting appearance not involving cloning experiments.
Insert appropriate "squee" here.
** See also:
(https://beezkneezblogblog.files.wordpress.com/2014/08/hulk-smash-loki-animated-gif-313.gif)
-
^ I have two things to say.
1. Congratulations on getting your bot visible! :)
2. Oi!! No messing with my head! :P
-
Well done, @Arcana!
I am taking a longer way around. I am working on a paragon chat plugin to ErrBot, so I'm having to learn those ins and outs as well as the general business of Python data structures.
-
Incidentally, why do you apparently send presence to atlaspark_meta@ConferenceServer.FQDN/Resource when sending to atlaspark_meta@ConferenceServer.FQDN seems to work just as well?
- Standard compliance.
In order to participate in the discussions held in a multi-user chat room, a user MUST first become an occupant by entering the room. In the old groupchat 1.0 protocol, this was done by sending presence with no 'type' attribute to <room@service/nick>, where "room" is the room ID, "service" is the hostname of the chat service, and "nick" is the user's desired nickname within the room
If the user does not specify a room nickname (note the bare JID on the 'from' address in the following example), the service MUST return a <jid-malformed/> error:
(the next section explains that MUC builds on the old groupchat protocol and adds an extra namespaced 'x' element to signal MUC capabilities).
Openfire is apparently being more lenient than is allowed by the spec in generating a nickname for you. Either that or slickxmpp is silently fixing it before sending.
- The resource used when joining a room is used as the 'nickname' and is what shows up in the member list in clients like Pidgin, as well as the name the chat messages are attributed to. Paragon Chat tries to pick a suitable nickname for the context of the room -- if joining a global chat channel it will use your global name, if the room is for zone broadcast it will use your character name.
It also uses the room nickname as a fallback if it can't resolve the sender back to a character name or global handle. This can happen when non-Paragon Chat clients are in an anonymous room that doesn't allow members to see the full JID. We are not using any anonymous rooms on the Titan server, but the logic is there for completeness.
-
SleekXMPP treats the resource as an option and distinguishes between a "bare" JID and a "full" JID. I'd wager that it's OpenFire that is assigning the "abc123" style of resource to clients that supply a "bare" JID.
-
I figure it won't be long before you ask me for a notification via XMPP when a player is 'clicked'. I'll start thinking about how to do it efficiently, probably by including some sort of indication in the presence that this player is interested in receiving events about it.
-
SleekXMPP treats the resource as an option and distinguishes between a "bare" JID and a "full" JID. I'd wager that it's OpenFire that is assigning the "abc123" style of resource to clients that supply a "bare" JID.
Yup. But to be honest, because I was just trying to get off the ground, there's a lot of the XMPP implementation I shortcut around and am now going to implement for completeness, so I wasn't even running into boundjid/boundjid.bare issues because I was hardcoding jid.
I'm human, though (that's the story my parents keep telling), so I actually spent last night getting walk cycles to work. Because implementing service discovery protocol is fun and all, but watching your bot moonwalk (because of yaw cosine sign reversal) will keep you awake till 2am.
I have no specific plans to reimplement the sequencer (yet) so that too is being cheated in for now. Not sure I'm as ambitious as Codewalker in that regard, and I'm still going to have to do something with maps at some point, unless Codewalker beats me to version 1.3 first.
Also, bots don't have to obey gravity, and can enter any movement cycle that exists, so:
(https://farm1.staticflickr.com/396/19507123348_5c48702bd6_o.jpg)
If you can construct the correct messages in the right order, bots can fly.
This is currently my bot's "brain" that is doing nothing more than obeying my keystrokes:
jog_moves = ['RUNPOST', 'RUNPRE', 'RUN_BACK_PRE', 'RUN_LEFT_PRE', 'RUN_RIGHT_PRE']
fly_moves = ['A_FLYPOSE1_POST', 'A_FLYPOSE1_PRE', 'A_FLYPOSE1_PRE','A_FLYPOSE1_PRE','A_FLYPOSE1_PRE']
pc_moves = []
pc_moves.append(jog_moves)
pc_moves.append(fly_moves)
class Pcbot_brain:
def __init__(self):
self.oldkeys = sets.Set()
self.newkeys = sets.Set()
# newkeys has only recently pressed keys, oldkeys has all currently depressed keys
def processKeyState(self, kt):
#print "Debug: oldkeys: " + str(self.oldkeys) + " new keystate: " + str(kt.keystate)
self.newkeys = kt.keystate.difference(self.oldkeys)
if len(self.newkeys) > 0:
print "Debug: newkeys: " + str(self.newkeys)
self.oldkeys = kt.keystate.copy()
#print "Debug: new oldkeys: " + str(self.oldkeys)
return
def take_action(self, pcbot):
move_speed = 0.3
turn_speed = 0.05
move_mode = (303 in self.oldkeys) # Right-Shift
if 112 in self.newkeys: # P
pcbot.send_pcPresence()
print "Debug: sending pcPresence based on keypress"
if 119 in self.oldkeys: # w
pcbot.position[2] = pcbot.position[2] + move_speed * math.cos(pcbot.orientation[1])
pcbot.position[0] = pcbot.position[0] + move_speed * math.sin(pcbot.orientation[1])
if pcbot.animation != pc_moves[move_mode][1]:
pcbot.animation = pc_moves[move_mode][1]
pcbot.send_pcu()
elif pcbot.animation == pc_moves[move_mode][1]:
pcbot.animation = pc_moves[move_mode][0]
pcbot.send_pcu()
if 97 in self.oldkeys: # a
pcbot.position[2] = pcbot.position[2] + move_speed * math.sin(pcbot.orientation[1])
pcbot.position[0] = pcbot.position[0] - move_speed * math.cos(pcbot.orientation[1])
if pcbot.animation != pc_moves[move_mode][3]:
pcbot.animation = pc_moves[move_mode][3]
pcbot.send_pcu()
elif pcbot.animation == pc_moves[move_mode][3]:
pcbot.animation = pc_moves[move_mode][0]
pcbot.send_pcu()
if 115 in self.oldkeys: # s
pcbot.position[2] = pcbot.position[2] - move_speed * math.cos(pcbot.orientation[1])
pcbot.position[0] = pcbot.position[0] - move_speed * math.sin(pcbot.orientation[1])
if pcbot.animation != pc_moves[move_mode][2]:
pcbot.animation = pc_moves[move_mode][2]
pcbot.send_pcu()
elif pcbot.animation == pc_moves[move_mode][2]:
pcbot.animation = pc_moves[move_mode][0]
pcbot.send_pcu()
if 100 in self.oldkeys: # d
pcbot.position[2] = pcbot.position[2] - move_speed * math.sin(pcbot.orientation[1])
pcbot.position[0] = pcbot.position[0] + move_speed * math.cos(pcbot.orientation[1])
if pcbot.animation != pc_moves[move_mode][4]:
pcbot.animation = pc_moves[move_mode][4]
pcbot.send_pcu()
elif pcbot.animation == pc_moves[move_mode][4]:
pcbot.animation = pc_moves[move_mode][0]
pcbot.send_pcu()
if 113 in self.oldkeys: # q
pcbot.orientation[1] = pcbot.orientation[1] - 0.1
pcbot.send_pcu()
if 101 in self.oldkeys: # e
pcbot.orientation[1] = pcbot.orientation[1] + 0.1
pcbot.send_pcu()
if 32 in self.oldkeys: # SPACE
pcbot.position[1] = pcbot.position[1] + move_speed
pcbot.send_pcu()
if 120 in self.oldkeys: # x
pcbot.position[1] = pcbot.position[1] - move_speed
pcbot.send_pcu()
return
It doesn't batch updates, it doesn't do prediction, and my send_pcu code doesn't quite correctly send the initial sequence once (its supposed to send MOVE_PRE once and then stop sending but I goofed in a simple to correct way, but I'm rethinking the entire process) so it studders if you hold down movement keys because it keeps re-entering _PRE. But, you hold down the shift key, and she flies. For a few feet at a time, anyway. Then you release shift and she starts walking in the middle of the air.
Also, because of the way Paragon Chat works, there's a lot of lag. I'm not sure if that lag is because of how Paragon Chat works, because of Openfire latency, or what, yet. But that lag places limits on how instantly interactive a bot can be. But at least I am in striking distance of the original goal for the bot project, which was to make a bot that can shadow box and fall down. The lag I'm seeing makes the prospect of implementing react-moves almost comical, but we'll see what happens.
There's a difference between knowing, and seeing. There's a lot about the animation system I know. But its different when you have to pull every string, and you see what failing to pull a string or pulling the wrong string does. I didn't think I needed to do this, but now I'm thinking I should implement inertia. Movement just doesn't look right without it.
-
I figure it won't be long before you ask me for a notification via XMPP when a player is 'clicked'. I'll start thinking about how to do it efficiently, probably by including some sort of indication in the presence that this player is interested in receiving events about it.
Now that you mention it, I was thinking of asking for a "t=" target attribute for <u />, such that <u /> messages with t payload are unicast to the targeted entity rather than multicast throughout the meta room. You wouldn't even need to put a target in the attribute, because if you get a t= message you're guaranteed to be the target. If you later decide there's a benefit to everyone knowing who is targeting who, you could multicast that message with a target specified. So if you get a message with t= but no target (t="") presume the target is you. If you get a message with t=TARGET, presume this message means sender is targeting TARGET, which may or may not be you. Private rooms could be flagged to allow broadcast targeting for special cases, while general rooms normally only unicast target.
Just thinking out loud.
-
Shiny!
I was wondering about whether bots would be subject to gravity or not.
Now, you realize, that right now, with no further modifications at all, you could just put your bot on a rail where she flies over Atlas every five minutes and you'd be freaking everyone out, lol.
Congrats on your accomplishment. :)
-
I was wondering about whether bots would be subject to gravity or not.
So long as you keep the Y velocity equal to 0, they should be able to ignore gravity. If you have any Y velocity at all, PC will apply gravitational acceleration to its prediction model and cause the bot to appear to fall, then it'll rubber-band back where it was on the next position update.
When 1.1 comes around with travel modes, I'll add an extra attribute to updates to indicate flight mode and disable that.
-
Now, you realize, that right now, with no further modifications at all, you could just put your bot on a rail where she flies over Atlas every five minutes and you'd be freaking everyone out, lol.
I wouldn't do that to Codewalker. At least, not if he implements a VFX spawn primitive. If not, and I get bored because I can't spawn Power Blast projectiles, who knows *what* I might do. <sarcasm xmlns="Arcana:sarcasm" type="whistle" mode="nonchalant" dir="upward" />
Actually, Codewalker knows I would never do that to him because that would be mean. I *would* load two bots into Paragon Chat that started fighting each other in Pocket D, because *that* would be mean but awesome, and there's the EULA to think about.
-
Actually, Codewalker knows I would never do that to him because that would be mean. I *would* load two bots into Paragon Chat that started fighting each other in Pocket D, because *that* would be mean but awesome, and there's the EULA to think about.
Only if you put them in the monkey fight club ring. I think that would be sufficiently awesome.
-
So a question - in the character creator all power animations are there, where are those stored?
-
powers.bin, sequencers.bin, fxinfo.bin, particles.bin, and a few others... same as for regular power execution.
The character creator screen has a VERY hacked version of the sequencer and Fx spawning that would be sent from the server. I poked at it a while back but so many details are hardcoded that it's useless for anything except the power customization screen. It would be more productive to create a new power execution engine from scratch than to try to adapt that.
It's not like we don't have people who understand in complete detail exactly how to generate the visuals for any given power activation from the data. That's not an issue. It's implementing it where others can see it and doing something useful with that takes some work.
-
So a question - in the character creator all power animations are there, where are those stored?
I'm not sure I understand the question. They are stored in the client, in various places in pigg files depending on the specific technical question you're asking. The power database contains references to power animation files, one (basically) for each power. Each of those files contains the sequence bits the client should trigger at specific moments during the power activation. The animation sequence engine references a sequence database, also in a pigg file in the client, that tells the game client which animation "MOV" to play when a particular set of animation bits are activated, and also what to do by default after an animation plays (its called a sequence engine because when an animation is triggered, the sequence engine plays (or can play) a sequence of animations after that initial one without any further guidance or directive from the game servers). Each animation MOV references an animation file, also contained in the pigg files, that describes precisely how to move the individual skeleton elements of an entity, frame by frame, to perform a fluid atomic motion.
-
Only if you put them in the monkey fight club ring. I think that would be sufficiently awesome.
I was thinking of spawning them looking like NPC patrons chatting, and when they detected enough players announcing presence in the zone, they would start arguing, and eventually start punching and kicking each other.
But there's a lot of stuff I would want to do before unleashing these guys on the live server. For example, I noticed this morning that if the bot enters the zone second, Paragon Chat sees it, but when the bot enters the zone first Paragon Chat doesn't always see it. Must be another presence thing. Also, without inertia, gravity, and ground detection, knockdown and knockback isn't feasible to simulate. And I need to match PC's prediction with velocity vectors, or things will look jumpy on the PC side.
Also, dammit I really can't see any way around implementing a sequence engine if I want my animations to look right. Have to dig up my root calculator and adapt that. Not exactly hard, just ugh.
-
Also, dammit I really can't see any way around implementing a sequence engine if I want my animations to look right. Have to dig up my root calculator and adapt that. Not exactly hard, just ugh.
Heh. I suspect that your "ugh" is most people's idea of "exactly hard".
Question for @Codewalker - Okay, two questions, or maybe one question and one feature request -
If I'm already planning to use an SQL database for various sorts of persistent data storage, and I see some mileage in having access to the PC character/costume database, is there any downside to just tossing my bot's tables into ParagonChat.db? Aside from the obvious "we might decide to delete the whole .db file during a software update" kind of downside, that is.
Given that pretty much all of the other costume info is in there, would it affect the database adversely to save the costume's hash in there as well? While that's a blatant bid on my part to avoid having to learn yet another new thing,it's something you have to compute anyway so having it in there where a bot can just "grab and go" would be extremely handy.
-
Heh. I suspect that your "ugh" is most people's idea of "exactly hard".
Question for @Codewalker - Okay, two questions, or maybe one question and one feature request -
If I'm already planning to use an SQL database for various sorts of persistent data storage, and I see some mileage in having access to the PC character/costume database, is there any downside to just tossing my bot's tables into ParagonChat.db? Aside from the obvious "we might decide to delete the whole .db file during a software update" kind of downside, that is.
Given that pretty much all of the other costume info is in there, would it affect the database adversely to save the costume's hash in there as well? While that's a blatant bid on my part to avoid having to learn yet another new thing,it's something you have to compute anyway so having it in there where a bot can just "grab and go" would be extremely handy.
I believe Codewalker would likely say what any experienced programmer would say.
(https://images.weserv.nl/?url=hellogiggles.com%2Fwp-content%2Fuploads%2F2014%2F09%2F26%2Femma-stone-no-no-no-no-no-gif-bgod.gif)
Paragon Chat's database is a sqlite database. There's bindings for sqlite all over the place, and if you're going to use Paragon Chat's databases in any capacity you're going to have to learn that either way. But its far safer to store your own data in your own sqlite database file, and just use Paragon Chat's database as a read only information resource. In fact, you have to use a command line switch (http://www.cohtitan.com/forum/index.php/topic,11088.0.html) just to prevent sqlite from locking the entire thing in the first place.
Even in shared mode, having two separate programs writing to the same database program unaware of each other is generally a no-no. Having two separate programs writing to the same database one of which is being patched every nine seconds and could conceivably change schema one day is a definite no-no.
The learning curve is exactly the same either way (in terms of technicals), but one is safe and the other is banana hat bonkers.
As to storing the costume hash? I see where you're going there: you're lazy and don't want to implement the costume hash algorithm. I like the way you think. However, storing it in the Paragon Chat database seems like the wrong way to do that for a number of reasons, but in particular as it will only have hashes for costumes you literally create in the character creator. You need to think bigger, and lazier. I was thinking that maybe Paragon Chat could be convinced to give us the costume hashes, since it knows how to make them and all, and also that is implementation specific and could change over time. You should ask Codewalker to implement an Iq message where a bot can send to a Paragon Chat client a costume, and get the official hash returned**. Hypothetically speaking, then, the bot could ask any Paragon Chat client to help it compute hashes and then store them for any costume. Let me know what he says about that. I was going to get around to implementing hashing later, but maybe he'll take pity on you. I think he would just tell me to get off my ass and implement the hash.
** Tell Codewalker you think the costume exchange reminds you of ARP, and you want him to implement an analogous RARP. We technical types love to talk protocols.
-
but one is safe and the other is banana hat bonkers.
Well, I'll admit that I expect him to probably say, "Hell, no, just export the database to your own database", but curiosity and all that. Plus it's one less thing to have to explain in the eventual documentation of the bot.
You need to think bigger, and lazier. I was thinking that maybe Paragon Chat could be convinced to give us the costume hashes, since it knows how to make them and all, and also that is implementation specific and could change over time. You should ask Codewalker to implement an Iq message where a bot can send to a Paragon Chat client a costume, and get the official hash returned**.
Yeah, I thought that big already and when I said it to myself it came out sounding a lot like, "Hey, Codewalker, why not give us a Paragon Chat RPC API in your abundant free time?" and I decided to ask for something that sounded like it would take a few minutes to implement. :-p
Though I like the ARP/RARP analogy!
The plain fact is that the "ARP" would only be useful if the bot is making up new costumes on the fly. For most situations that the bot is going to change its costume, it should already have the costume defined and know the hash ahead of time. Basically, for any particular costume, the bot is already going to have a record identical what's already in the PC costume/costumeparts tables for any costume it might wear, though it would also have the hash already computed ahead of time, to save compute cycles if for no other reason.
-
Yeah, I thought that big already and when I said it to myself it came out sounding a lot like, "Hey, Codewalker, why not give us a Paragon Chat RPC API in your abundant free time?"
That's the spirit.
Or, you know, if you really want to recreate that City of Heroes developer nag feeling, ask Leandro to implement it instead.
-
I've been lurking. So far what y'all have come up with on such short notice is pretty amazing, nor going to lie. I have been day dreaming about the possibilities.
2 things that come to mind (and I saw this was touched on a little earlier at some point on regards to the ski timer). I was trying to watch the debug log but in all the jumble I didn't see anything- I know that the maps have different entities and triggers -- teleports, spawn points, gate 'zones' on the ski slopes, etc does the trigger zone information get passed to PC/other players or is it ignored? Any chance it could be if it isn't? Or would there be more problems to solves before that might be possible?
The other thing is thumbtacks. I forgot to check if these are working, so pardon me on this... Would this be shared with pc and possibly set by PC at some point in the future?
I hope to be able to dive into this more this week end. Keep up the awesome.!
//I hate typing on tablets...
-
That's the spirit.
Or, you know, if you really want to recreate that City of Heroes developer nag feeling, ask Leandro to implement it instead.
*laugh* This forum really needs a "like" button.
-
I've been lurking. So far what y'all have come up with on such short notice is pretty amazing, nor going to lie. I have been day dreaming about the possibilities.
2 things that come to mind (and I saw this was touched on a little earlier at some point on regards to the ski timer). I was trying to watch the debug log but in all the jumble I didn't see anything- I know that the maps have different entities and triggers -- teleports, spawn points, gate 'zones' on the ski slopes, etc does the trigger zone information get passed to PC/other players or is it ignored? Any chance it could be if it isn't? Or would there be more problems to solves before that might be possible?
Except for players and doors, most of the rest of the world in Paragon Chat is "inert" - it was the server that handled all of that, and we don't have mapservers. But, there are things we can do that are not what the servers were capable of doing, but could in some cases creatively replace them. For example, if you were looking at the debug console, one of the things you saw (whether you understood it or not) was everyone constantly announcing their positions in space, so everyone else's game client could display them properly. Well, if you know where people are, and you know where things are, there are things you could do. For example, and we discussed this earlier, suppose you wanted to allow players to run a race course, like the Ski slopes. Until Codewalker makes those elements of the game function somehow, you can't actually use the active elements of the race course. But you could monitor the location of players, and notify them when they crossed the starting line if you personally know where it is. That's just geometric programming. You could also detect when they crossed a finish line, and you could time those two events and announce it to players. Lag would be potentially problematic, and the players don't get visual feedback beyond chat messages, but its something.
Out of convention, Codewalker is calling Paragon Chat's version 0.97-something-whatever Beta, but the truth is this is all really 0.03 early Alpha. We're just scratching the surface of what's possible. And some of what's going to be possible will be seeing how far Codewalker is willing and able to push the platform. And some of what's going to be possible will be seeing how far other people are willing and able to push the boundaries of what the system is capable of doing with the right extra add-ons. That's sort of where I've decided to play in all of this.
Standing between those that are trying to build it and those trying to use it is, actually, not entirely unfamiliar territory for me.
On the subject of costume hashes, I decided to break down and start thinking about it, so I can bang out the code when I get home. However, I did notice some ambiguities in the spec. For example, when the costume file data has quotation marks surrounding the name of the element, I am guessing those are just for the file format and they get stripped when the costume file is uploaded into City of Heroes. What I don't know is whether the quotation marks are escaped and sent in costume replies, and whether they are preserved and used within the hash computation. Second, for the 3D scales byte order is not explicitly specified. It actually says the data is packed into a 32-bit integer and then converted back into hex, but the intervening int representation poses an odd and avoidable ambiguity. Is it sufficient for that data to be directly converted to four hex strings, such that if the tuple is (X, Y, Z) what gets hashed is 00zzyyxx, where xx is the hex representation (minus the 0x) of X if X is positive, and abs(X)+128 if X is negative.
The other thing is thumbtacks. I forgot to check if these are working, so pardon me on this... Would this be shared with pc and possibly set by PC at some point in the future?
I suspect the only things that Paragon Chat will actually send to XMPP are things other players need to know. No one needs to know where your thumbtacks are located, any more than other players need to know what your power tray looks like. But as a separate issue, I don't think those have been implemented yet on any level.
-
For example, when the costume file data has quotation marks surrounding the name of the element, I am guessing those are just for the file format and they get stripped when the costume file is uploaded into City of Heroes.
Yes, those are just part of the generic text format the COH uses for everything. The quote marks are not part of the name and are parsed out by the client.
Second, for the 3D scales byte order is not explicitly specified. It actually says the data is packed into a 32-bit integer and then converted back into hex, but the intervening int representation poses an odd and avoidable ambiguity.
Byte order shouldn't matter, as it's the hex representation of the number regardless of platform, not the representation of how that number is stored in memory. If it helps, assume big endian.
Or you could just call it "00" followed by three 8-bit signed integers, that would work, too.
-
Also, dammit I really can't see any way around implementing a sequence engine if I want my animations to look right. Have to dig up my root calculator and adapt that. Not exactly hard, just ugh.
After thinking about it for a while and conducting experiments, there is a possibility that involves not implementing an entire sequence engine, which I think is overkill for a bot. You could make a skeleton one that only has the moves you're interested in, and that would also be simple enough for people to tinker with.
This is extremely unoptimized, but I was fiddling with an idea like this:
jog_moves = [['RUNPRE', 'RUN_BACK_PRE', 'RUN_LEFT_PRE', 'RUN_RIGHT_PRE'],
['pcRUNCYCLE','pcRUN_BACK_CYCLE','pcRUN_LEFT_CYCLE','pcRUN_RUGHT_CYCLE'],
['RUNPOST', 'RUNPOST', 'RUNPOST', 'RUNPOST']]
fly_moves = [['A_FLYPOSE1_PRE', 'A_FLYPOSE1_PRE','A_FLYPOSE1_PRE','A_FLYPOSE1_PRE'],
['pcA_FLYPOSE1_CYCLE','pcA_FLYPOSE1_CYCLE','pcA_FLYPOSE1_CYCLE','pcA_FLYPOSE1_CYCLE'],
['A_FLYPOSE1_POST', 'A_FLYPOSE1_POST','A_FLYPOSE1_POST','A_FLYPOSE1_POST']]
walk_moves = [['PLAYER_WALK_FORWARD','PLAYER_WALK_BACKWARD','WALK_LEFT_PRE','WALK_RIGHT_PRE'],
['pcPLAYER_WALK_FORWARD_CYCLE','pcPLAYER_WALK_BACKWARD_CYCLE','pcWALK_LEFT_PRE_CYCLE','pcWALK_RIGHT_PRE_CYCLE'],
['PLAYER_WALK_POST','PLAYER_WALK_POST','PLAYER_WALK_POST','PLAYER_WALK_POST']]
pc_moves = []
pc_moves.append(jog_moves)
pc_moves.append(fly_moves)
pc_moves.append(walk_moves)
So I create an array of moves. Each array element is itself a three element array which has intro moves, cycle moves, and exit moves. These happen to handle movement and there's four movement directions (that the animation system honors): forward, left, right, and back, so there's four entries per. The idea is that when you hit a movement key, the system looks up what MOV to send based on your current movement mode (run, fly, walk) and the direction you're moving (basically, wasd keys). It then sets that animation and sends it. On the next clock tick, if the current animation MOV state is any of the Prefix ones (by searching for them), the system automatically sets the next in the sequence. If no keys are being pushed, the system checks to see if we are currently in any of the CYCLE states, and if so it sets to the POST MOV. And one gimmick, when I send <u /> if the current animation state begins with "pc" I don't send that MOV. Basically, I could have called them pcANYTHING but I just chose to call those moves what they actually are for (my) sanity sake.
The general idea is to make a state engine that is a micro version of the real sequencer with animation sequences with a single entry point and an exit point and a criteria for when to stay in the middle of the cycle and when to exit. Then the bot writer can decide which sequences they care about and hard code them into the bot. If they want to have the entire sequence library at their disposal, well, there is that dump of the sequence file they could choose to import and parse. But this simplifies things greatly in theory, although I need to think a bit more about how to deal with non-movement sequences: the movement ones were special cases I wanted to fool around with first.
You know, I'm beginning to think this might have actually taken some serious time for Codewalker to implement.
Anyway, here's some test footage of my bot running, walking, and sort of flying. Also, flying backwards. Unlike game clients, bots don't have to make sense.
https://www.youtube.com/watch?v=Vd8FPTYcktw
I got half way through writing a costume hasher, then got bored. Yawn. I should have that finished tomorrow, and then I'm going to take a swing at maps. Just enough to get by, not enough to re-render the entire environment. Floor and wall collision mostly. I still want to kick something, but I can't do simulated knock until I find the ground, and without that what's the point.
-
Have you considered looking into generating nav meshes for the maps? In concept its much more realiayuc than hardcoded or breadcrumb style nav, but honestly its way beyond me at this point, but ive been thinking about ots off and on for years. I have seen it implemented for other game bots... gonna stop rambling here...
-
Have you considered looking into generating nav meshes for the maps?
The client has some remnants of the nav mesh system that Paragon used (called beacons, with the process of generating them called beaconizer). I don't know if CW is planning to reimplement the same format in order to preserve the bits in the client that are compatible with it (mostly just debugging tools) or go in a different direction. I'd bring it up to him before trying to implement anything, to make sure the effort isn't duplicated or gone to waste.
-
So what if you had someone who was working on rendering all of the City of Heroes maps into like a new game engine or something would that help Arcana?
I am kind of thinking perhaps meshing this effort with Irish Girls.....................
-
So what if you had someone who was working on rendering all of the City of Heroes maps into like a new game engine or something would that help Arcana?
I am kind of thinking perhaps meshing this effort with Irish Girls.....................
On the one hand, that's massive overkill for what I need. On the other hand, its also not what I need at all; what I need is less "how to read the map?" and more "what should I do with the map?" This is not as straight forward as it sounds, as nothing really is. For example, most players think of performing an animation as "perform animation" but to get those movements looking correctly above I needed to send the movement prefix animation, let the game fall naturally into the cycling animation, never send the pre again, check to see if the movement should no longer be happening, check to see if it is actually still happening, and if so send the prologue animation and then never send that again and let the game fall naturally out of it. It basically requires a state engine, to send something when a key is pressed and but only once, to send another thing when the key is released but only once and only if you're in another state. And even then you learn things. For example, there's no such thing as walk backwards. Interestingly, the walk backwards animation actually pivots the character 180 degrees. Notice in the video above there is a way to run backwards and there is even sort of a way to fly backwards, but walk backwards does something unexpected.
Well, reading the map is one thing. Then there's the question of how you honor the map. Its not just a matter of "don't be below the ground level." A naive approach would simply to be to move in any direction you want, and then clamp your y axis minimum to the map minimum. But the map is three dimensional: walking around steel canyon would cause you to sometimes warp underground. And if you move towards a building, I don't think you want to instantly teleport to the roof (or maybe you do, but probably not by default). You need some logic for deciding what moves are legal, and which have to be simply truncated due to collision with geometry. And you could try to figure out what the game itself did, or you could decide the bot should have different rules. Should it automatically hop fences, like the NPCs did? Should it walk up walls (I think that will gimbal lock the character, but we'll see).
The rules are really the thing. The map itself is an inconvenience.
-
I guess that's what I am saying - does she have those rules already in place or is what she is doing strictly graphical in nature at this point?
I know you don't need the game engine at this point but she is recreating Paragon mapping as well. I guess I don't know if those rules would port over from that engine or not.
-
Have you considered looking into generating nav meshes for the maps? In concept its much more realiayuc than hardcoded or breadcrumb style nav, but honestly its way beyond me at this point, but ive been thinking about ots off and on for years. I have seen it implemented for other game bots... gonna stop rambling here...
Generating? No. At least, not at the moment. Supporting? Maybe. Which is to say, I've been thinking about a way for the bot writers themselves to create their own "safe zone navigation areas" for their bots, within which the bots could have more autonomy in motion, or goal-seeking behavior. In fact, it occurred to me when Codewalker mentioned monkey fight club that it would be an interesting idea if the bots could have a preprogrammed pseudo-fence within which they were allowed to maneuver pseudo-randomly while they fought.
But autogenerating navigation meshes? I'll leave that as an exercise for the reader.
-
I guess that's what I am saying - does she have those rules already in place or is what she is doing strictly graphical in nature at this point?
I don't have first hand knowledge, but I thought her efforts were strictly modeling in nature. There's no algorithmic nature to her project, nor would there be any logical reason for her to construct one (at the moment).
-
The client has some remnants of the nav mesh system that Paragon used (called beacons, with the process of generating them called beaconizer). I don't know if CW is planning to reimplement the same format in order to preserve the bits in the client that are compatible with it (mostly just debugging tools) or go in a different direction. I'd bring it up to him before trying to implement anything, to make sure the effort isn't duplicated or gone to waste.
My concern with the beaconizers is that they may have been subtly optimized for what the devs wanted NPCs to do, pathing-wise, which might be contrary to what some bot writers would want to do, particularly as bot writers might explicitly want to do "unsafe" things relative to conventional NPC spawns. But honestly I have no real idea until I get there, and I'm a long ways away from there. I generally tell Codewalker whenever I am thinking about an area outside of what Paragon Chat has already implemented, usually by asking him to increase the area within which Paragon Chat is implemented.
-
(I think that will gimbal lock the character, but we'll see).
I would recommend not using PYR for your internal state. Paragon Chat itself uses a 4x4 matrix to represent the state for each entity and only normalizes the orientation to PYR in order to save in the database and transmit over the wire. The protocol to talk to the client itself uses quaternions to represent orientation, but I didn't duplicate that in XMPP because I didn't want to make people's head explode when they tried to write something to speak the XMPP version.
PYR works well enough for gross orientation sync, and since no transformations are actually taking place on it in that form, gimbal lock isn't an issue.
Eventually I'd like for the scripting interface (or even some sort of external API) to make this a lot easier, so that bots can plug in to Paragon Chat and take advantage of its geometry representation, animation sequencer, and collision engine that is reasonably close to the client's. That would avoid having to re-invent the wheel (again). It's just a matter of time, but for now the hard way is the only way.
-
I would recommend not using PYR for your internal state. Paragon Chat itself uses a 4x4 matrix to represent the state for each entity and only normalizes the orientation to PYR in order to save in the database and transmit over the wire. The protocol to talk to the client itself uses quaternions to represent orientation, but I didn't duplicate that in XMPP because I didn't want to make people's head explode when they tried to write something to speak the XMPP version.
PYR works well enough for gross orientation sync, and since no transformations are actually taking place on it in that form, gimbal lock isn't an issue.
At the moment I use PYR for the bot state just because its simple, and because I currently have no way to tell the bot to face upward anyway. As soon as I do, yeah, I would need to switch to a non-PYR (almost certainly quaternion) model to avoid problems. The bot is doing an enormous amount of cheating to keep things simple at present, specifically to avoid the very problem I recommended Joshex avoid.
Also, if I went straight to numpy's matrix math for transformations, that would obscure some of the mechanics of what was happening behind less than transparent math, and Linear Algebra was 29 years ago even for me. I hope at some point normal people can read the code and figure out what it is doing. Although I was already thinking that computing rotational paths between two random defined orientations was already going to probably require (or rather be easier with) quaternions. For when I can actually *face* random orientations outside the xz plane.
Eventually I'd like for the scripting interface (or even some sort of external API) to make this a lot easier, so that bots can plug in to Paragon Chat and take advantage of its geometry representation, animation sequencer, and collision engine that is reasonably close to the client's. That would avoid having to re-invent the wheel (again). It's just a matter of time, but for now the hard way is the only way.
Which is partially why I'm trying to reinvent as little wheel as I can get away with, while still being functional (and why I'm often looking for ugly cheats rather than complete solutions in some cases). Ordinarily, I would have just jumped straight to the finish line, because every intermediate step between here and there is somewhat redundant when it comes to implementing solutions. But that seems sideways to the point here.
-
So I might as well take a quick look at maps before I head to work, give me some time to think about those. It shouldn't be too hard to get just the information I want from...
...hmm...
Well. Of course it would have to be. I see. I'm going to have to deal with all of this, aren't I.
Patch Notes: July 15, 2015
Ghostbot version 0.2a
New features:
* ArcanaBot has been renamed Ghostbot
* GhostBot can pass through walls, ground surfaces, objects, and NPCs. This is not a bug, its a feature.
Known Issues:
* Enabling gravity can have unintended consequences
-
So I might as well take a quick look at maps before I head to work, give me some time to think about those. It shouldn't be too hard to get just the information I want from...
...hmm...
Well. Of course it would have to be. I see. I'm going to have to deal with all of this, aren't I.
Yes, it's a pain in the butt.
You know, I'm beginning to think this might have actually taken some serious time for Codewalker to implement.
I mentioned the XMPP phase of the project started around last November/December, but built on some existing work. What do you think I've been doing for the last couple years? :o
-
Yes, it's a pain in the butt.
I'm sure Codewalker realizes this, but for the benefit of a few people who are recommending computer graphics books to me, I know a little more about this than I'm often letting on, just because I'm trying to approach this from the perspective of someone who might want to write a bot, and has to learn everything from scratch. Also, I'm busy and lazy, and so I'm playing this completely by ear and detecting problems by running into them face first. And I think it might be helpful if I discuss this (mostly but not always talking to myself) at a slower pace than normal.
In this case, my idea for somehow extracting just what I wanted from the map files was a bit naive. I can always hope, but the problem with that idea is the notion that there is anything remotely called "the ground" I could pull from the map files, or anything like that. Map files are organized by object (and probably implicitly space) trees.
See, there's just no way any game could figure out fast enough whether I hit every single thing on the map: it would take too long to check if I potentially hit thousands of different objects. To speed that up, games with 3D collision detection all use some form of subdivision algorithm. The idea is that rather than check to see if I hit any of a hundred trees in Atlas Park, Atlas Park itself is subdivided into groups of objects. Each group is surrounded by a bounding box that contains all of them. Before you go checking to see if you hit anything in the box, you first check to see if you are *inside* the box. If you aren't, you can ignore everything in it. If you make groups of objects, and groups of groups of objects, and groups of groups of groups of objects, you can eventually check for collisions much faster. The ultimate in speed (but with caveats) is the binary space partitioning algorithm, where your entire map is split roughly into two parts, and each part is split into two parts, etc etc. Even if you have a million objects, checking for collision is just a matter of making 20 checks for collision with ever smaller boxes, and then finally checking for detailed collisions between you and the few things in that final box. That's fast.
Whether you use BSP or some other algorithm, it helps if you design the space in a manner that reflects how you intend to subdivide it. Everything should be in one and only one box (or in a box inside a box inside a...), and the boxes should connect together in a reasonable fashion (or your map designers will go crazy). So big maps like Atlas Park are actually "islands" of geometry, with City Hall here and the Tram station there, each with its own "ground" and all of the "grounds" connected together in a patchwork of ground. You don't have to specifically deal with the tram, but you still have to handle the "Tram island" of geometry, which includes the ground in that area.
This is all redundant discussion for computer graphics geeks, but its something bot writers who are not computer graphics geeks need to be moderately aware of. Or they need to make ghostbots. There's nothing wrong with ghostbots per se.
While I'm thinking about that, I am going to redouble my efforts to get costumes and costume hashes working (of course, so far I've put in almost nothing, so that's not saying a lot). At first, I thought combat was more interesting to get working (pseudo, no effects, shadow boxing type combat). But last night, before I went to bed, I had an inspiration. Now I think getting the costume hash is a lot more important. Hopefully, by the weekend, I'll be able to demonstrate why.
-
When such a request is received, if the hash corresponds to a known costume, Paragon Chat will reply with a copy of the costume in the following format:
<iq to="user@domain/resource" id="id12345" type="result">
<costume xmlns="pc:costume">
<appearance bodytype="1" scale="2.1" bone="0.5" head="0.5"
shoulder="0.5" chest="0.5" waist="0.5" hip="0.5" leg="0.5"
head3d="123456ab" brow3d="123456ab" cheek3d="123456ab"
chin3d="123456ab" cranium3d="123456ab" jaw3d="123456ab"
nose3d="123456ab" />
I don't see skin color in this description of the Appearance stanza. Is it safe to assume it should be included as "skincolor=123456ab"?
-
I don't see skin color in this description of the Appearance stanza. Is it safe to assume it should be included as "skincolor=123456ab"?
Whoops, yes, it should be an attribute of appearance. Except that it's 123456ff because alpha is always forced to ff.
Updated the original, thanks!
-
Okay.
The skin color value in the paragonchat.db database does not work out to anything that I can make sense of.
It sort of works when it's a positive value. Here's an excerpt from the .costume file for Element Guy:
JawScales -0.7371, -1, 1
NoseScales -0.5371, 0.661, 0.3314
SkinColor 255, 178, 155
NumParts 28
Here's his character appearance record in ParagonChat.db:
select skincolor from costume where character = 3 and costume = 0;
10203903
A little handy python code tells me:
print(hex(255),hex(178),hex(155))
0xff 0xb2 0x9b
print(hex(10203903))
0x9bb2ff
So far, so good. It looks like BGR rather than RGB but otherwise the numbers are right.
Another character, Artiste, has a skin color of -9721635 in the SQL database.
His costume file defines it as: SkinColor 221, 168, 107
Here's what python says about that:
print(hex(221),hex(169),hex(107))
0xdd 0xa8 0x6b
print(hex(-9721635))
-0x945723
The minus sign on the on the hex number just represents an infinity of FF as far as I can determine.
I won't bore you with the various permutations I've tried to apply to make this work out, especially when there's probably a simple explanation for it.
So, what's going on? Why is it apparently backward for a positive value and seemingly unresolvable for a negative value?
-
The order thing sounds like the thing I mentioned earlier involving pickle-packing into integers. As to the negative numbers, that's a signed/unsigned thing. Hint: try two's complement.
-
The order thing sounds like the thing I mentioned earlier involving pickle-packing into integers. As to the negative numbers, that's a signed/unsigned thing. Hint: try two's complement.
Bah. I knew about the two's complement but it helps things when you shift by the right number of bits when you decode.
Still, thanks for getting me to take a second look at it. I suppose I'll be shaking the rust out of a lot of barely used old skills by the time this is all said and done.
So, bottom line - the colors stored in the SQL database are BGR, and negative colors need to have (1 << 24) added to them before hexing them. I'm presuming that the other colors are encoded the same way as the skin color.
One step closer to that costume hash.
-
So, bottom line - the colors stored in the SQL database are BGR, and negative colors need to have (1 << 24) added to them before hexing them. I'm presuming that the other colors are encoded the same way as the skin color.
That sounds complicated and probably wrong. Ignore the negative hex value, negative hex is usually meaningless.
It's just a 32-bit unsigned int being interpreted as a signed int. The same as how FXTINT shows up in demorecords iirc.
print(hex(221),hex(169),hex(107))
0xdd 0xa8 0x6b
print(hex(-9721635))
-0x945723
Instead, do
import struct
bin = struct.pack("i", -9721635)
num = struct.unpack("I", bin)
print(hex(num[0]))
0xff6ba8ddL
Or, if you don't want to use struct and want to do it the slower mathy way that only works for negative numbers,
print(hex(2**32 - 9721635))
0xff6ba8ddL
The bytes are in ABGR order because the integer originated on a little-endian architecture.
One step closer to that costume hash.
The costume hash doesn't really have anything to do with the database storage format. The database isn't intended to be used by other applications and the schema is subject to change at any time. That's why colors are normalized to RGBA hex-digits-as-strings for the hash and for transmission on the wire.
-
Instead, do
import struct
bin = struct.pack("i", -9721635)
num = struct.unpack("I", bin)
print(hex(num[0]))
0xff6ba8ddL
Technically, I think you should do "bin = struct.pack("=i",-9721635)". I'm not 100% certain, but if there are any budding Mac OSX bot writers out there using python compiled natively, sizeof(int) might be 8. I know its 4 on Windows whatever, no matter the cpu. But I think on OSX and Linux gcc will with the right flags compile an 8 byte integer. I think python struct will expose that.
-
Technically, I should be writing this in C or some language with a sensible grasp of unsigned integers and other niceties.
At this point,though, I'm halfway to being adequate in Python so I suppose I'll keep plugging at it.
I wasn't implying that the database storage formats had anything to do with costume hashing. The *data* in the database, however, is pretty much the meat of the hash. ;-)
Sure, I completely agree. The PC database is not intended for anything but PC, etc... How wasteful is it, though, to take a perfectly functional, structured relational database, unload it to a passel of individual, minimally structured and unrelated flat text files, and then munge all of those individual files back into a database whose structure is nearly identical to what it originally had?
Yeah, the "proper" thing to do is to bite the bullet and write a Costume2SQL() parser but it sure seems like a bitch to have to do it.
For now, anyway, it's more sensible to dump the costume tables and use them my for own purposes and if the format changes or the Emperor of Blefuscu ordains that colors be stored in a different order then I'll deal with that.
***EDIT***
The reason the 24-bit thing feels wrong is that it masks off the last byte. We don't actually care about the last byte though - it's always FF.
-
I realized something this morning.
A bot will be able to edit a base.
It won't be able to see what it's doing, of course, but it will be able to follow a script that it knows will pick up object A and move it to point B or replace it with another object.
Theoretically, a bot could play a basic "shell game" by dropping three crates on the ground then announcing the shuffling of the crates or making some visual show of sliding them around and putting something small under one of the crates to be the "pea".
The limits of possibility would mostly be the ability of the bot to ignore placement restrictions that the client UI imposes on a player, just like ArcanaBot ignores gravity by hard-setting her position in space while the client UI imposes gravity on a player.
-
The base edit process won't be over XMPP. That wouldn't make sense. There's no reason for a player to edit another player's base instead of just editing it themselves.
A bot could _upload_ a finished base into whatever mechanism we end up using to share the base map between players. But you don't need a bot to do that anyway since you can just load one from a file.
-
The base edit process won't be over XMPP. That wouldn't make sense. There's no reason for a player to edit another player's base instead of just editing it themselves.
A bot could _upload_ a finished base into whatever mechanism we end up using to share the base map between players. But you don't need a bot to do that anyway since you can just load one from a file.
Yes, you're right. That would involve scripting the client in some way. Scratch that, then or at least table it until the day that a LUA engine is added to the client.
OTOH - If you can load a base file while someone is in the base, then you can still have your bot do "scene changes" on command. The practicality of it would depend on the time involved in "repainting" the base. It might be something a RP group would find interesting.
-
Venting ahead. Skip to the last line for the non-venting content.
Just so to illustrate why I dislike working with .costume files, here are two examples from characters that I saved out to .costume yesterday afternoon.
{
BoneScale 0.451
ShoulderScale 0.336
WaistScale 0.158
HipScale 0.226
HeadScales 0, 0, 0
BrowScales 0, 0, 0
CheekScales 0, 0, 0
ChinScales 0, 0, 0
CraniumScales 0, 0, 0
JawScales 0, 0, 0
NoseScales 0, 0, 0
SkinColor 246, 187, 170
NumParts 28
BodyType 1
CostumePart ""
{
and
{
Scale -5.336
BoneScale 0.255
ShoulderScale 0.174
ChestScale 0.188
WaistScale 0.176
HipScale 0.208
LegScale 0.005
HeadScales 0, 0, 0
BrowScales 0, 0, 0
CheekScales 0, 0, 0
ChinScales 0, 0, 0
CraniumScales 0, 0, 0
JawScales 0, 0, 0
NoseScales 0, 0, 0
SkinColor 221, 168, 107
NumParts 28
CostumePart ""
{
A comparison of the two shows that neither file contains all of the definitions. Char 1 has no general scale, no chest scale, no leg scale. Char 2 has no body type. What does it mean? The missing scales should default to zero? Default to 1.0? Who knows? Sure, @Codewalker and @Leandro probably have worked it out already but for someone whose only access to a costume is this file, how are you supposed to know that?
What else is missing that I can't even see because I don't know about it?
If I'm building a costume stanza, do I make up values for those things that seem "reasonable" or do I leave them blank and let the client that's looking at me figure it out? This matters because, for instance, the <Appearance> stanza has an attribute called "head=". Presumably this is a HeadScale that's invoked when you change your head shape. I don't typically mess with that, though, so *none* of my .costume files have a HeadScale in them.So, what do I do? What does that mean for the costume hash? Ignore the missing scale? Do I just drop the body type of Char 2 out of the hash? Do I just leave off the Scale attribute for Char 1 and hope for the best?
Yes, I know that @Codewalker has previously defined a Body Type = 0 as "male" but that's not the point. The costume file doesn't tell me that it's a male body type. It doesn't tell me anything at all, so I have to fall back on "Well, it's not this or that so it must be the other". It's a pain in the posterior.
It should hardly be a wonder that anybody would want to skip .costume files entirely and "cheat" by grabbing the costumes straight out of the client's database.
Okay, venting over.
I've decided to do the correct thing and parse the damn .costume files, but that means I need to know what Paragon Chat/CoH uses as a default value when a .costume dispenses with recording one or more attributes. Do they always default to 0 or 0.0?
-
All of the scales in a costume file default to 0 if they're left out. I kept it the same for PC to make sure it would be compatible. I agree depending on parser defaults can make it ambiguous, which is why I added the explicit 'omitted if 0' language to the formal definition for PC.
I think "Head" may actually be obsolete and predate the individual face sliders. I snagged the names from the client's built-in costume file parser but didn't check to see if they were actually used. So that one will probably go away.
-
What else is missing that I can't even see because I don't know about it?
Back when the Architect was in beta, one of the things we were wondering was how the Architect itself stored costumes internally, because it was that size, and not our own local file size, that counted against our mission quote. So what I did was to start creating costumes. I created AllBlackMan, who used black as the color for everything and AllWhiteMan who used white as the color for everything. I made costumes with as few parts as possible, and with as many parts as possible. I made BlandGuy that moved none of the sliders, and CrazyGirl that moved all of them, and reported differential file sizes. That gave us an idea of how "expensive" it was to do various things, and answered questions like did certain default colors get dropped from the storage and thus actually make the costume smaller (not that I recall) or did using multiple colors require more storage (generally it did not).
You could do the same: make "MaximalCostumeMan" that moves every slider, adds parts to every possible part node, and turns fx on in as many places as is legal, and save that. Then use that to test your costume code to make sure you are handling every single field correctly.
Also, Codewalker did state in his costume stanza description that "zeros" could be dropped from transmission. In fact for certain things he suggested you either SHOULD, or MUST drop them because they could subtly cause hash glitches which then cause invisibility issues (although its possible by now his code guards against that either way).
I got home, wrote up my costume parser, then got sleepy and went to bed. No idea if it works, because I fell asleep before writing like two lines of SHA code. I should just hire a twenty year old to write this for me for lunch money.
-
Yeah, it's more the principle of the whole thing rather than the difficulty of any individual bit of it. I shouldn't have to make Max Slider Man and Min Slider Girl. Those silly Cryptic and Paragon Studios devs ought to have anticipated that I would want this information years after their game was dead and packaged it up with full documentation and a bow on top.
And I still feel like using costume files in the first place is a bit like walking six blocks down to the end of the fence separating my yard from my neighbor's yard just so that I can step around the end of it and walk six blocks back up to visit my neighbor.
On the plus side, going the .costume route has made me consider some alternate implementations that may end up being more interesting than going the lazy route.
-
And I still feel like using costume files in the first place is a bit like walking six blocks down to the end of the fence separating my yard from my neighbor's yard just so that I can step around the end of it and walk six blocks back up to visit my neighbor.
I don't see it that way, for the simple reason that absolutely nothing in the Paragon Chat databases isn't also stored in costume files for me, because its not like I don't trust Codewalker, but its called beta software for a reason. Whether its Microsoft or Codewalker, I'm not storing the only copy of anything in beta software. Furthermore, I have a huge library of costumes saved from playing the game, and I would want my bot to be able to tap into that without having to create a character in Paragon Chat and load the costume. In other words, I consider the costume files to be "the source" and Paragon Chat to be just an intermediary incomplete source when it comes to the question of bots. I should, in fact, be able to create a costume for a bot by directly creating a costume file and loading it into the bot without having to even run Paragon Chat, or even without having even *seen* Paragon Chat in theory.
To go even further, a bot should be able to create and display a costume without ever touching anything else, not even costume files. But if you are going to save costumes at all, the costume file seems to be the best way to do so, because you're now using a standard file format other people could use for other purposes. A bot could enter a zone and record everyone's costumes and write them to costume files. You could then peruse them later in the costume creator. If you want to know how someone created a costume, you could have a bot snapshot it for later. All of this makes costume files better to figure out how to read and write, and none of it would be helped by looking at the Paragon Chat databases. To be honest, I have yet to even *peek* into them, because while I'm curious, there is literally nothing in there that would be remotely helpful to me at the moment. In fact as you've experienced, it can actually lead you astray.
On the subject of lazy, this is how I read costume files:
pc_costume_dict = {}
pc_costume_dict['Parts'] = {}
PartNum = 0
clevel = 0
for pc_costume_line in pc_costume:
if pc_costume_line[0] == '\n':
pass
elif clevel == 0:
if pc_costume_line[0] == '{':
clevel = clevel + 1
elif clevel == 1:
if pc_costume_line[0] == '{':
clevel = clevel + 1
elif pc_costume_line[0] == '}':
clevel = clevel - 1
else:
pc_costume_dict[pc_costume_line.split()[0]] = ' '.join(pc_costume_line.split()[1:])
elif clevel == 2:
if pc_costume_line[0] == '}':
clevel = clevel - 1
PartNum = PartNum + 1
else:
if not pc_costume_dict['Parts'].has_key(PartNum):
pc_costume_dict['Parts'][PartNum] = {}
pc_costume_dict['Parts'][PartNum][pc_costume_line.split()[0]] = ' '.join(pc_costume_line.split()[1:])
pass
else:
print "Debug: parsing error"
And this is how I convert them into ParagonChat-ese:
for key in pc_costume_dict:
if key in scale_list:
print "Debug: element: " + key + " value: " + pc_costume_dict[key]
pc_costume_dict[key] = dezero(pc_costume_dict[key])
print "Debug: converted: " + pc_costume_dict[key]
pass
if key in scales_list:
print "Debug: element: " + key + " value: " + pc_costume_dict[key]
pc_costume_dict[key] = tuple2packint(pc_costume_dict[key])
print "Debug: converted: " + pc_costume_dict[key]
pass
if key in color_list:
if key == 'SkinColor':
print "Debug: element: " + key + " value: " + pc_costume_dict[key]
pc_costume_dict[key] = colorstr2hex(pc_costume_dict[key])
print "Debug: converted: " + pc_costume_dict[key]
else:
print "Debug: element: " + key + " value: " + pc_costume_dict[key]
pc_costume_dict[key] = key[5]+colorstr2hex(pc_costume_dict[key])
print "Debug: converted: " + pc_costume_dict[key]
for part in pc_costume_dict['Parts']:
for key in pc_costume_dict['Parts'][part]:
if key in color_list:
if key == 'SkinColor':
print "Debug: element: " + key + " value: " + pc_costume_dict['Parts'][part][key]
pc_costume_dict['Parts'][part][key] = colorstr2hex(pc_costume_dict['Parts'][part][key])
print "Debug: converted: " + pc_costume_dict['Parts'][part][key]
else:
print "Debug: element: " + key + " value: " + pc_costume_dict['Parts'][part][key]
pc_costume_dict['Parts'][part][key] = key[5]+colorstr2hex(pc_costume_dict['Parts'][part][key])
print "Debug: converted: " + pc_costume_dict['Parts'][part][key]
It doesn't get a lot more lazy than that. I didn't even use closures. I even did extra work forgetting SkinColor was the only non-numbered Color (duh) and I left in my placeholder passes.
Its hard to be creative when you're doing this an hour or so at a time. Honestly, I'm a crap programmer unless I really get into the swing. I have a habit of doing this:
def colorstr2hex(colorstr):
## colorhex = hex(int(colorstr.split()[0].split(',')[0]))[2:] + \
## hex(int(colorstr.split()[1].split(',')[0]))[2:] + \
## hex(int(colorstr.split()[2].split(',')[0]))[2:] + 'ff'
colorhex = "{0:0{1}x}".format(int(colorstr.split()[0].split(',')[0]),2) + \
"{0:0{1}x}".format(int(colorstr.split()[1].split(',')[0]),2) + \
"{0:0{1}x}".format(int(colorstr.split()[2].split(',')[0]),2) + 'ff'
return colorhex
Because first I do it the wrong way, then I realize its the wrong way, then I remember the right way, then I write a hybrid love-child mutant of the wrong way and the right way and call it a day.
Codewalker would probably be horrified to learn I did this:
def tuple2packint(tuplestr):
print "Debug: tuplestr: " + str(tuplestr)
x = tuplestr.split()[0].split(',')[0]
#print "debug: x=" + str(x)
y = tuplestr.split()[1].split(',')[0]
z = tuplestr.split()[2].split(',')[0]
xx = abs(int(x * 100))
yy = abs(int(y * 100))
zz = abs(int(z * 100))
if cmp(x,0) == -1:
xx = xx + 128
if cmp(y,0) == -1:
yy == yy + 128
if cmp(z,0) == -1:
zz == zz + 128
# return '00' + hex(zz)[2:] + hex(yy)[2:] + hex(xx)[2:]
return '00' + "{0:0{1}x}".format(zz,2) + "{0:0{1}x}".format(yy,2) + "{0:0{1}x}".format(xx,2)
I didn't even guard against overflow, bah. Notice: same mistake again. Dr. Dobbs flashback quiz: what was the mistake?
-
Heh. You know you can avoid that whole "split once on space and then split off the comma" business by splitting on ", "? I started to go the double-split route also until I read the docs again and verified that the separator can be a string and not just a character. ;-)
Furthermore, I have a huge library of costumes saved from playing the game, and I would want my bot to be able to tap into that without having to create a character in Paragon Chat and load the costume. In other words, I consider the costume files to be "the source" and Paragon Chat to be just an intermediary incomplete source when it comes to the question of bots. I should, in fact, be able to create a costume for a bot by directly creating a costume file and loading it into the bot without having to even run Paragon Chat, or even without having even *seen* Paragon Chat in theory.
I came around to a similar way of thinking today while I was considering how I wanted to approach the .costume parser. I don't have a library of costumes from the old days (grand total of four and two of those have been overwritten by PC-generated costume files), but when I reviewed my pie-in-the-sky goals for the bot, one of them is that I want to be able to tell it to "fetch!" and have it reach out over the internet to a web repository and pull down a costume or a package of props and accessories and install it on command.
Making everything SQL based was pretty much the antithesis of that.
However, I had a realization - I do need a data structure to store the costume data, and while a bunch of dicts immediately comes to mind, there's another structure available that I have to build anyway in order to interact with Paragon Chat - The costume stanza.
The stanza not only acts as an in-memory data structure for the costume, it's simple to export and it's human-readable. By saving the stanza XML to a text file and naming the text file with the hash, a simple database is automatically created that allows the bot, or a human, to quickly load and display any costume with no computation involved.
Meanwhile, the local SQL database can be used to associate a given stanza with a character name, description, and any other relevant relational data. The bot can even save the stanzas it receives from other players and cache them or otherwise save them for later re-use.
So, I'm meh on costume files but I'm finding the idea of .stanza files to be something interesting that works the same way while providing more functionality than a .costume currently provides.
***EDIT***
Oh, pro tip - If you've got a big library of older costume files, you'll want to add a rule that translates "CostumeFilePrefix [male | female | huge]"
to the appropriate 'bodytype'.
-
So, I sort of have costume hashes computing, but things are not matching up. First problem I noticed: what is in a costume file is not necessarily what is in the costume. I took the costume that I saved on the last day of the game, loaded it into Paragon Chat, then immediately resaved it. The two files are not identical. There are some subtle differences (in particular, upper/lower cases changes) and a few major ones (an unused part disappeared in the newly saved version, which implies I24 had different costume part checking than I23). To eliminate these problems I loaded and saved a costume file and hashed that. The hashes still don't match.
Question: since hashing is case sensitive, is case forced all up or all lower for consistency? The sample suggested it was not ("Leather" retained its capitalization). Also, this probably has nothing to do with anything, but I24 itself *created* a Part that was not in the I23 costume file, and weirdly its not a functional part entry: it looks like this in Part 25:
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P546439883
RegionName "Lower Body"
BodySetName NewSkirts
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
Here's the costume in question (the I24 version):
{
BoneScale -0.85
ChestScale -0.53
WaistScale -0.78
HeadScales 0, 0, 0
BrowScales 0, 0, 0
CheekScales 0, 0, 0
ChinScales 0, 0, 0
CraniumScales 0, 0, 0
JawScales 0, 0, 0
NoseScales 0, 0, 0
SkinColor 246, 187, 170
NumParts 28
BodyType 1
CostumePart ""
{
Geometry shorts
Texture1 skin_tights
Texture2 miniskirt_1
DisplayName P1525729866
RegionName "Lower Body"
BodySetName NewSkirts
Color1 255, 0, 0
Color2 0, 69, 204
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Tight
Texture1 skin_tights
Texture2 Top_5
DisplayName P566009771
RegionName "Upper Body"
BodySetName Tights/Skin
Color1 255, 0, 88
Color2 212, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry V_fem_Head.GEO/GEO_Head_V_Asym_Standard
Texture1 !v_sf_face_skin_head_10
Texture2 none
DisplayName P687117166
RegionName Head
BodySetName standard
Color1 255, 101, 205
Color2 204, 202, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Bracer_02
Texture1 Skin_Bracer_02a
Texture2 Skin_Bracer_02b
DisplayName P3937616722
RegionName "Upper Body"
BodySetName Tights/Skin
Color1 255, 253, 0
Color2 212, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Hi_Heels_02
Texture1 skin_Hi_Heels_02a
Texture2 skin_Hi_Heels_02b
DisplayName P2104750136
RegionName "Lower Body"
BodySetName NewSkirts
Color1 153, 0, 49
Color2 0, 0, 255
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry standard
Texture1 MARTIAL_ARTS_01
Texture2 none
DisplayName P177456852
RegionName "Upper Body"
BodySetName Tights/Skin
Color1 153, 0, 49
Color2 31, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Long_04
Texture1 Long_01a
Texture2 Long_01b
DisplayName P2681220175
RegionName Head
BodySetName standard
Color1 255, 0, 0
Color2 170, 56, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Tiara_01
Texture1 Tiara_01a
Texture2 Tiara_01b
DisplayName P2793026233
RegionName Head
BodySetName standard
Color1 204, 202, 0
Color2 212, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Tight
Texture1 Base
Texture2 Star_10
DisplayName P2281134661
RegionName "Upper Body"
BodySetName Tights/Skin
Color1 0, 0, 0
Color2 255, 253, 76
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P772741860
RegionName "Upper Body"
BodySetName Tights/Skin
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 31, 31, 31
Color2 227, 227, 227
Color3 31, 31, 31
Color4 227, 227, 227
}
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P2371314042
RegionName Head
BodySetName standard
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry Full
Texture1 Cape_Top_01
Texture2 none
DisplayName P1987928225
RegionName Capes
BodySetName FullMantle
Color1 255, 0, 88
Color2 255, 0, 88
Color3 255, 0, 88
Color4 255, 0, 88
}
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P33398819
RegionName Capes
BodySetName FullMantle
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx capes/CapeLongFem.fx
Geometry none
Texture1 !X_Valkyrie_Cape_01
Texture2 !Cape_Valkyrie_01_Mask
DisplayName P4269015351
RegionName Capes
BodySetName FullMantle
Color1 255, 0, 88
Color2 255, 0, 88
Color3 255, 0, 88
Color4 255, 0, 88
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry ShortSkirt_01
Texture1 pleated
Texture2 plaid_01b
DisplayName P3930658043
RegionName "Lower Body"
BodySetName NewSkirts
Color1 255, 0, 88
Color2 255, 0, 88
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P1848153390
RegionName Head
BodySetName standard
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Fx none
Geometry none
Texture1 none
Texture2 none
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Geometry none
Texture1 none
Texture2 none
DisplayName P546439883
RegionName "Lower Body"
BodySetName NewSkirts
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
CostumePart ""
{
Color1 0, 0, 0
Color2 0, 0, 0
Color3 0, 0, 0
Color4 0, 0, 0
}
}
Paragon Chat seems to be advertising the hash as: Lxdw1TQl8g/dEYfAJNh7LE7oe6c=
I am calculating it as: ycY42S40EIPgvh6GaGWYI+H/tdk=
For the record, the string of data being hashed looks like this:
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
And if you want to see what my parser is reading and writing and tossing:
Debug: adding element: BodyType with value: 1
Debug: adding element: SkinColor with value: f6bbaaff
Debug: skipping element: Scale
Debug: adding element: BoneScale with value: -0.85
Debug: skipping element: HeadScale
Debug: skipping element: ShoulderScale
Debug: adding element: ChestScale with value: -0.53
Debug: adding element: WaistScale with value: -0.78
Debug: skipping element: HipScale
Debug: skipping element: LegScale
Debug: skipping element: HeadScales
Debug: skipping element: BrowScales
Debug: skipping element: CheekScales
Debug: skipping element: ChinScales
Debug: skipping element: CraniumScales
Debug: skipping element: JawScales
Debug: skipping element: NoseScales
Debug: adding Part: 0
Debug: adding element: Geometry with value: shorts
Debug: adding element: Texture1 with value: skin_tights
Debug: adding element: Texture2 with value: miniskirt_1
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff0000ff
Debug: adding color: Color2 with value: 20045ccff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 1
Debug: adding element: Geometry with value: Tight
Debug: adding element: Texture1 with value: skin_tights
Debug: adding element: Texture2 with value: Top_5
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff0058ff
Debug: adding color: Color2 with value: 2d40000ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 2
Debug: adding element: Geometry with value: V_fem_Head.GEO/GEO_Head_V_Asym_Standard
Debug: adding element: Texture1 with value: !v_sf_face_skin_head_10
Debug: skipping element: Texture2
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff65cdff
Debug: adding color: Color2 with value: 2ccca00ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 3
Debug: adding element: Geometry with value: Bracer_02
Debug: adding element: Texture1 with value: Skin_Bracer_02a
Debug: adding element: Texture2 with value: Skin_Bracer_02b
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1fffd00ff
Debug: adding color: Color2 with value: 2d40000ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 4
Debug: adding element: Geometry with value: Hi_Heels_02
Debug: adding element: Texture1 with value: skin_Hi_Heels_02a
Debug: adding element: Texture2 with value: skin_Hi_Heels_02b
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1990031ff
Debug: adding color: Color2 with value: 20000ffff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 5
Debug: adding element: Geometry with value: standard
Debug: adding element: Texture1 with value: MARTIAL_ARTS_01
Debug: skipping element: Texture2
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1990031ff
Debug: adding color: Color2 with value: 21f0000ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 6
Debug: adding element: Geometry with value: Long_04
Debug: adding element: Texture1 with value: Long_01a
Debug: adding element: Texture2 with value: Long_01b
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff0000ff
Debug: adding color: Color2 with value: 2aa3800ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: skipping Part: 7
Debug: adding Part: 8
Debug: adding element: Geometry with value: Tiara_01
Debug: adding element: Texture1 with value: Tiara_01a
Debug: adding element: Texture2 with value: Tiara_01b
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ccca00ff
Debug: adding color: Color2 with value: 2d40000ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: adding Part: 9
Debug: adding element: Geometry with value: Tight
Debug: adding element: Texture1 with value: Base
Debug: adding element: Texture2 with value: Star_10
Debug: skipping element: Fx
Debug: skipping color: Color1
Debug: adding color: Color2 with value: 2fffd4cff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: skipping Part: 10
Debug: skipping Part: 11
Debug: skipping Part: 12
Debug: skipping Part: 13
Debug: skipping Part: 14
Debug: adding Part: 15
Debug: adding element: Geometry with value: Full
Debug: adding element: Texture1 with value: Cape_Top_01
Debug: skipping element: Texture2
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff0058ff
Debug: adding color: Color2 with value: 2ff0058ff
Debug: adding color: Color3 with value: 3ff0058ff
Debug: adding color: Color4 with value: 4ff0058ff
Debug: skipping Part: 16
Debug: adding Part: 17
Debug: skipping element: Geometry
Debug: adding element: Texture1 with value: !X_Valkyrie_Cape_01
Debug: adding element: Texture2 with value: !Cape_Valkyrie_01_Mask
Debug: adding element: Fx with value: capes/CapeLongFem.fx
Debug: adding color: Color1 with value: 1ff0058ff
Debug: adding color: Color2 with value: 2ff0058ff
Debug: adding color: Color3 with value: 3ff0058ff
Debug: adding color: Color4 with value: 4ff0058ff
Debug: skipping Part: 18
Debug: adding Part: 19
Debug: adding element: Geometry with value: ShortSkirt_01
Debug: adding element: Texture1 with value: pleated
Debug: adding element: Texture2 with value: plaid_01b
Debug: skipping element: Fx
Debug: adding color: Color1 with value: 1ff0058ff
Debug: adding color: Color2 with value: 2ff0058ff
Debug: skipping color: Color3
Debug: skipping color: Color4
Debug: skipping Part: 20
Debug: skipping Part: 21
Debug: skipping Part: 22
Debug: skipping Part: 23
Debug: skipping Part: 24
Debug: skipping Part: 25
Debug: skipping Part: 26
Debug: skipping Part: 27
I'll keep banging on it, but Codewalker if you could pass this costume through your hash algorithm manually and tell me what your code passes to SHA1 (I know you do it incrementally but a concatenation of what you toss into the hash algorithm would be appreciated) then a diff of the two might tell me where your algorithm diverges from mine.
Edit: a horrifying thought crossed my mind: what if the internal representation of the costume is not identical to what the game saves to costume files? All it takes is one word capitalized differently, and the hash will fail. Ugh, I might have to implement the Iq costume request just to be able to compare the two. If that's the case, costume files have a major fail.
-
Bingo. Irdumb. I logged two different Paragon Chat clients into my server, so they could send each other the costumes. This is what LA sends to VR when asked for her costume:
<costume xmlns="pc:costume"><appearance bone="-0.85" skincolor="f6bbaaff" waist="-0.78" bodytype="1" chest="-0.53"/>
<part color2="0045ccff" color1="ff0000ff" tex2="miniskirt_1" tex1="skin_tights" n="0" geom="shorts"/>
<part color2="d40000ff" color1="ff0058ff" tex2="Top_5" tex1="skin_tights" n="1" geom="Tight"/>
<part color2="ccca00ff" color1="ff65cdff" tex1="!v_sf_face_skin_head_10" n="2" geom="V_fem_Head.GEO/GEO_Head_V_Asym_Standard"/>
<part color2="d40000ff" color1="fffd00ff" tex2="Skin_Bracer_02b" tex1="Skin_Bracer_02a" n="3" geom="Bracer_02"/>
<part color2="0000ffff" color1="990031ff" tex2="skin_Hi_Heels_02b" tex1="skin_Hi_Heels_02a" n="4" geom="Hi_Heels_02"/>
<part color2="1f0000ff" color1="990031ff" tex1="MARTIAL_ARTS_01" n="5" geom="standard"/>
<part color2="aa3800ff" color1="ff0000ff" tex2="Long_01b" tex1="Long_01a" n="6" geom="Long_04"/>
<part color2="d40000ff" color1="ccca00ff" tex2="Tiara_01b" tex1="Tiara_01a" n="8" geom="Tiara_01"/>
<part color2="fffd4cff" tex2="Star_10" tex1="Base" n="9" geom="Tight"/>
<part color2="ff0058ff" color1="ff0058ff" tex1="Cape_Top_01" n="15" geom="Full"/>
<part color4="ff0058ff" color3="ff0058ff" color2="ff0058ff" color1="ff0058ff" tex2="!Cape_Valkyrie_01_Mask" tex1="!X_Valkyrie_Cape_01" fx="capes/CapeLongFem.fx" n="17"/>
<part color2="ff0058ff" color1="ff0058ff" tex2="plaid_01b" tex1="pleated" n="19" geom="ShortSkirt_01"/></costume>
Look at Part 15. This is Part 15 in the costume file:
CostumePart ""
{
Geometry Full
Texture1 Cape_Top_01
Texture2 none
DisplayName P1987928225
RegionName Capes
BodySetName FullMantle
Color1 255, 0, 88
Color2 255, 0, 88
Color3 255, 0, 88
Color4 255, 0, 88
}
For some reason, at least at first glance, Paragon Chat is dropping Color 3 and Color 4. It does not do that on part 17 which also has four colors in the part in the costume file, and Paragon Chat sends those:
CostumePart ""
{
Fx capes/CapeLongFem.fx
Geometry none
Texture1 !X_Valkyrie_Cape_01
Texture2 !Cape_Valkyrie_01_Mask
DisplayName P4269015351
RegionName Capes
BodySetName FullMantle
Color1 255, 0, 88
Color2 255, 0, 88
Color3 255, 0, 88
Color4 255, 0, 88
}
Not sure what's going on here. Maybe those colors are redundant, and the costume file has them but the game doesn't actually use them, and somehow Paragon Chat has sniffed this out and drops them. But this is a file that I resaved with I24, and it even made actual changes to the file, so this might be one of those things that needs to be hard coded around. Help me Codewalker Kenobi, you're my only hope.
-
Additional information: if I nullify (i.e. 0,0,0) those two colors in a new costume file and rehash, I end up apparently with identical information being put into the hash as Paragon Chat sends for that costume in its Iq costume response. However, the hash still doesn't match: now I get 1XAXLw/yJTTAhxHdLHvYJKd76E4= for the hash, and the string being hashed is now:
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
Something is definitely amiss: even if I force the hasher to hash the same data inside the costume Iq, I can't replicate the hash value Paragon Chat generates. Perhaps the costume hash generation algorithm has subtly changed since Codewalker first posted it, or maybe there's a subtle difference I'm not seeing at the moment. I'll keep looking, but something seems to be wrong with the algorithm I'm using.
-
I am calculating it as: ycY42S40EIPgvh6GaGWYI+H/tdk=
Hmm, I plugged your costume into Paragon Chat and ran its hash algorithm on it. Here's what I got:
----- BEGIN COSTUME DUMP -----
Hash: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=
Bodytype: 1
Skin color: f6bbaaff
BoneScale: -0.85
ChestScale: -0.53
WaistScale: -0.78
Part 0
Geom: shorts
Tex1: skin_tights
Tex2: miniskirt_1
Color1: ff0000ff
Color2: 0045ccff
Part 1
Geom: Tight
Tex1: skin_tights
Tex2: Top_5
Color1: ff0058ff
Color2: d40000ff
Part 2
Geom: V_fem_Head.GEO/GEO_Head_V_Asym_Standard
Tex1: !v_sf_face_skin_head_10
Color1: ff65cdff
Color2: ccca00ff
Part 3
Geom: Bracer_02
Tex1: Skin_Bracer_02a
Tex2: Skin_Bracer_02b
Color1: fffd00ff
Color2: d40000ff
Part 4
Geom: Hi_Heels_02
Tex1: skin_Hi_Heels_02a
Tex2: skin_Hi_Heels_02b
Color1: 990031ff
Color2: 0000ffff
Part 5
Geom: standard
Tex1: MARTIAL_ARTS_01
Color1: 990031ff
Color2: 1f0000ff
Part 6
Geom: Long_04
Tex1: Long_01a
Tex2: Long_01b
Color1: ff0000ff
Color2: aa3800ff
Part 8
Geom: Tiara_01
Tex1: Tiara_01a
Tex2: Tiara_01b
Color1: ccca00ff
Color2: d40000ff
Part 9
Geom: Tight
Tex1: Base
Tex2: Star_10
Color2: fffd4cff
Part 15
Geom: Full
Tex1: Cape_Top_01
Color1: ff0058ff
Color2: ff0058ff
Color3: ff0058ff
Color4: ff0058ff
Part 17
Tex1: !X_Valkyrie_Cape_01
Tex2: !Cape_Valkyrie_01_Mask
Fx: capes/CapeLongFem.fx
Color1: ff0058ff
Color2: ff0058ff
Color3: ff0058ff
Color4: ff0058ff
Part 19
Geom: ShortSkirt_01
Tex1: pleated
Tex2: plaid_01b
Color1: ff0058ff
Color2: ff0058ff
----- END COSTUME DUMP -----
I modified the hash routine to concatenate instead of feeding to the SHA1 digest, so here is the string that is being hashed:
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
As far as I can tell, it's identical to your results. Just for fun I took the concatenated string that you posted and ran it through sha1 and base64 on the command line. I got this.
c9c638d92e341083e0be1e8668659823e1ffb5d9
And ah dammit I got the endianness backwards.
ycY42S40EIPgvh6GaGWYI+H/tdk=
c9c638d92e341083e0be1e8668659823e1ffb5d9
2TjGyYMQNC6GHr7gI5hlaNm1/+E=
d938c6c98310342e861ebee023986568d9b5ffe1
I had been byte swapping OpenSSL's output because I was putting it in 32-bit unsigned ints and using sprintf with %08x to create the original hash, which was a hex string. When I switched to Base64 I forgot to undo the byte swapping. Well, I can't fix it right now without making everybody invisible, but I'm planning to do a "v2" costume hash for 1.0 to fix some nits (mostly making the hashing of the floating point scales less ambiguous and dependent on truncation/printf semantics), so I'll fix it then. 0.98 is going to include some compatibility code to ease the transition when we get to 1.0.
-
Edit: a horrifying thought crossed my mind: what if the internal representation of the costume is not identical to what the game saves to costume files? All it takes is one word capitalized differently, and the hash will fail. Ugh, I might have to implement the Iq costume request just to be able to compare the two. If that's the case, costume files have a major fail.
This shouldn't be an issue in practice. Yes, the game client will modify a .costume file that you load in odd ways before sending it to the server. But your bot can send the costume file as-is, and so long as the hash matches what you sent, Paragon Chat will accept it. It doesn't matter if the client mangles it once it's been transmitted because PC doesn't care what the client thinks at that point.
It just means that a costume file loaded by the game and a costume file loaded by your bot will have slightly different contents (and hashes). But so long as everything is consistent it will still work, you'll just have two very slightly different costumes that look identical.
Question: since hashing is case sensitive, is case forced all up or all lower for consistency? The sample suggested it was not ("Leather" retained its capitalization). Also, this probably has nothing to do with anything, but I24 itself *created* a Part that was not in the I23 costume file, and weirdly its not a functional part entry: it looks like this in Part 25:
Along the same lines, the client likes to swap those back and forth all the time. It's kind of annoying but doesn't affect the hashing. While the client happily considers them equivalent, Paragon Chat does not and is very careful to preserve case between the database and the wire. So the costume with "leather" will have a different hash than the one with "Leather", but once PC has them they will both work and won't randomly change.
Forcing them to lowercase for hashing would make the hashes equivalent and be slightly more efficient if two people are using the same costume differing only by case. That shouldn't really happen, but it's still a bit cleaner to ignore case for costume parts, so I'll do that in the V2 hash.
-
Out of curiosity:
How resilient is the costume protocol? There are a lot of places where the informal spec (i.e., page two of this thread) says "may omit" in regards to essentially null data, like parts that have a color but no geometry or fx. What happens if I decide to send those anyway? Just extra clutter on the wire?
What if I send all 30 parts from the costume file? Does it care about the extra two that the game's costume handler apparently never got around to defining?
What if I attach some extra attributes to a costume stanza, or inject some XML into it because, reasons...? I want to send a costume-related payload to another bot or app or something and I want to broadcast it rather than target the recipient and I'm too lazy to write my own special protocol to handle my data. Will Paragon Chat reject it or will it ignore the stuff it doesn't care about?
Note that I have no current intentions to do any of the above things, though the answer might affect how I handle this idea I'm experimenting with of saving costume stanzas to permanent storage and using them in various ways.
-
What happens if I decide to send those anyway? Just extra clutter on the wire?
Exactly. It will still set it to 0, but it was 0 anyway since that is the default that it's initialized to.
What if I send all 30 parts from the costume file? Does it care about the extra two that the game's costume handler apparently never got around to defining?
It will faithfully pass them on to the game client. I have no idea how the client handles it. I suspect you can put extra pieces there and they would work fine, but be uneditable in the tailor. It won't pass more than 30 though, that's the absolute max and anything beyond n="29" is discarded.
What if I attach some extra attributes to a costume stanza, or inject some XML into it because, reasons...? I want to send a costume-related payload to another bot or app or something and I want to broadcast it rather than target the recipient and I'm too lazy to write my own special protocol to handle my data. Will Paragon Chat reject it or will it ignore the stuff it doesn't care about?
Per XML handling best practices, it will silently ignore any attributes or elements that it doesn't know about.
Paragon Chat ignores any 'broadcasted' costumes in any event; it only parses them if they are sent in direct response to one of its costume IQ queries (i.e. has an id on the iq that matches a query it is waiting for a response to).
-
As far as I can tell, it's identical to your results.
Is it possible there's a string escape bug in the costume code? There appears to be a single character difference between what I have and what you have, and its a forward slash. The Fx in Part 17 is "capes/CapeLongFem.fx" and it shows up that way in my parser, and also in your parser, but your hash string shows it as "capesCapeLongFem.fx." Is that just a glitch in the hash string printing, or is that also happening in the incremental SHA hash as well?
-
Weird. It appears to be a cut and paste error somehow, because I still have the notepad window open that I copied it from, and it definitely says "...1_Maskcapes/CapeLongFem.fx1ff0058..." there. But in the post the slash is missing.
-
The byteswap seems to fix the mismatch. I won't be able to check it in-game until tonight, because I need to finish up the actual Iq response stanza to see if I can send this to anyone. But with luck that's the last impediment to getting this particular element working. Nice to see I didn't fail at writing a simple parser. Parsers are like 75% of my programming history.
Wait, now I have to support a version number? Aw man I was totally ignoring that attribute.
-
You don't have to, if you support the V2 format when it comes out (I'll post details ahead of time), you can just switch to that and your bot simply won't be visible to anyone running old versions. It's mostly incremental improvements that are not backwards compatible, but shouldn't be hard to adapt what you already have to.
The update check / nag feature will happen before the hash change happens, so hopefully there won't be many people running old versions anyway.
Most likely case is that clients with protocol=6 will use the new hash format in their costume attribute, while also (for a time) providing the old hash format in "oldcostume". protocol=5 clients will only know about the old format, but versions after 0.98 will know to check for "oldcostume" and use it if present, so that things will still work while there's overlap of 1.0 and 0.98 - 0.999 (repeating of course). Versions older than 0.98 won't be supported at all and will lose the ability to see people running 1.0 or later.
I don't expect the transition period to last extremely long, and will probably drop the oldcostume support in 1.1.
-
You don't have to, if you support the V2 format when it comes out (I'll post details ahead of time), you can just switch to that and your bot simply won't be visible to anyone running old versions. It's mostly incremental improvements that are not backwards compatible, but shouldn't be hard to adapt what you already have to.
The update check / nag feature will happen before the hash change happens, so hopefully there won't be many people running old versions anyway.
Most likely case is that clients with protocol=6 will use the new hash format in their costume attribute, while also (for a time) providing the old hash format in "oldcostume". protocol=5 clients will only know about the old format, but versions after 0.98 will know to check for "oldcostume" and use it if present, so that things will still work while there's overlap of 1.0 and 0.98 - 0.999 (repeating of course). Versions older than 0.98 won't be supported at all and will lose the ability to see people running 1.0 or later.
I don't expect the transition period to last extremely long, and will probably drop the oldcostume support in 1.1.
For backward compatibility shouldn't newer Paragon Chat clients advertise the old style hash as costume= and the newer hash as newcostume= or costume6=? That way older clients will simply ignore the newcostume attribute and default to using the older one.
Incidentally, did you figure out why Paragon Chat appeared to be dropping Color3 and Color4 from Part 15? I can get the hash to match now, but only by editing my costume file to zero out those two colors. If I load the original costume with nonzero Color3 and Color4, Paragon Chat seems to ignore those two colors; it doesn't send them as part of its <costume /> stanza, and its hash is consistent with those two colors simply vanishing.
-
Okay. I'm clearly doing something wrong with the SHA1 stuff because I put in Arcana's string and get entirely different results than Codewalker posted. (I'm testing Arcana's string because it's a known quantity at this point. Needless to say, my own costumes' hashes are also entirely different than what my XMPP traffic logs say they should be.)
Here's my snippet.py:
import hashlib
# Arcana's costume string
snippet = "1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_MaskcapesCapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff"
h = hashlib.sha1() # initializes a new sha-1 digest
h.update(snippet.encode('utf-8')) # the update barfs without some sort of string char encoding
print(h.hexdigest())
The result of this is: 21151b461e3ef228d0c21ce94e9a984761730dfb
Never mind bit swapping. That looks nothing like c9c638d92e341083e0be1e8668659823e1ffb5d9.
Despite this appearing to be straight forward, I'm clearly missing something important, especially since Arcana is working with the identical software and she's getting the right results.
What am I doing wrong?
***edit***
and now I'm really confused - I pasted Arcana's costume string into an online sha-1 encoder and got the identical result that Python's hashlib gave me.
What is it that I'm not "getting" about the instruction to "take the SHA-1 of the combined string"?
-
What am I doing wrong?
You're not pasting my string, you're pasting Codewalker's quote of my string, which somehow ended up missing a character. Note the "capesCapeLongFem.fx" sequence in your string. It should be "capes/CapeLongFem.fx". Remove the "/", and the hex hash ends up being 21151b461e3ef228d0c21ce94e9a984761730dfb which is I believe what you are getting.
-
You're not pasting my string, you're pasting Codewalker's quote of my string, which somehow ended up missing a character. Note the "capesCapeLongFem.fx" sequence in your string. It should be "capes/CapeLongFem.fx". Remove the "/", and the hex hash ends up being 21151b461e3ef228d0c21ce94e9a984761730dfb which is I believe what you are getting.
Right. You even posted that later in the thread.
/headsmack
Well, at least that means that any problems in my costume hashes are my own fault rather than my being an idiot at following simple instructions like "put tab A in slot B".
-
Codewalker - Am I understanding correctly that when you put this sha-1 digest: d938c6c98310342e861ebee023986568d9b5ffe1
through a base64 encoding, what you get out the other end is this: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=?
When I encode that I get - ZDkzOGM2Yzk4MzEwMzQyZTg2MWViZWUwMjM5ODY1NjhkOWI1ZmZlMQ==
That's what the online coder/decoder websites I've tried also get. Attempting to decode the base64 costume hash results in non-displayable binary data.
Is there an intermediate step that I'm missing?
-
Codewalker - Am I understanding correctly that when you put this sha-1 digest: d938c6c98310342e861ebee023986568d9b5ffe1
through a base64 encoding, what you get out the other end is this: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=?
When I encode that I get - ZDkzOGM2Yzk4MzEwMzQyZTg2MWViZWUwMjM5ODY1NjhkOWI1ZmZlMQ==
Don't base64 encode the hex string. The hex digits only represent a human-readable form of the "non-displayable binary data" that is the real SHA-1 hash. That's what you need to encode. With hashlib I think it's digest() rather than hexdigest().
-
Don't base64 encode the hex string. The hex digits only represent a human-readable form of the "non-displayable binary data" that is the real SHA-1 hash. That's what you need to encode. With hashlib I think it's digest() rather than hexdigest().
Yep.
ch = hashlib.sha1()
ch.update(costume_string)
ch_hash = binascii.b2a_base64(ch.digest())
That will do it.
Note that ch.digest() is a binary string, which means you can't do anything with it but process it with binary-aware functions. It can include zeros (nulls) with all that implies.
-
Incidentally, did you figure out why Paragon Chat appeared to be dropping Color3 and Color4 from Part 15? I can get the hash to match now, but only by editing my costume file to zero out those two colors. If I load the original costume with nonzero Color3 and Color4, Paragon Chat seems to ignore those two colors; it doesn't send them as part of its <costume /> stanza, and its hash is consistent with those two colors simply vanishing.
Are you sure that it's Paragon Chat that's removing those and not the game client when you load the costume? Easiest way to determine that is to see if those colors are 0 or the color in the sqlite database.
-
Groovy!
Thanks to both of you for the advice.
-
Are you sure that it's Paragon Chat that's removing those and not the game client when you load the costume? Easiest way to determine that is to see if those colors are 0 or the color in the sqlite database.
Weirdly, those colors are blank in the Paragon Chat database, *but* the costume file was actually *written* by the game client. What I did was enter the character creator, import the costume, then re-export it. Maybe the creator itself isn't stripping the colors, but entering the game is what triggers the color strip. Not sure. It seems the character creator can both import and export costume files that it doesn't actually apply in full to actual character entities.
Also, multiple <part /> stanzas? More work than I thought to make those work in sleekxmpp. Or rather, not so much work as a lot of typing. I probably won't even get the whole thing in a testable format until tomorrow. Fortunately, I actually have a stretch of a few hours tonight that nothing is demanding my time, so I'm going to see if I can focus on this for more than an hour.
Its hokey, but I really want to get this working because I really want to see what happens when I try something, because I'm curious what the entire system of client, xmpp server, and paragon chat will do.
-
Paragon Chat seems to be advertising the hash as: Lxdw1TQl8g/dEYfAJNh7LE7oe6c=
I am calculating it as: ycY42S40EIPgvh6GaGWYI+H/tdk=
For the record, the string of data being hashed looks like this:
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
Note - Paragon chat version referenced here is 0.97f
So, what's up with the hash that Paragon Chat is using? How "sacred" is it?
I'm asking because we never really addressed this issue of Paragon Chat computing a different costume hash than what Arcana was calculating for the identical costume.
We seem to have established that for the costume shown above, that Arcana's hash is correct (or, rather,the byteswapped version 2TjGyYMQNC6GHr7gI5hlaNm1/+E= is technically correct for now).
Despite that, Paragon Chat was sending her something else entirely when she logged in a character using that costume, as opposed to the bot being the one wearing it.
It gets murkier - since Arcana posted Arcanabot's costume file, I used it as a test case for my own parser. My parser can open Arcanabot's costume and end up with the exact same hash that we've denoted as "correct". So far, so good.
But when I login a female character and dress her up as Arcanabot, Paragon Chat is sending me a costume hash of i7yXq1nBm+PltncTA+pg50DfGRA=.
Clearly, that's different than the costume hash spec and different from what Aracana's own Paragon Chat client sent to her own server.
So, what's going on here? Should we be using the Paragon Chat client as a yardstick for judging what's correct from what's incorrect? If not, what yardstick do we use?
****EDIT****
Just pointing out that I haven't forgotten what Arcana was mentioning about the color discrepancy on part 15. However, if that was itself a consistent thing then my chat client ought to be producing the same hash that her chat client was producing, even if it differed from the "vanilla" hash.
-
I'm asking because we never really addressed this issue of Paragon Chat computing a different costume hash than what Arcana was calculating for the identical costume.
No, Paragon Chat was computing a hash for a different costume. The COH game client does not maintain costumes in a particularly stable format, often changing the case of strings or changing / removing colors that "don't matter" (i.e. aren't actually used by the costume part in question). Paragon Chat will calculate different hashes for these variations, because while they're visually identical to each other, they are not the same costume.
The hash that Arcana calculated, once adjusted for the byte swapping issue, is valid for the exact costume file that she posted. The costume that she was receiving after loading that file into the game was not exactly the same as the costume files.
Just calculate the hash for the costumes your bot is sending and you'll be fine. Don't worry about whether or not they exactly match one loaded into the game.
-
Weirdly, those colors are blank in the Paragon Chat database, *but* the costume file was actually *written* by the game client. What I did was enter the character creator, import the costume, then re-export it. Maybe the creator itself isn't stripping the colors, but entering the game is what triggers the color strip. Not sure. It seems the character creator can both import and export costume files that it doesn't actually apply in full to actual character entities.
It's quite possible that it's not the creator that's doing it, but rather the network protocol it transits over to make it to PC and back. It's quite... prematurely optimized IMO. I mean I know they were probably worried about zoning in and needing a hundred costumes sent over a 56k modem, but once you're in game how often do you really need to receive new costumes...
Alternatively, those colors may be getting inserted by the client somehow after the costume is received, but before it's written to the .costume file.
-
So, what's going on here? Should we be using the Paragon Chat client as a yardstick for judging what's correct from what's incorrect?
Well, if you want your bot to work, yes (since you have to agree with Paragon Chat clients to be visible to them).
Just to amplify what Codewalker said, what I noticed is a curious anomaly, but its not a fatal issue for bots. The issue I'm seeing is that if I give a bot a costume file and also give that costume file to a City of Heroes / Paragon Chat character, somehow they don't always end up with exactly the same costume data. The bot gets literally what's inside the file. But the CoH entity sometimes gets something different. Which means the CoH entity will send a different hash for its version of that costume than what the bot would. But the important thing is that so long as your hash matches *your* costume data, the costume data you literally send, Paragon Chat will validate the hash against what you send correctly and display it.
To put it another way, suppose I had a costume file with the word "cape." But I noticed that when I make a character in CoH/PC and load that costume file, somehow "cape" becomes "Cape" with a different capitalization. The Paragon Chat database shows "Cape" and when I watch the debug logs Paragon Chat sends "Cape" in its costume stanza. The hash is consistent with "Cape" and not "cape." That doesn't really matter to my bot, because if my bot sends "cape" even though its doing something different than what CoH itself does, so long as my hash matches "cape" and I send "cape" the hash will be consistent and Paragon Chat will be happy.
Why its happening is one of life's mysteries, and it means you cannot predict what the hash will be for a specific costume *file* when *City of Heroes* (via Paragon Chat) sends it. But you can predict what the hash will need to be when your bot sends it, because you know how your bot will send it.
-
Bah, another problem, this time with Iq stanzas. I can now compute my own hashes and send them accurately, but when Paragon Chat requests them...
conn DEBUG SENT: <iq id="0000001d-4d5a05cf" to="arcanabot@core3770/ArcanaBot_meta" type="get"><costume hash="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" xmlns="pc:costume"/></iq>
xmpp DEBUG RECV: <iq id="0000001c-4d5a05cf" to="arcanacoh@core3770/Violet" type="error" from="arcanabot@core3770/ArcanaBot_meta"><costume hash="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" xmlns="pc:costume"/><error code="503" type="cancel"><service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>
Its probably a bug somewhere in how I'm registering Iq request stanza callbacks (almost certainly because I don't even see them show up in any of my message handlers), but I'm not sure where that is yet. Because I registered them exactly the same way I registered Presence and Message stanzas, I presume the problem is that I *shouldn't* register them in precisely the same way because of something unique to Iq handling. Its possible one problem is that sleekxmpp appears to want queries to show up in <query> blocks by default, although I should be able to override that with the appropriate stanza matching. But that ain't going nowhere and I've exhausted every method of matching. I might be forced to RTFM again.
-
The XMPP spec requires that an Iq 'get' MUST be responded to, by either a 'result' or an 'error'.
It wouldn't surprise me if sleekxmpp has a special form of registering to handle Iqs, so that it can take care of sending back the 'error' for otherwise unhandled ones.
-
Just calculate the hash for the costumes your bot is sending and you'll be fine. Don't worry about whether or not they exactly match one loaded into the game.
That sounds good.
I can see how future bot programmers are going to find it very helpful to have a pre-made test bed that includes Max Slider Man and Crazy Slider Girl and Red Eyes Guy and so on as a way of validating their costume hashing code.
The important thing then is not that every given costume has a unique hash(because one costume can be "compiled" in an assortment of ways that can affect the hash), it's that a given costume stanza received off the wire can be validated by the hash presented in the associated <Character> stanza. Does that sound correct?
***EDIT***
Reading back now it looks like that's pretty much what Arcana said, so, good. I'm going to have to come up with some sort of testbed for costume hashes, though. If we can't just compare our hash to the hash presented by PC, then there's no practical way to tell whether your hash is actually correct or not. I only felt fairly confident about my hashing function after Codewalker certified Arcana's results and I was able to replicate her results. When you look at your function spit out a string of base64, it isn't intuitively obvious whether that string is correct or incorrect.
-
Reading back now it looks like that's pretty much what Arcana said, so, good. I'm going to have to come up with some sort of testbed for costume hashes, though. If we can't just compare our hash to the hash presented by PC, then there's no practical way to tell whether your hash is actually correct or not. I only felt fairly confident about my hashing function after Codewalker certified Arcana's results and I was able to replicate her results. When you look at your function spit out a string of base64, it isn't intuitively obvious whether that string is correct or incorrect.
Back up. I did this in a particular order to boot strap the process. If you want your bot to be visible, it actually doesn't have to process any costume stanzas at all as long as you recognize you can cheat: if you advertise a costume has that is identical to one everyone already knows about then they won't ask you for your costume hash because they already know what the costume corresponding to that hash looks like. In particular, if you are just testing, if your bot presents a costume hash that is identical to the one your Paragon Chat client presents for its logged in character (assuming you are testing on your own server and you have Paragon Chat logged in so you can actually attempt to see your bot) then because you're echoing back Paragon Chat's character hash, it will never ask you for your costume, because it will always know that costume and obviously it can't timeout its own costume in its hash table.
But there's no actual reason why your hashes must match Paragon Chat hashes in the specific. Your hashes must match the costumes you *send* to Paragon Chat. Even if the costume you send is a costume Paragon Chat would never actually create on its own, so long as your hash matches your costume stanza, Paragon Chat will at least validate it as correct and then use that costume data.
Unfortunately, there's no way to know if your hash is correct just by looking at it. In fact, that's the whole point of how hashes work: you can't predict what they should look like without just plain computing them. Which is why rather than use incremental SHA (as Codewalker did in Paragon Chat) I actually compute the long-form string. The reason for doing that is that in theory the place you're going to screw up is not in computing the hash, but determining what to feed to the hash engine. You and I and even Codewalker are just feeding OpenSSL's SHA1 algorithm: we have to generate the same hashes given the same input. By recording the string that goes in, I can compare that with the costume stanza data that Paragon Chat sends out for costumes. If ever I think something has gone wrong with my costume code, I can feed that costume to City of Heroes and ask Paragon Chat (by running it in debug mode) to tell me both what the hash is for that costume *and* what data it would send for that costume (in the costume reply stanza). If the costume data matches my costume file, I can then compare hashes freely. If the data doesn't match, as my costume did not, I need to modify my costume file to match what Paragon Chat sent so that my data and its data are identical, and then compare hashes. Or alternatively, I could conduct a detailed investigation into why the costume data changes inside of City of Heroes: there are a lot of weird redundancies in costume files, and City of Heroes often seems content to randomly fiddle with its don't-cares (which is a sign of extremely demented programmers).
In summary:
1. A specific set of costume data always generates the same (SHA1) hash.
2. Two sets of costume data that differ by any amount however small generate different hashes (ignoring cosmicly rare collisions), even if the only difference is upper/lower case differences.
3. A specific costume *file* is not guaranteed to generate costume data that is exactly identical within the game itself.
4. Two different costume files and two different costume data sets can generate the same appearance. For example, a costume data set in which all of the alpha characters are lower case generates the same appearance as one that is identical but all the alpha characters are upper case. However, these two costume data sets generate different hashes.
5. Paragon Chat compares the hash you send with the hash it computes for the data you send. Paragon Chat does not (and SHOULD not) do anything to the data you send before computing its hash, so if you send the exact data you used to compute your hash, it should work.
I should point out that I hit the bullseye on essentially the first try: this is not hard to do. If it wasn't for the endian issue (see: endianness can be an issue :p ) and the loss of two color elements (still need to track that down) I would have had it on the first go around.
-
Hmm. I'm not sure if SleekXMPP is doing the wrong thing, or I'm using it the wrong way. No matter what I do, I cannot seem to actually receive the Iq costume request sent by Paragon Chat. I even registered a stanza callback to show me *all* Iq requests, and it goes so far as to show me the Iq traffic that sets up the SSL handshake. It shows me the Iq traffic for rosters. And yet it does not ever show me the Iq costume request.
Completely separately, I seem to be not understanding substanza semantics quite right. I tried to do this:
class pcAppearanceStanza(ElementBase):
namespace = 'pc:costume'
name = 'appearance'
plugin_attrib = 'appearance'
interfaces = set(('bodytype','skincolor','scale','bone','head','shoulder','chest',
'waist','hip','leg','head3d','brow3d','cheek3d','chin3d','cranium3d',
'jaw3d','nose3d'))
class pcPartStanza(ElementBase):
namespace = 'pc:costume'
name = 'part'
plugin_attrib = 'part'
interfaces = set(('n','geom','tex1','tex2','fx','color1','color2','color3','color4'))
class pcCostumeStanza(ElementBase):
namespace = 'pc:costume'
name = 'costume'
plugin_attrib = 'costume'
interfaces = set(('hash', 'appearance', 'parts'))
subitem = (pcAppearanceStanza, pcPartStanza)
def getParts(self):
parts = {}
for pt in self.xml.findall('{%s}part' % pcPartStanza.namespace):
part = pcPartStanza(pt)
parts[part['n']] = part['value']
return parts
def setParts(self,parts):
for part in parts:
self.addPart(part)
def delParts(self):
parts = self.xml.findall('{%s}part' % pcPartStanza.namespace)
for part in parts:
self.xml.remove(part)
return
def addPart(self, part_dict):
part_obj = Part(None, self)
for element in ['n','tex1','tex2','fx','color1','color2','color3','color4']:
if element in part_dict:
part_obj[element] = part_dict[element]
return
def delPart(self, n):
parts = self.xml.findall('{%s}part' % pcPartStanza.namespace)
for partXML in parts:
part = pcPartStanza(partXML)
if part['n'] == n:
self.xml.remove(partXML)
return
registerStanzaPlugin(Iq,pcCostumeStanza)
That should register <costume /> as a substanza of Iq, and <appearance /> and <part /> as substanzas of <costume /> but that doesn't quite seem to work right either, because this doesn't work:
cStanza['costume']['appearance']['bodytype'] = pcCostumeDict['BodyType']
That throws a unicode error, implying I'm indirecting incorrectly off of that Stanza object. I'm beginning to wonder if doing this from literal raw TCP traffic wouldn't have been faster. It gets off the ground quickly, but past the 50% mark I've been fighting the framework almost as often as using it. Something simple like "show me every Iq stanza that arrives on the wire" should be bulletproof. It ain't.
JSON would have, in fact, been quicker. :p
-
JSON would have, in fact, been quicker. :p
VINDICATION!! :P
-
One problem solved: the reason why this doesn't work:
cStanza['costume']['appearance']['bodytype'] = pcCostumeDict['BodyType']
Is because while some stuff implies it works, it has no chance of working. Looking at the way the code works carefully, its now clear that cStanza['costume']['appearance'] cannot be further subscripted because you can only assign it a substanza. So you're supposed to do this:
tiq = xmpp.make_iq(id="abc123")
ap = pcAppearanceStanza(None, tiq['costume'])
ap['bodytype'] = '1'
That generates <iq id="abc123"><costume xmlns="pc:costume"><appearance bodytype="2" /></costume></iq>
I'm pretty sure I can now send the correct response to Iq requests I'm still not getting. The problem was figuring out what the heck the library writer was trying to accomplish, and doing that particular gyration with substanza classes proves to me the library writer was completely insane. It only makes sense if you write romantic love letters as XMPP xeps.
-
VINDICATION!! :P
How many times did you hit F5 before that happened?
-
Unfortunately, there's no way to know if your hash is correct just by looking at it.
Yes, and that's pretty much the reason why a testbed with a reference implementation is a useful, even necessary tool. The only real test for correctness is that you prove that someone else using your input achieves the same output.
To discover whether you were hashing correctly, Codewalker had to run your costume through his encryption routines. He even modified his normal process to do things the same way that you did them, even though technically it wasn't necessary to do that. Doing so revealed that *he* had accidentally introduced an unintended factor into the hash, that he otherwise would not have found.
When I was testing my hashing function, I knew your string and your results; your input and output. That meant that using your input I should get your output. When I didn't, I was able to run through my code and discover the typo where I'd used a dict index of 'Chest ' instead of 'Chest'.
With access to your costume, your string, and your final hash, I debugged that in a couple of minutes. Without those things, it would have taken considerably longer and I would have been obliged to wait until I was running the bot on the wire, realizing that Paragon Chat was not displaying my bot, and then trying to figure out which of the moving parts wasn't moving quite right.
That's all I was saying - That a lone developer without the support of a conversation like this one would be unable to verify his costume hash. The best he could do is run his bot and say, "Well, that didn't work..."
A reference program with a bunch of known costumes and known hashes, though, would give a bot writer the inputs and outputs he needed to prove the accuracy of his own hash routine.
As for the costume hash, it's true that I initially misunderstood its purpose. That's mostly because I didn't see much point in having a costume hash if it wasn't some sort of global identifier for a given combination of costume parts. Trusting anybody's costume hash to give you a "valid" costume is the same as trusting a self-signed security certificate. The only validation happening is that you received all the data in the costume stanza. TCP/IP and your XMPP framework already handle that validation for you.
If the value of a costume hash is that it gives you a unique identifier for filing the costume in your cache, then you could dispense with all of the encoding and just provide "mybot=costume_french_maid_123"
In any case - a reference implementation would avoid the need for a developer to test against a live server with a dozen other possible complications affecting the test.
-
How many times did you hit F5 before that happened?
I would never hit F5.
I caress it, lovingly. :P
-
I'm pretty sure I can now send the correct response to Iq requests I'm still not getting. The problem was figuring out what the heck the library writer was trying to accomplish, and doing that particular gyration with substanza classes proves to me the library writer was completely insane. It only makes sense if you write romantic love letters as XMPP xeps.
That's the main reason I went with libstrophe for Paragon Chat. Yes, it's a bit more work since I have to implement a lot of the higher level XMPP logic myself. I can't just tell it to join a MUC room, I have to actually build the stanza to do so and handle the results. But I have complete control over every bit of XML sent or received and don't have to worry about figuring out what the library author was trying to do to 'protect' me. The library just handles login and very basic stanza construction / parsing to make things easier.
-
That's the main reason I went with libstrophe for Paragon Chat. Yes, it's a bit more work since I have to implement a lot of the higher level XMPP logic myself. I can't just tell it to join a MUC room, I have to actually build the stanza to do so and handle the results. But I have complete control over every bit of XML sent or received and don't have to worry about figuring out what the library author was trying to do to 'protect' me. The library just handles login and very basic stanza construction / parsing to make things easier.
If I was writing in C, that's what I would have used. I tend to prefer minimal libraries, unless I am literally doing only what the library explicitly is designed to do. SleekXMPP was written by someone who would have been a LISP programmer if they could only find a good parenthesis autocompleter. There's a lot of cleverness in it that gets confusing if you're not the author of the cleverness and your primary documentation is sample code.
I suspect there's just one last bit of cleverness I have to figure out. Although there's also some XMPP issues I can loop back around to figure out while I'm thinking about that. I need to read the RFCs carefully because I'm still not sure I'm doing Presence quite right. If the bot enters second, its fine. If the bot enters first, its fine. If the bot leaves and comes back, it doesn't always reappear. And if I try to update Presence, that update doesn't actually get sent to PC. I seem to recall something in the spec about Presence updates being sent to roster instead of the room, but my memory is a little hazy there.
-
And if I try to update Presence, that update doesn't actually get sent to PC. I seem to recall something in the spec about Presence updates being sent to roster instead of the room, but my memory is a little hazy there.
Yes, if you broadcast presence to the server with no 'to' attribute, it only goes to the roster. It doesn't automatically update the presence in the rooms that you're in. You have to send directed presence to the room to update everyone who's in it. You'll notice that when you do a costume change in Paragon Chat, it re-sends presence several times to different destinations. A couple of those could probably be avoided now that <character> isn't sent to global channels, but are harmless as it is.
Keep in mind that MUC is usually implemented as a standalone service on a subdomain and doesn't necessarily have any special privileges, so it can only see stanzas that are explicitly sent to its subdomain.
-
Yes, if you broadcast presence to the server with no 'to' attribute, it only goes to the roster. It doesn't automatically update the presence in the rooms that you're in. You have to send directed presence to the room to update everyone who's in it. You'll notice that when you do a costume change in Paragon Chat, it re-sends presence several times to different destinations. A couple of those could probably be avoided now that <character> isn't sent to global channels, but are harmless as it is.
Keep in mind that MUC is usually implemented as a standalone service on a subdomain and doesn't necessarily have any special privileges, so it can only see stanzas that are explicitly sent to its subdomain.
Actually, the problem is more subtle than that. I don't broadcast presence without destination anymore, in fact, I only broadcast presence to the room(s) I'm in. But sometimes, I notice when Paragon Chat logs in, it gets an abbreviated Presence for the bot that is missing the "pc:xxx" namespace stuff. I am wondering, though, if the problem is specific to the way I'm testing the bots: sometimes the bots crash, and when they do they leave zombie connections on Openfire. Its possible Openfire is getting confused by the bot constantly relogging without doing any of the appropriate protocol to even discover if nickname is currently occupied (which is another disco I have to implement).
I should have just made that ski slope bot.
-
I should have just made that ski slope bot.
And 5 minutes after you had, you'd be doing this bot! You know you're never happy with the easy route. *grins*
-
For backward compatibility shouldn't newer Paragon Chat clients advertise the old style hash as costume= and the newer hash as newcostume= or costume6=? That way older clients will simply ignore the newcostume attribute and default to using the older one.
That was my first idea and would be simpler. However after some thought I realized that it would cause an OCD violation if the name of the attribute for the life of the project was changed due to a previous implementation briefly existing during what is effectively open beta.
-
That was my first idea and would be simpler. However after some thought I realized that it would cause an OCD violation if the name of the attribute for the life of the project was changed due to a previous implementation briefly existing during what is effectively open beta.
The flip side is: what happens if you have to change it again?
-
No question now sleekxmpp does something a little different when it comes to iq/get queries. Even when I try to register a callback for *all* Iq stanzas (i.e. <iq />) all I see in the debug logs are the replies to Iq queries (like replies to my own requests for roster). I never see *any* iq/get stanzas at all. Not even the trivially simple iq/appearance one.
Looking at ClientXMPP, BaseXMPP, and XMLStream, I don't see any reason why a simple registerHandler wouldn't work on Iq stanzas no different than any other, so it must be something not in the actual event handler but something in the way the library registers its own handlers to Iq. Maybe their own handlers somehow override all iq/gets and thus they never get passed to me, although that *also* appears to violate the actual documentation of the library. It also seems to violate all of the sample code for custom iq.
Anyone wanna jump right on in and start reading the iq source with me, I won't mind, honest. I suspect its also possible this isn't in the base handlers, but a side effect of something going on in the xep plugins. Oh well, maybe I'll have it working by next weekend.
-
I'm probably going to regret this but... Wouldn't it be an idea to get your bot running on the same XMPP server as Titan are using for now, and then see about alternatives later?
-
I'm probably going to regret this but... Wouldn't it be an idea to get your bot running on the same XMPP server as Titan are using for now, and then see about alternatives later?
SleekXMPP is a python library for making XMPP programs. Its not an XMPP server. I'm using Openfire as my server, the same server software Titan uses. And it works perfectly fine with minimal set up actually: Codewalker even programmed Paragon Chat to automatically create any conference room it needs to represent a zone if it doesn't already exist (your Paragon Chat account becomes the admin of that room as a side effect, but who cares on a dev server).
If you want a test environment, on Windows you can run Openfire, then run Paragon Chat, then run Paragon Chat a second time with -localdb and -localhost switches (so it uses different databases and different IP addresses to keep them separate), and have an entire server all to yourself with just two "players" logged in and your bot, so you can test players interacting with bots and bots observing players interacting with each other. It helps to have two players logged in sometimes because that way I can see precisely what Codewalker programmed PC to do - even if he makes a typo in his descriptions, Paragon Chat itself will tell you exactly what it was told to do.
Also it just occurred to me that I'm an idiot, because instead of looking at sample code or even the source code of sleekxmpp, what I should have been doing this weekend is realizing that xep 0199 (XMPP Ping), which I enable to satisfy Codewalker's xep prereqs, is actually a sleekxmpp plugin. Now I have to wait until I get home to look at that (I don't have my full sources at work, because if I did I'd never get anything done at work).
-
See, that's what I get for not looking up what SleekXMPP was. :P I presumed it was a different server you were using for a test bed.
-
See, that's what I get for not looking up what SleekXMPP was. :P I presumed it was a different server you were using for a test bed.
Should have been reading my dev commentary in between all those F5s.
-
Success! Sort of!
Apparently the reason why none of my stanza filters were working was because I was never getting the stanzas in the first place. Weirdly, the bot was logged into the meta channel as "unavailable." I'm not sure why, but it survived multiple stops and restarts. Must have been something sticky in Openfire, even though I have no code anywhere that ever sends unavailable Presence. Once I unstuck that, I actually started to get Iq/costume stanzas.
Which of course meant for the first time ever, the code I was using to parse them was tested, and of course absolutely none of it worked. It took a while to realize that sleekxmpp really, really wants you to reply to Iq stanzas, and not generate replies to those stanzas. Meaning: if you get an iq@type=get stanza X, bad things will happen if you try to create a reply Y that happens to have the right data, because sleekxmpp will never know what happened to X. It wants you to construct a reply Y that includes everything *except* the Iq wrapper, and do a x.reply().setPayload(Y.xml). Once I fixed that, and a few more squirrely nested stanza bugs, I started to get iq costume gets and replies working.
Its such a little thing, but an important one. Without any cheats whatsoever, I now have a visible bot:
(https://farm1.staticflickr.com/461/19689756580_6a43974cab_o.jpg)
The bot no longer relies on the cheat of using a costume identical to the player to be visible. It loads its costume from an I24 costume file, converts it into Paragon Chat data format, generates a hash, and sends the computed contents upon request, with no hardcoding whatsoever. There's a lot I want to experiment with, but first I'm going to clean up my code. There are now more DEBUG statements in the bot than actual code, all trying to track down my inability to process messages I was never getting in the first place. I should also loop back and try to figure out how many dozens of steps I skipped over in the XMPP protocol that might be causing issues, like sometimes logging in unavailable.
So let's see, from release to now ... thirteen days to make a visible Paragon Chat bot. That's embarrassingly long even averaging only a couple hours per night. I must be getting senile. Still, a bot in about thirty hours means I'm not totally useless yet. And the roadmap worked fairly well. So congratulations, Codewalker, your documentations works.
Man, if we had gotten this working when the game was still alive TopDoc would have farmed the crap out of the game with it.
-
So what can I do with costume stanza support. Well, I can do this:
https://www.youtube.com/watch?v=6WkWLy6wDsk
Its a little jumpy, but its interesting to me that at least some of the time the bot is both animating and resizing. It suggests there's a lot of room for improvement.
-
Pivoting away from troubleshooting mode back to implementation mode, now that I've had my fun with size manipulation, I'm going to go back to making a combat dummy. And it occurs to me that I'm basically going to refactor a lot of ancillary code again.
To be able to test the bot, I added keyboard controls. The bot is really a remote controlled drone. The problem is that the keyboard controls are a hack that directly fiddle with the bot's position and orientation. My goal is for the bot to be semi-autonomous, meaning I can give it very simple goals like "move to this position" and the bot will do so, stupidly at first and perhaps more intelligently later. This is important for a target dummy because the dummy needs to be able to do things like face an opponent, move towards it, and periodically initiate attacks. Scripting that is possible, but not optimal. I would like to be able to create goals and feed them into an execution engine like "move to X, turn towards Y, and execute animation Z." It would be really nice if goals could be nested under prereqs: Goal "Execute Animation Z" prereq "RangeTo Y 7 feet".
That's not difficult, but it means rethinking keyboard controls. Rather than have keyboard controls mess with the bot's position directly, keyboard controls should work with the goal seeking system by creating high priority tasks that basically tell the bot to honor the keyboard input. So if I press W, what the keyboard system should do is say "player is pushing FORWARD, I will make a high priority goal causing the bot to want to move to a location about ten feet in the direction of its current orientation." If the player releases W, I then revoke the goal (so the ten feet thing is really arbitrary, it just has to be larger than the maximum distance the bot could travel in one clock tick).
I'm thinking of making a pcGoal class, with three main elements: a prereq clause, an action clause, and a flags clause. I'm thinking of a few basic action primitives: MoveTo (location), ExecuteAnim (animation), TurnTo (target). At the moment two prereqs: RangeTo (target, val) (i.e range is less than val), CurrentLoc (loc) (i.e. I'm at loc within margin of error). And a few flags (Orientation? would disable the default behavior of facing in the direction of motion). Then some kind of structure to process them. Still thinking about the best way to do that, and whether I want to be able to execute more than one goal at a time (imagine a bot that had two goals: MoveTo location and ShootTarget. The bot could execute the first and keep checking the prereq of the second one, shoot when in range, and keep on moving.
The ski slope bot would not have been as fun as a goal seeking semi-autonomous pseudo-combat bot. Now if only I had gotten this working *before* Codewalker put the clamps on size scalers.
-
Hey, all. I've got a presence-related question.
About this:
A visible bot MUST announce Presence using <pc /> and <character /> stanzas in the pc:presence and pc:character namespaces respectively, when you login, when you zone, and when you change costumes. Presence will be automatically sent to new Paragon Chat users when they first login to Paragon Chat: you do not need to periodically announce Presence (pro forma XMPP, but mentioned here for completeness sake).
That "automatically sent" part - is that strictly true?
Your XMPP framework will send standard XMPP presence when someone new logs into your chat room, but don't you still have to track presence changes and send a copy of your bot's special Paragon Chat presence when you see someone new arrive?
"Automatic" in this context means that "you have to program your bot to automatically send presence when someone logs in", not "your XMPP client will remember your previously sent presence and repeat it whenever it sees someone new", correct?
-
No, per the MUC spec, chat rooms are required to keep a copy of the last presence stanza received from each member and send it to new members when they join. Once you send it to the room you never need to send it again unless its contents change, and doing so is just a waste of bandwidth.
"Special Paragon Chat presence" is standard XMPP presence, just with an extra namespaced XML element. There's no reason to ever send a presence stanza without that element if you are Paragon Chat or a Paragon Chat-compatible bot.
-
Speaking specifically about SleekXMPP, a ClientXMPP object can only connect to one room (well, not strictly true, but I wouldn't try anything else). So if you want your bot to zone into Atlas Park and actually participate in the Atlas Park broadcast channel, you need to join both atlaspark and atlaspark_meta. You need one ClientXMPP object for atlaspark and one for atlaspark_meta. In the atlaspark ClientXMPP object you would initially announce presence to myaccount@conference.server/CharacterName per xmpp standard. You would do similar in the atlaspark_meta room (as far as I'm aware, the nicknames can but don't need to be the same for connections coming from the same bot, although you would want your nickname in the broadcast channel to be what you want your chat to look like its coming from - no one sees the meta channel traffic).
Sample code tends to show the session_start function for ClientXMPP as doing a self.send_presence() and self.get_roster(). You would replace the "send_presence()" with your own function, or override send_presence before doing that.
Once you send Presence, the only time you need to resend it is if any parameter in the Presence message has changed, and the most likely reason for that to happen is if your bot changes costume. That changes the hash, and you need to send Presence again this time with the new hash in place of the old one (send the entire thing: don't just send the hash). Also, *initial* Presence should be sent to account@conference.server/nickname, but *updates* should be sent to just account@conference.server (no nickname)**. That's probably more standards stuff, but I checked the debug logs to see what Paragon Chat would do in that circumstance (costume change) and emulated that behavior. That's faster than reading the entire spec in many cases :)
An important rule to remember about XMPP is that (like most XML based protocols) XMPP is a take what you need, leave the rest behind protocol. In other words, <presence /> is a special XMPP message that has proscribed processing. There are special rules for how to handle that. But when Paragon Chat "piggybacks" its own data in presence stanzas, that doesn't change how presence is handled at all. Its still a <presence /> stanza, so XMPP servers will do exactly the same things to it that it would do to any other presence stanza. It caches presence from you. It sends it to people joining that context. When I log into Paragon Chat with pidgin, pidgin gets that same stanza packet. Pidgin has no idea what to do with <pc /> or <character /> but that's fine: pidgin takes what it needs, and leaves the rest behind. When I connect with Paragon Chat, Paragon Chat gets the exact same presence stanza but it actually looks for those <pc /> stanzas, and when it finds them it uses them.
This form of piggy backing is how XMPP can be extended in a way that applications can add features which everyone else will just ignore, while continuing to process everything else the same way it did before.
** Apparently when you change costume Paragon Chat sends two presence updates, one to the room and one with no destination - basically root. I don't think the second one is strictly necessary, but for now I'm emulating Paragon Chat's behavior until I get a chance to loop back around to looking it up
-
Also, *initial* Presence should be sent to account@conference.server/nickname, but *updates* should be sent to just account@conference.server (no nickname)**.
Oops, that's a mistake. Presence updates to rooms are supposed to be sent to your occupant JID with the nickname, not to the room JID. Openfire apparently does the right thing, but that needs to be fixed as other implementations may not.
** Apparently when you change costume Paragon Chat sends two presence updates, one to the room and one with no destination - basically root. I don't think the second one is strictly necessary, but for now I'm emulating Paragon Chat's behavior until I get a chance to loop back around to looking it up
That's because the costume hash is included in the presence. The one sent to the server itself is remembered by the server and is used for updates to people who have a presence subscription to you (i.e. you're on their friends list), and is broadcast to them when updated. If it weren't updated, the server would remember an old copy of it with a stale costume hash, and send that stale hash to your friends when they log in.
The costume hash could probably be omitted from the roster presence without ill effect, as it only needs the character name and map in order to show in the global friends list, but I was trying to keep the presence generation code somewhat generic instead of having so many special cases.
The very first proof of concept iteration actually included the full XML of the costume in your presence inside the <character>, but I very quickly decided that would be too unwieldy.
-
The one sent to the server itself is remembered by the server and is used for updates to people who have a presence subscription to you (i.e. you're on their friends list), and is broadcast to them when updated. If it weren't updated, the server would remember an old copy of it with a stale costume hash, and send that stale hash to your friends when they log in.
That's what I figured, although I wasn't certain. What I should have said is that I don't think its strictly necessary for bots, because I don't think too many bots would end up on friends lists. Although that's not a good assumption to make.
Its for the same reason that I currently skip a lot of the xmpp protocol like discovering conference server name. A bot writer wouldn't necessarily need to do that if they know what it is on the target server, although it would be nice to autodiscover. I don't do any significant disco of anything, because the bot isn't likely to operate in generic random environments at the moment although once again, at some point it would be nice to do. However, I'm making a tactical decision not to add too much code to autoperform things a bot writer would just know and be able to hard code in with a lot less code, because it makes the bot more readable. Eventually, I'll library-out more xmpp compliance stuff.
In fact, my bot currently can't zone. Its hard coded to go to Atlas Park. Not a problem at this stage of the game. But I am continuing to compile notes on what is necessary, and what should happen, for the guide to bot writing I'm still compiling. Which is getting longer, and longer, and longer (but still, honestly, not all that hard).
-
I generally test in Ouroboros instead, it loads much quicker than Nouveau-Atlas, resulting in shorter test cycles.
-
I generally test in Ouroboros instead, it loads much quicker than Nouveau-Atlas, resulting in shorter test cycles.
Hmm, didn't think of that. I guess my brain just defaulted to Atlas after years of thinking of that as the first CoH zone.
On the subject of testing, not directly related to bot implementation but it would be nice if /seqbits was reimplemented in Paragon Chat. That would make it much quicker to experiment with animations than editing emote.cfg and restarting Paragon Chat.
-
Quick progress report, not that I imagine anybody is waiting with bated breath,lol.
The bot can load a costume file, compute its hash,and save the resulting <Costume> stanza as a costume.xml file. I'm on the fence about whether this is a value add or not, but it does mean the bot can easily implement a costume cache if that becomes a useful thing to do.
The bot defaults to "reasonable initial values" for costume, zone, etc...; things that will be made configurable at some point. At startup, it reads a copy of zone.cfg and creates a mapping of chat room names to map files.
It initially joins Atlas Park (heh) as a "reasonable default", along with Paragon Chat.
When it joins any room, it grabs the name of the room and compares it to its list of zones. If the room name matches one of them, then it:
- Joins the appropriate metadata room
- Initializes a new presence stanza for that room
- Copies the default costume record to an array of costumes by zone chat room (so, a dict in Pythonese or an associative array anywhere else)
- Set's the costume attribute and loads up the new presence stanza with the <pc> and <character> stanzas, then saves the completed presence stanza in another dict keyed on zone chatroom name
- Sends the presence
When it leaves a chat room that matches a zone name, it also leaves the metadata room and it deletes the associated presence and costume stanzas.
So far, so good
Next steps -
Initialize the starting point on a per zone basis. This is a bit sticky as far as "reasonable default" values go. I might have to visit all of the zones and build a database of "default" locations for each one. Once I have that, then I'll have another mapping similar to the costume and presence mappings that hold a location+orientation per zone. (Eventually all of these individual bits will be normalized into a single record but for now the individual bits are easier to work with.)
Once the bot knows where it is in any given zone, then we start broadcasting <U> stanzas. Motion is a stretch goal here, so that will come later. I'll have to experiment with the polling routines I've got easily available but I'm thinking to emulate Paragon Chat - Watch for a presence change, advertise location, wait a semi-random amount of time, then check again.
Finally, setup handlers for the costume and bio IQ stanzas and reply to them as triggered.
The result of all of this is that the bot won't be a single avatar. It will be an avatar in every zone it joins, capable of carrying on conversations with multiple people at the same time. How that works out from a performance standpoint remains to be seen. However, at a bare minimum, the bot becomes a kind of "NPC Dispatcher". Without needing to move at all, it could greet a player as NPC A in Atlas Park, then as NPC B in Perez Park, then as NPC C in Galaxy City, all the while tracking the progress of the player from A to B to C and back again, while having all three avatars appearing to be in the game at the same time.
Voila. Quest chain.
How to use that creatively within the current constraints of interaction is a different matter, but the basics will be there.
The bot could potentially communicate with a website that records the player's activities and keeps a kind of "quest scoreboard" with the player's achievements recorded for everyone to see. It's not "inf" but it's a cookie for completing a "mission".
Anyway, that's where I'm heading with this. Motion is obviously an important stretch goal but it's still a stretch goal and I'm happy to let Arcana lead the way on the stuff that is undoubtedly going to get mathy pretty quickly.
The interesting thing to me here is that the bot is not an avatar or an NPC; it's the brain behind a stable of NPC's. How big a stable is difficult to guess at this point. Once it can carry on a conversation with one person, we'll see how well it does with multiple simultaneous conversations.
***EDIT***
One of the associated goals is to build a database of "points of interest". That is, a database of zones and locations labeled and possibly with a description. The goal would be to command the bot "!teleport Icon" and have it move the avatar for the current zone (whatever "current zone" means) to the front step of the local Icon store. Likewise, "!teleport Icon Steel Canyon" would lookup the avatar in that zone, add one if none existed, and place it in the appropriate spot.
-
Say....
Going along the same "bootstrapping" lines as wearing a costume you know is in cache:
Couldn't you setup your bot to record a series of U stanzas from a metadata room and play them back?
Your "real" City of Heroes client avatar could lay down the rails between point A and point B in a zone, and the stream of U stanzas it generates would then be the "beacons" that tell your bot how to travel on the rail. For most outdoor areas, flight would just be a matter of minorly adjusting the Y axis (yeah, that's a little weird but there it is) and using appropriate animations.
-
The interesting thing to me here is that the bot is not an avatar or an NPC; it's the brain behind a stable of NPC's. How big a stable is difficult to guess at this point. Once it can carry on a conversation with one person, we'll see how well it does with multiple simultaneous conversations.
SleekXMPP is threaded, which means every ClientXMPP runs its own stanza processing threads. If you load each ClientXMPP instance with its own message processing logic (or even copies of the same processing logic) then every ClientXMPP object servicing an XMPP room presence can carry on (essentially) simultaneous conversations.
One of the associated goals is to build a database of "points of interest". That is, a database of zones and locations labeled and possibly with a description. The goal would be to command the bot "!teleport Icon" and have it move the avatar for the current zone (whatever "current zone" means) to the front step of the local Icon store. Likewise, "!teleport Icon Steel Canyon" would lookup the avatar in that zone, add one if none existed, and place it in the appropriate spot.
One thing you should consider doing is having your bot respond to commands as tells. So when you're logged into Paragon Chat, you can send your bot a command like /tell SlickBot "SlickBot: !teleport Icon" and that SleekXMPP instance would then teleport itself to that location. Although I have keyboard controls to test things like movement (because it makes more sense), I can also send my bot tells (i.e. <message />) to do things. Helps a lot for testing, and would be interesting to give you a means to control the bot from within Paragon Chat itself. You can add a special directive that prevents others from sending the bot commands: I do that with a magic number: /tell ArcanaBot "ArcanaBot6871: anim ThunderKick" tells the bot to play the ThunderKick emote (defined in my emote.cfg) and the 6871 nonsense is a preprogrammed ID for the bot that is not broadcast to anyone.
-
Say....
Going along the same "bootstrapping" lines as wearing a costume you know is in cache:
Couldn't you setup your bot to record a series of U stanzas from a metadata room and play them back?
Your "real" City of Heroes client avatar could lay down the rails between point A and point B in a zone, and the stream of U stanzas it generates would then be the "beacons" that tell your bot how to travel on the rail. For most outdoor areas, flight would just be a matter of minorly adjusting the Y axis (yeah, that's a little weird but there it is) and using appropriate animations.
Case in point: you could program SlickBot to automatically record your position whenever it saw you do something, like send it a tell. So you could log in with Paragon Chat and run around, and when you wanted to record a position you could say "/tell Slickbot "Slickbot: record 'This is Atlas tram'" and it would record your jid, position, and message to a file (you could do this from multiple alts if you wanted to, or you could even have volunteers help, thus recording the source of the tell).
If you know the order you're going to visit locations and you don't need to record text and you just want to record a sequence of locations, and you want SlickBot to be really slick about it, have Slickbot watch for jump to show up in your MOV in the <u /> and when it sees that, record the location. Then you can run around and when you want to record a location, you can just hit the space bar.
-
One thing you should consider doing is having your bot respond to commands as tells. So when you're logged into Paragon Chat, you can send your bot a command like /tell SlickBot "SlickBot: !teleport Icon" and that SleekXMPP instance would then teleport itself to that location.
This is one area where building on an existing bot framework buys me something. My bot is being built as a plug-in to Errbot, a chat bot that runs on top of SleekXMPP and that actually supports around a dozen modular backends. If Codewalker took it into his head to switch to an IRC server, I could handle that with some minor reprogramming of the room logic.
Bot commands like !Teleport have a simple definition template and bang, they work. Likewise, they're configurable so that some commands are "administrator only" and others are "public".
I don't have to manage rooms, for instance. I just tell the bot to !room join atlaspark@paragon.chat.cohtitan.com and it handles the rest. My code only has to worry about handling the callback when a room is joined and doing something in reaction (like joining the appropriate metadata_room).
The downside is that it's another moving part to learn, particularly when I end up having to fix the occasional bug in Errbot, and when the interfaces between Errbot and SleekXMPP aren't documented in any way. Never mind that certain things still have to be done at the lowest levels of SleekXMPP's client interface so I still have to learn all that crap anyway, lol but I also have to do them in Errbot's XML stream and, did I mention about how well (ahem) some parts of it are (un)documented?
Yeah.
It'll all be worth it in the end but right now I'm in the middle of four simultaneous and criss-crossing learning curves between XMPP, Python, SleekXMPP and Errbot. How starry-eyed was I to imagine that I was going to just use an existing bot and "abstract away" all the nuts and bolts?
Yeah, heh
-
Let's say I have a bot that starts transmitting presence, messages, et al from two different JID's: bot@chat.cohtitan.com/ClarkKent and bot@chat.cohtitan.com/Superman.
Assuming that the presence, character, U stanzas, all of that are consistent and correct for the two different full JID's, will that cause Paragon Chat to display two different avatars in the zone or will it simply display one avatar who keeps changing his identity between ClarkKent and Superman?
-
Assuming you're using two separate XMPP connections (which you have to; the server won't let you spoof the 'from' address as a JID other than the one you logged in with), you'll get two separate avatars.
-
Let's say I have a bot that starts transmitting presence, messages, et al from two different JID's: bot@chat.cohtitan.com/ClarkKent and bot@chat.cohtitan.com/Superman.
Assuming that the presence, character, U stanzas, all of that are consistent and correct for the two different full JID's, will that cause Paragon Chat to display two different avatars in the zone or will it simply display one avatar who keeps changing his identity between ClarkKent and Superman?
It just occurred to me to ask a question: you're overriding Errbot's presence routines, right? Because presence is an integral part of XMPP chat, so Errbot by default probably sends presence automatically as part of its housekeeping. If you are constructing your own presence stanzas and sending those "raw" so to speak, strange things might happen if Errbot also sends presence stanzas because depending on who sends when Errbot's could override yours (you don't get the superset of both presence stanzas, you get whatever the last one was, period).
I ask because it just occurred to me that in your progress report you mention that you don't send <u /> stanzas yet, which means your bot can't be visible yet. Are you certain that your presence stanzas are actually working correctly, and you aren't sending stanzas you shouldn't be sending? I'm assuming you're testing all of this on your own Openfire server and can check things like whether or not Paragon Chat is happy with what you're sending by watching its debug logs even if you haven't gotten to <u /> stanzas yet. Also, I would have thought Errbot would automatically handle setting from in your Presence stanzas.
Also, Codewalker's point is an important XMPP point. Presence isn't just an information message, in many ways its almost like a pseudo-login name. XMPP connections are authenticated with a login name and password, but every time you join a room its Presence that says "this is who I am." Its not semantically correct to say "this is who I am, and also that is who I am." Your jid can be different in different rooms, because your nickname/resource can be different. But while Paragon Chat itself would be perfectly happy seeing you send two different Presence stanzas with two different jids, that's not kosher for XMPP itself and most servers won' t let you do that.
If you write your bot to make two separate XMPP connections, then of course anything it does in one of them would be independent of the second one, and that situation is indistinguishable from running two copies of the bot software itself.
-
It just occurred to me to ask a question: you're overriding Errbot's presence routines, right? ...(you don't get the superset of both presence stanzas, you get whatever the last one was, period).
So far, I haven't had to do that. It's been enough to establish a callback that happens *after* a room is successfully joined. This question is why I was asking recently about how Presence worked in certain situations. I was exactly concerned with the question you've raised.
My testing so far shows that as long as I establish a callback that triggers after the room has been joined and post my presence at that time, that my presence is the last one received by the server and that's the one that gets sent to other clients that join the room.
If I do establish that there are problems in this area then I'm prepared to copy the XMPP backend and hack on it to make a special Paragon Chat backend that inserts the presence at the lowest levels of the bot's communication framework.
Also, I would have thought Errbot would automatically handle setting from in your Presence stanzas.
Errbot handles things just fine but it's built on SleekXMPP so I can do anything to Errbot's client connection(s) that I can do to custom generated client connection. That means that I'm registering stanzas and writing handlers pretty much like you are probably doing it, but I'm accessing the code from higher up in Errbot instead of defining them explicitly as a SleekXMPP plugin. Handling Iq "get" requests will show whether that's going to continue to be a realistic way to handle things or if an explicit SleekXMPP plugin is going to be necessary.
In an ideal world, when this is all finished, I'd hand the package to someone and say, "Install X, Y, Z and you're golden" and they'd never have to learn any of this stuff.
If you write your bot to make two separate XMPP connections, then of course anything it does in one of them would be independent of the second one, and that situation is indistinguishable from running two copies of the bot software itself.
Yeah, and as you already mentioned, SleekXMPP is threaded so that's the "right" way to do such a thing. I have to ask about the lazy way, though. ;-)
It occurred to me that Paragon Chat uses the resource but the server mostly ignores it. If the server doesn't care what your resource is from minute to minute, and Paragon Chat treats the resource as a significant part of an avatar's "signature", so to speak, then it occurred to me that it might treat "bot@chat.server.com/Name1" and "bot@chat.server.com/Name2" as different identities and display them both. The fact that messages to both of them would be going to the same bare JID destination in the end would be irrelevant.
I expected to be told that it wouldn't work for the obvious reasons of a full JID being a full JID even if the server doesn't care what you assign to your resource when you login; it still wants it to stay constant for the duration of your session. But if that was *NOT* the case, well, that would make certain things a heckuva lot easier to implement.
-
Paragon Chat doesn't care what your resource is. It tends to use your character name as the resource for the sake of other non-PC clients, but it doesn't really care about what other players set it to. The only thing it uses the resource for is a unique identifier to tell multiple connections on the same account apart from each other.
Note that the server *does* care what your resource is from minute to minute. Namely, once you log in, that's your resource for the duration of the session, you can't change it.
You can change your nickname in a chat room, but that's equally as unimportant. You can't join the same room multiple times from the same "from" resource; if you try to join again with a different nickname, what you're actually doing is signaling that you want to change your nick.
-
If the server doesn't care what your resource is from minute to minute, and Paragon Chat treats the resource as a significant part of an avatar's "signature", so to speak, then it occurred to me that it might treat "bot@chat.server.com/Name1" and "bot@chat.server.com/Name2" as different identities and display them both. The fact that messages to both of them would be going to the same bare JID destination in the end would be irrelevant.
Hypothetically speaking Codewalker could have done something like that, but that would violate the XMPP standard, and break normal chat using that protocol. As a rule, I think its fair to state that in all respects Paragon Chat is a perfectly legal XMPP chat client that does everything basically by the book as a chat client, and only *extends* the protocol to add City of Heroes game client information, it doesn't break or otherwise change the protocol in any way. So if a chat client can't or shouldn't do it, Paragon Chat won't do it either (at least in terms of protocol: I'm excluding things like jid leakage as a question of best practice, not adherence to protocol).
-
Double-checking here -
Does this:
m is the current animation that is playing. This is the same MOV names that are used in demorecords. Paragon chat optimizes this by only sending a new MOV if the client couldn't reasonably predict it by using the "next move" and cycle info from the sequencers.
mean that Paragon Chat will interpret m="" to mean "do whatever you think is appropriate"?
So, if I send a U stanza like <U p='1.0 2.0 3.0' o='0.0 0.0 0.0' v='0.0' m=''> and Paragon Chat is already showing my avatar located at /loc=1.0, 2.0, 3.0 where it's standing around doing the arms crossed default emote, when it receives this stanza it will say, "Okay, keep standing there and keep doing arms crossed"?
Also, is there a slash command like /loc that shows your character's current orientation?
***EDIT***
I'm not really differentiating here between Paragon Chat proper and the CoH game client - Draw the lines between them as appropriate.
-
I caught this interesting bit on Reddit...by Leandro.
It's not something I've noticed posted on here...unless I missed it.
'Most of this work has already been done by Titan. Reverse-engineering the main protocol (the one that DBserver and map server use) was a nightmare, but it was finally 'done' mid to late last year. It has been partially implemented and in testing since. The first public release is happening very, very soon.'
...and...
'You can make the COH client connect directly to a DB server, skipping the authserver entirely and have full functionality.'
So. The 'umbilical' cord that talks to the CoH client has been reverse engineered? ie. The puppet strings can now by attached to a 'replica' server. In Paragon Chat's instance...a 'chat server.' And it will 'talk' to the CoH client?
In theory, if you have emulated the protocol that CoH used...you could attach the client to a real-time(?) server for 'combat'? Or create a local micro server unreal tournament style and have LAN or limited P2P combat?
I'm not technical. Stay with me... ;)
So...we have the replicated language of the protocol to talk to COH in the client/server relationship. That means, sure, we don't have the original server content. eg. the missions are all gone? But we can recreate missions if we unlock the AE confidentiality..? The original combat server is gone. But the combat system is well understood and can be replicated. Travel powers can be simulated. The powers can be replicated with combat dummies. NPCs are difficult but not impossible. So blocks of the game can be replicated or unbundled because we have the 'protocol'. ?
One idea I did have that dawned on me. 'Be NPC.'
Instead of 'A.I' spawns (though these will come a long time down the track?) you could have 'be npc' . Imagine 50 paragon chat players being 'NPC' outcasts and some paragon chat players being the 'pc'. You could...have 'real life' npc spawns as real players which would be far more unpredictable than A.I npcs. ie. PVP 'npc' vs hero style. It would redefine pvp. The 'red zones' on any given map...if you went into them. You could(!) be attacked. Green zone/tram lines neutral. Yellow zones 'sniped at.' An honour code pvp system. I always thought instead of separate pvp zones...City of Villains should just have been villains added to the normal zones but with an honour code to only allow 'pvp' in the red tram/tracks of each zone. IF you had 'be npc' for outcasts, trolls...clocks...you could have the players themselves(!) bringing 'mob' life to each zone. And as most mobs don't have super travel powers...it wouldn't be 'running' around combat pvp style...but you'd have 'force multiplier' player attacks on a hero.
'Real NPCs' vs hero PCs. To the untutored eye, Paragon City zones would live again. All that would be needed would be missions. And they could be replicated in a similar style.
Anybody have any thoughts on this?
If PVP has to come before A.I combat...then I thought my 'be npc' idea could replicate 'spawns' in a 'real' way in the interim.
Azrael.
-
I think the thing there is that some parts have been reverse engineered and some haven't. Combat and powers would be two of the things that haven't, and even without any knowledge of how the client actually works I could see those being massively harder to do than simple movement and chat.
In essence, we have some of the language used between the client and the server. But there's still many words we don't have.
-
I'm behind the curve because I only heard about Paragon Chat earlier this week (I used Icon last year, but I've been a bit distracted with various issues and other projects). Nevertheless, I'm diving in feet first and poking around with my own Python bot. This is the first successful test from my Costume module. I'm trying to make sure I'm getting the same results from the costume Arcana was testing earlier in the thread
Validator concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============
My concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============
Congrats! Concatenations match!
Validator Hash (P.C.):
=============
2TjGyYMQNC6GHr7gI5hlaNm1/+E=
=============
Validator Hash (Arcana):
=============
ycY42S40EIPgvh6GaGWYI+H/tdk=
=============
My Hash:
=============
ycY42S40EIPgvh6GaGWYI+H/tdk=
=============
Congrats? Hash matches Arcana.
Paragon Chat Hex:
=============
c9c638d92e341083e0be1e8668659823e1ffb5d9
=============
Arcana Hex:
=============
d938c6c98310342e861ebee023986568d9b5ffe1
=============
My Hex:
=============
c9c638d92e341083e0be1e8668659823e1ffb5d9
=============
Figured I'd get this bit out of the way before trying to tackle actual XMPP functionality
It's a start, anyway.
-
In a reminder about the difference between looking at forests and looking at trees - When everything looks right and it's still not working, ask yourself if something unrelated to your assumed causes could be wrong...
As it turns out, class="Blaster" is not actually the same as class="Class_Blaster". /headdesk
On the plus side, my bot is visible. I probably need to learn adjust my YPR order, though.
(https://dl.dropboxusercontent.com/u/18795363/leaning.jpg)
Anyway, next up - IQ costume get requests
-
Double-checking here -
Does this:
mean that Paragon Chat will interpret m="" to mean "do whatever you think is appropriate"?
So, if I send a U stanza like <U p='1.0 2.0 3.0' o='0.0 0.0 0.0' v='0.0' m=''> and Paragon Chat is already showing my avatar located at /loc=1.0, 2.0, 3.0 where it's standing around doing the arms crossed default emote, when it receives this stanza it will say, "Okay, keep standing there and keep doing arms crossed"?
Also, is there a slash command like /loc that shows your character's current orientation?
***EDIT***
I'm not really differentiating here between Paragon Chat proper and the CoH game client - Draw the lines between them as appropriate.
I'm not in a position to test today as I'm in Hilo, but in general it is best if the attribute is null to just not send it at all. However, I believe that whenever Paragon Chat doesn't understand what you sent, it ignores it. That's probably the same behavior as when the attribute is null.
But what Codewalker meant was that Paragon Chat has the same animation sequencer that the client has, so for example if your intent is to play SOME_ANIMATION_PRE and the sequencer says the next animation in the sequence is SOME_ANIMATION_CYCLE, Paragon Chat doesn't send m="SOME_ANIMATION_CYCLE" because it knows the game client will do that anyway, and your bot doesn't have to do that either.
-
Anybody have any thoughts on this?
I have some thoughts on this. Thus, all my posts in this thread. Given the current architecture of Paragon Chat, a central server is possible but would be fighting upstream against the system design. But bots could replicate much of the capabilities of a mapserver, if and when Codewalker adds the appropriate XMPP message support for those features. It then becomes a question of implementing a bot that uses those things in the right way. Paragon Chat itself will likely never get all the way there, but its an interesting question to ask how far can it get.
When Paragon Chat gets as far as it will ever likely get and Codewalker open sources the software, other things may become possible for enterprising software writers.
-
It's a start, anyway.
We all started from the same blank page. The important thing is to find the learning curve you're comfortable with.
PS: make sure you pay attention to all the discussion that followed that post. As it turns out, Codewalker byte-swapped his hash so Paragon Chat's hash is different from what I originally computed and what Codewalker originally documented, but with the appropriate byte swap they match. Also, *my* costume file doesn't match what the client and Paragon Chat import, and I had to delete two colors from it for the "in-game" costume to match the contents of the file itself. With those two corrections, Codewalker and I compute identical hashes.
-
On the plus side, my bot is visible.
Gratz. Visibility is a key milestone, because now you can *see* what you're doing right and wrong.
Also, I made the same silly mistake with Class names about a hundred times. Although in my case I made that silly mistake back in the I10-ish timeframe and got it out of my system long ago, about the same time I learned to live with "Gadgets."
-
Thanks. I don't think it will boil down to so simple a fix for whatever is wrong with my IQ costume stanzas. I'm thinking I'll have my char wear the test costume on cohtitan's server and record what paragon chat sends to another paragon chat client, then compare that to what paragon chat is receiving is receiving from my bot's IQ result.
-
I'm not in a position to test today as I'm in Hilo...
"From the mind that brought you 'Hilo'." (https://youtu.be/4PwMWyu8BcU)
-
PS: make sure you pay attention to all the discussion that followed that post. As it turns out, Codewalker byte-swapped his hash so Paragon Chat's hash is different from what I originally computed and what Codewalker originally documented, but with the appropriate byte swap they match. Also, *my* costume file doesn't match what the client and Paragon Chat import, and I had to delete two colors from it for the "in-game" costume to match the contents of the file itself. With those two corrections, Codewalker and I compute identical hashes.
Yep, I read through the entire thread (as well as the Coming Soon thread that preceded it). How much of the thread I can manage to keep straight/retain is a different question, of course. :)
I hadn't gotten around to the endianness issue yet; I was happy enough to get concatenation working. Once I did get there, I got stumped for awhile because my brain blew a fuse and I couldn't figure out how swapping the bytes on the base64 hash was going to produce anything at all correct.
Yeah.
I forgot that the base64 hash is not the equivalent of the hexdigest, the digest is. The base64 is just a rendition of the latter. It's the (non-hex) digest that needs to be byte swapped, not the hexdigest (which was obvious) nor the base64 hash (which was where my brain faceplanted and started calling for rez).
Just ignore me, I'll be over here slamming my head against the desk.
-
Just ignore me, I'll be over here slamming my head against the desk.
At least Slickriptide can probably tell you which orientation parameter to use to do that.
-
At least Slickriptide can probably tell you which orientation parameter to use to do that.
LMAO.
In other news - I decided to use the -pcdata -localhost trick to login a second client and find out what Paragon Chat was sending back and forth in the costume stanzas. Imagine my surprise when I saw this on the second client.
(https://dl.dropboxusercontent.com/u/18795363/PC_Bot.jpg)
Only thing I can figure is that the first client must have accidentally /ignored the bot or something. I'm not sure how to clear an ignore list. Anybody know?
So, Phase One is complete. The bot can manifest itself in Paragon City and it can even return its bio.
I guess the next thing is to figure out whether motion or chat interactivity is higher on my priority list.
-
I guess the next thing is to figure out whether motion or chat interactivity is higher on my priority list.
You should break "motion" down into two parts, one easy and one not so easy. The easy part is that if you don't need to "move" but you do want to animate, that's easy. Just send the appropriate emote MOV and your bot will perform the proper animation. You could then connect that to chat: when someone sends you a chat message, you could respond by both sending a chat reply and doing an appropriate animation: saluting, waving, holding signs, barfing, whatever.
"Motion" interpreted to be trying to get your bot to travel from point A to point B in a way that is both smooth in terms of change of location and orientation, combined with the appropriate animations that make you seem like you are walking/running/flying and not just gliding there in mid-air like Magneto, is a little bit harder to do in the general case. It involves combining acceleration (so you don't instantly start moving at full speed), changing orientation smoothly (so you face in the direction you're moving, assuming you want to do that), and executing the right animations at the right time to look like you're walking/running/flying and *also* you stop doing that when you stop moving so you don't keep moonwalking in place. For what you're apparently trying to do, that seems to be an optional stretch goal since traveling seems less important to the kinds of interactivity you are interested in replicating.
But barfing on someone's shoes, that I think you should try to implement.
-
Shoe barfing. Definitely need to move that to the top of the list. ;-)
Movement-wise, I could see experimenting with a "follow" command where you can tell the bot to follow and it literally copies your <U> stream as it receives it from you, which should result in the bot mimicing your route, velocity, vector of travel, etc...
Travel animation is going to require some sort of time coding to mark the transition between various parts of a travel animation; Pre, Cycle, and Post as examples.
I think my first order of business may be to convert all of my hard-coded initialization into configurable ErrBot parameters with an appropriate requirements file for initial state. After that, define a bunch of basic utility commands like "change costume" and "relocate to point XYZ" and "change Nick" and etc... Once I've got that stuff in the state I want, I can setup a GitHub for a Paragon Chat plugin and make it so that anybody who installs ErrBot can just command ErrBot to download and run the Paragon Chat plugin.
At that point, "phase one" will truly be finished and "phase two" will commence. In an ideal world, I'd have all kinds of bot functionality, broken up into groupings of features that can be loaded, run and unloaded as needed or desired. My pie-in-the-sky goal is not so much to have a bot as to have a NPC management platform. That's going to involve a lot of record keeping about the state of a NPC as well as some sort of simple scripting language to handle various sorts of NPC activities and decision making.
I suppose that somewhere in there I'll have to put up a Markov chain powered chatbot also, heh, just to see how people react to something like that inside of a graphical game environment.
-
Travel animation is going to require some sort of time coding to mark the transition between various parts of a travel animation; Pre, Cycle, and Post as examples.
For most movement animations, the _pre animation automatically executes the cycle animation after it completes, and the cycle animation automatically cycles. So you only need to send _pre to start and _post to end and cycle happens in the middle without your intervention. Because _pre is always interruptable (so far as I've seen), you can always send _post when you stop moving even if you are in the middle of _pre. There are corner cases, but that's the basics.
-
For most movement animations, the _pre animation automatically executes the cycle animation after it completes, and the cycle animation automatically cycles. So you only need to send _pre to start and _post to end and cycle happens in the middle without your intervention. Because _pre is always interruptable (so far as I've seen), you can always send _post when you stop moving even if you are in the middle of _pre. There are corner cases, but that's the basics.
Groovy. Thanks for the helpful advice along the way, Arcana, and for sharing your experiences as the bleeding edge. It was immensely helpful at times to have an example to compare to or even just an assurance that, "Yes, this can actually be made to work".
Likewise, thanks Codewalker for the specs to make it all happen.
On a different note, I've found a reliable crash scenario for Paragon Chat v. 0.98beta. It involves a long message being sent. Specifically, this one:
2015-07-29 07:00:16,545 DEBUG SEND: <message type="groupchat" xml:lang="en" to="atlaspark@conference.cjj-pc"><body>Available commands for ErrBot:
!about: Returns some information about this err instance
!apropos: Returns a help string listing available options.
!blacklist: Blacklist a plugin so that it will not be loaded automatically during bot startup
!config: configure or get the configuration / configuration template for a specific plugin
!echo: A simple echo command. Useful for encoding tests etc ...
!export configs: Returns all the configs in form of a string you can backup
!help: Returns a help string listing available options.
!history: display the command history
!import configs: Restore the configs from an export from !export configs
!load: load a plugin
!log tail: Display a tail of the log of n lines or 40 by default
!reload: reload a plugin
!repos export: Returns all the repos in form of a string you can backup
!repos install: install a plugin repository from the given source or a known public repo (see !repos to find those).
!repos uninstall: uninstall a plugin repository by name.
!repos update: update the bot and/or plugins
!repos: list the current active plugin repositories
!restart: restart the bot
!status gc: shows the garbage collection details
!status load: shows the load status
!status plugins: shows the plugin status
!status: If I am alive I should be able to respond to this one
!unblacklist: Remove a plugin from the blacklist
!unload: unload a plugin
!uptime: Return the uptime of the bot
!zap configs: WARNING : Deletes all the configuration of all the plugins</body></message>
This is a help message from ErrBot. I send "!help ErrBot" to the bot and it replies with the above message.
The three test cases are as follows:
1) /tell @errbot, !help ErrBot -- performs as expected
2) With paragonchat set as the "current" channel, enter !help Errbot -- An error "Parameter too long" is generated and the entire message, preceded by the error message, is received as a "tell" (or at least it's in yellow text like a "tell").
3) Send on broadcast, /b !help ErrBot -- Paragon Chat crashes to desktop.
Most likely a buffer overrun or something like that. If you need any more info about it, let me know.
-
Paragon Chat probably doesn't bounds check everything perfectly since normally the City of Heroes client itself has enforced limits on things like the maximum length of a tell. Its probably a good idea to try to stay within those bounds ourselves, although given the fact that we're not the only people that will eventually figure out how to do this, its also probably a good idea for Paragon Chat to bounds check all XMPP messages just for anti-griefing purposes alone.
-
I agree, @Arcana, and there are steps I can take if that's a necessary thing. I'm pretty sure this is new behavior,though. I don't remember any previous versions of PC crashing just from listing a help menu in the bot. I wasn't actively testing for that kind of thing, so I can't state categorically that it worked previously but I've used those menus before this time and I have to think that at least once would have been in the Paragon Chat channel if not in Broadcast.
-
I agree, @Arcana, and there are steps I can take if that's a necessary thing. I'm pretty sure this is new behavior,though. I don't remember any previous versions of PC crashing just from listing a help menu in the bot. I wasn't actively testing for that kind of thing, so I can't state categorically that it worked previously but I've used those menus before this time and I have to think that at least once would have been in the Paragon Chat channel if not in Broadcast.
Could it be this is the first time you did this with a visible bot? Some aspects of chat might be handled slightly differently when you are visible verses not visible or not on the map. If I get a chance I might try this tonight, since I have a mostly-complete library of different Paragon Chat versions (all the numbers, not necessarily all the letters).
-
Ah, you're thinking chat bubbles might be involved?
I guess that 0.98beta would be probably be the first time with a visible bot.
Chat bubbles are probably a good reason to consider limiting message size regardless.
I guess we'll see what Codewalker thinks next time he stops in.
-
Arcana, I need your math whizdom.
Let's say I want to order my bot to turn and face me, wherever I happen to be standing.
Simple, right? It would be for human. For a bear of little brain, it's a bit more difficult.
Here's what I think needs to happen:
1) I move to a random X,Y,Z point in the zone.
2) I send the command "Errbot, face me".
3) The bot now assumes a facing of zero. This seems more useful than trying to figure out where things are in relation to its own random orientation. For the sake of discussion, we'll call this facing "North". It doesn't matter what the compass on a map of Paragon City would call it.
4) Errbot draws a line segment between itself and my X,Z position. (In Paragonspace, Y is the height axis. We don't care about height for this.)
5) Errbot now draws another line segment of equal length towards "North".
6) Errbot now has three reference points. Itself and the two endpoints. Using those, it can compute the angle between North and me. Using that angle, it should be able to compute the number of radians in an arc between those two endpoints.
7) Errbot sets its facing equal to the results of step 6.
8) Fini
Does that sound right?
-
Actually, you only need one directional vector (and rotation order) to calculate orientation. You are at (X,Y,Z). Your target is at (X',Y',Z'). Your directional vector is (X'-X,Y'-Y,Z'-Z) which we will abbreviate as (x,y,z).
According to Codewalker, City of Heroes (and Paragon Chat) apply rotations in YPR order. So we will do the same. We start by computing the yaw angle which is in the XZ plane. Given the projected vector (x,z) you can use basic trig to calculate the yaw angle for that vector in radians. Then, given the triangle in that vector's plane composed of the length of (x,z) which is sqrt(x^2+z^2) and the inclination height y, you can calculate the pitch angle up or down.
You never specifically need to change roll angle to face any target in space (which may be why the devs decided to use YPR order). In YPR order, roll would spin you around the axis that connects you and your target without changing the fact you are continuously pointed at it (obvious, since roll is defined loosely as rotation about the axis you're facing, it doesn't change the direction you're facing).
-
"Motion" interpreted to be trying to get your bot to travel from point A to point B in a way that is both smooth in terms of change of location and orientation, combined with the appropriate animations that make you seem like you are walking/running/flying and not just gliding there in mid-air like Magneto, is a little bit harder to do in the general case. It involves combining acceleration (so you don't instantly start moving at full speed), changing orientation smoothly (so you face in the direction you're moving, assuming you want to do that), and executing the right animations at the right time to look like you're walking/running/flying and *also* you stop doing that when you stop moving so you don't keep moonwalking in place.
For myself, I'm going to attempt to adapt the basic acceleration logic of the elevator system I'm making in UE4 to handle some of that, since I already have that working.
In other news:
Hard-coded concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============
Generated concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============
Congrats! Concatenations match!
PChat's Hard-coded Base64 Hash: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=
Arcana's Hard-coded Base64 Hash: ycY42S40EIPgvh6GaGWYI+H/tdk=
Arcana's Generated Base64 Hash (byteswapped): 2TjGyYMQNC6GHr7gI5hlaNm1/+E=
My Generated Base64 Hash (byteswapped): 2TjGyYMQNC6GHr7gI5hlaNm1/+E=
My Generated Base64 Hash (normal, for comparison): ycY42S40EIPgvh6GaGWYI+H/tdk=
A WINNER IS YOU! Byteswapped Hash matches Paragon Chat.
Pro-tip: In Python 3.3, you need to slice byte objects like foo[0:1], not merely index them like foo[0]. Attempting to byte swap a 20 byte digest with the same logic that worked under Python 2.7 first required dealing with a type mismatch (the index version returns an int under 3.3) and then would only give me 2765 byte output.
-
Well, Arcana beat me to it, but I was going to describe the process in a slightly different way.
Step 1: Subtract the bot's coordinates from the target's coordinates. That effectively gives you the target's position in what we'll call "bot-space" -- a coordinate system in which the bot is the center of the universe.
Step 2: Normalize the resulting vector. That means calculate the magnitude of it (sqrt(x^2 + y^2 + z^2)), and divide each coordinate by that magnitude. What that gives you is a unit vector pointing at the target. This is also known as the desired forward vector, since it's the direction you want to face.
Step 3: Use any of the well-known formulas to convert a unit vector to YPR. Assuming my trig is right (please feel free to verify and correct if necessary), you can get yaw and pitch from your vector like this:
P = asin(y)
Y = atan2(x/cos(P), z/cos(P))
R = 0
You'll need special handling for y=+/- 1 to avoid a division by zero. In that case just set P=+/- pi/2 (90 degrees) and Y=atan2(-z,x).
To deal with roll you would need a full rotation matrix, or at least both a forward and an up vector, but you probably don't have any reason to need it so you can simply leave it zero. I got the idea of dividing by cos(P) from this publication (http://staff.city.ac.uk/~sbbh653/publications/euler.pdf), which goes into a much more detailed discussion of methods for extracting Euler angles from a rotation matrix (of which the 'forward' vector we're discussing is the third row that defines the Z axis). Though be aware that it describes PYR order and needs some adjustment for YPR.
At some point in the future I'll probably add protocol support for optional alternative methods of specifying orientation that may be more convenient than Euler angles. Likely quaternions. In that case it's relatively simple to convert the forward unit vector into a quaternion; just a cross product to get a parallel vector and a dot product for the magnitude of the rotation.
-
Pro-tip: In Python 3.3, you need to slice byte objects like foo[0:1], not merely index them like foo[0]. Attempting to byte swap a 20 byte digest with the same logic that worked under Python 2.7 first required dealing with a type mismatch (the index version returns an int under 3.3) and then would only give me 2765 byte output.
One of the many reasons why I tend to still use Python 2.x. Byte objects are strings in Python 2.x but integer sequences in Python 3.x. That means you have to deal with the craziness that is the fact that foo[0:1] is a sequence of bytes (with exactly one item) but foo[0] is the first integer in the integer sequence that comprises the byte (type) object. In Python 2.x, foo[0] is a character - which is essentially a byte - and foo[0:1] is a character sequence of length 1, which is also a byte.
-
Step 3: Use any of the well-known formulas to convert a unit vector to YPR. Assuming my trig is right (please feel free to verify and correct if necessary), you can get yaw and pitch from your vector like this:
P = asin(y)
Y = atan2(x/cos(P), z/cos(P))
R = 0
You'll need special handling for y=+/- 1 to avoid a division by zero. In that case just set P=+/- pi/2 (90 degrees) and Y=atan2(-z,x).
I *think* I did something like this (code not in front of me):
k = sqrt(x^2+z^2)
k' = sqrt(x^2+y^2+z^2)
Y = acos(x/k) * sign(z)
P = asin(y/k')
Technically I haven't tested my "go run that way" code yet because I was away from computer for an extended weekend, so hopefully my trig has survived thirty years of degradation. I will probably be testing limited movement scripting this weekend (also, I'm putting off loading and collision checking maps because ick).
-
I decided to do a quick hack to test orientation by adding a "/faceme" command to the bot. As it turns out, there was some flipping and such required:
x1 = self.position[0]
y1 = self.position[1]
z1 = self.position[2]
x2 = float(self.cmdQueue[0][1].split()[0])
y2 = float(self.cmdQueue[0][1].split()[1])
z2 = float(self.cmdQueue[0][1].split()[2])
xx = x2-x1
yy = y2-y1
zz = z2-z1
k = math.sqrt(xx*xx + zz*zz)
k2 = math.sqrt(xx*xx + yy*yy + zz*zz)
yaw = math.acos(zz/k) * cmp(xx,0)
pit = -math.asin(yy/k2)
rol = 0.0
Should be fairly self-explanatory.
-
Fortunately (or not) I'm cutting my teeth on Python 3 so it's way is the only way, I know, heh.
-
Incidentally, there's what's mathematically true, and what's mechanically logical. When you tell the bot "/faceme" the math says to turn directly towards the coordinates of the target, but that's not necessarily what makes sense to do. For example:
(https://farm1.staticflickr.com/499/20152982901_d808c297b9_b.jpg)
A person would look up. A person would not tilt backwards to stare upward. So separate from the question of how to write the code, and separate from the question of how to do the math, is always the question of what you actually want to do. Do you really want to face the target in three-space (maybe, if you were flying) or do you want to confine your orientation to the xz plane (which makes more sense if you are standing).
These are, in fact, the kinds of questions game programmers have to answer also.
-
Fortunately (or not) I'm cutting my teeth on Python 3 so it's way is the only way, I know, heh.
I cut my teeth on Python zodknowswhat back in the early-mid 90s. I code stuff so infrequently that I'm still thinking of parts of Python 2 as fancy and new-fangled. :-[
But seriously, if you deduce from my previous post that I still haven't used Python 3 very much, you'd be correct.
-
I've got an odd problem that's caused me some trouble.
I have two installations of Tequila/I-24/Paragon Chat on two different computer, each with its own Openfire server. They share my python folder via dropbox, so they typically have the same bot code on each (not always, but usually).
My bot works perfectly on the one. The other shows it sometimes and then stops showing it unless it's being viewed on a new account with no cache/history. Then once the bot goes into cache, if it logs off it is gone for good. No amount of restarting of the bot will cause it to show up for that client. Sometimes restarting the Paragon Chat/CoH clients will cause the bot to reappear, but just as frequently the bot will remain invisible through restarts of the entire software suite.
This was a big part of the reason I thought my bot was failing when it was actually working as intended. I just couldn't see it until I moved to the other computer to test on.
I don't really have a clue what's going on except that it seems like it could be peripherally related to the costume cache in the client. However; given that the other computer and its clients work beautifully, I can't really point to anything in Paragon Chat as the cuplrit, other than to note that it seems to happen most frequently after the bot costume is "known" by the client.
Since I'm not even positive that it is an issue with Paragon Chat, I'm not sure what to look at as far as fixing it. I might have to just stop using that computer for bot development and stick to just the one that properly displays things like they ought to be displayed.
Any suggestions, Codewalker? There's nothing in the debug logs of either the bot or of Paragon Chat that suggests that anything is out of whack.
-
One problem I've encountered with sleekxmpp is that if you create a bug that crashes a sleekxmpp thread, other threads can sometimes survive the death of the bot. When that happens, I end up with zombie python.exe processes that maintain an XMPP connection. Then when you try to restart the bot, Openfire doesn't let you log in properly with the same name and resource. A complete implementation of XMPP would first ask if the resource was available, but I don't do that. So you end up with the bot essentially trying to log in with credentials that are already in use. Openfire effectively "kicks" you off the server by announcing you as "unavailable." Paragon Chat sees that as a log out, and what happens is my bot appears briefly (because I send a valid presence and location stanza) and then disappears.
When that happens, I check to see if I have rogue python processes, and I also check to see if I stop the bot do I still see sessions in the Openfire console. If I do, then I know zombies are interfering with the bot. I have to kill those python processes and sometimes also reset those Openfire connections (killing usually does that for me, but once it didn't). Then when I log in cleanly, the bot starts working correctly again.
That doesn't sound precisely like the problem you're having, but perhaps it might spark an idea or two.
-
Paragon Chat probably doesn't bounds check everything perfectly since normally the City of Heroes client itself has enforced limits on things like the maximum length of a tell. Its probably a good idea to try to stay within those bounds ourselves, although given the fact that we're not the only people that will eventually figure out how to do this, its also probably a good idea for Paragon Chat to bounds check all XMPP messages just for anti-griefing purposes alone.
It is a maximum length issue, but it's not a bounds checking / buffer overflow problem per se, so things like remote code execution aren't possible. It actually is doing bounds checking on the buffer, however the default behavior unless otherwise specified is to assert() on exceeding the space available. That's a fail-safe as silently altering input when not expected could result in undefined behavior.
In this case, truncating overly long chat messages is harmless, and being able to remotely trigger an assert() is not so good, so I'll fix those cases to just chop off any extra.
I agree, @Arcana, and there are steps I can take if that's a necessary thing. I'm pretty sure this is new behavior,though. I don't remember any previous versions of PC crashing just from listing a help menu in the bot. I wasn't actively testing for that kind of thing, so I can't state categorically that it worked previously but I've used those menus before this time and I have to think that at least once would have been in the Paragon Chat channel if not in Broadcast.
Private tells and global channel messages were already being truncated if they exceed the maximum message length. Only broadcast and local were triggering the assert. That'll be fixed in the next incremental update.
-
Private tells and global channel messages were already being truncated if they exceed the maximum message length. Only broadcast and local were triggering the assert. That'll be fixed in the next incremental update.
In the meantime, I can configure ErrBot to break up messages that exceed a given size. What is a good message length to use to stay under the assertion/truncation limit?
I gave the bot an emote command last night and I was playing with having it perform lots of different emotes. (A strangely empowering experience. Dance, Puppet, Dance!) I wanted to double-check about the whole sequencer thing. All the bot ever sees from the game client on the wire is a MOV. That's what a bot will always choose to send on the wire itself in order to play an animation, correct? A bot doesn't know or care about the game client's sequencers, at least not at this point in time.
-
Well, the in-game chat window doesn't let you type more than 256 characters I think.
Paragon Chat's internal buffer for chat messages is 1024 bytes and is what will assert (or truncate in future versions). That's what it uses to format the entire message "Character: text" or "@global: text", however, so the maximum message length varies depending on the length of the sender's name.
Global Channels operate over an entirely different system, using so-called shardcomm commands that work differently from the rest of the game. Theoretically, those can be up to 8192 bytes in length, so longer messages are possible there. That 8192 must also fit the text "ChanMsg", the name of the global channel, the sender's global name, the message, and quote marks around each of those. Various characters are also escaped to two-byte sequences, for example ' becomes \s and " becomes \q.
To be on the safe side, I wouldn't send messages longer than 900 bytes or so. 512 if you want to be conservative.
-
Just to be clear, you don't need to worry about any of the shardcomm or escaping stuff when writing a bot. Paragon Chat takes care of all those details; all you need to send it is well-formed XML.
-
I gave the bot an emote command last night and I was playing with having it perform lots of different emotes. (A strangely empowering experience. Dance, Puppet, Dance!) I wanted to double-check about the whole sequencer thing. All the bot ever sees from the game client on the wire is a MOV. That's what a bot will always choose to send on the wire itself in order to play an animation, correct? A bot doesn't know or care about the game client's sequencers, at least not at this point in time.
At the moment, all a bot needs to send are MOVs. All a bot will ever see another entity perform are MOVs. You don't need to understand sequencers to animate. However you do need to understand sequencers if you want to know what an entity is doing right now.
For example, suppose your bot is logged into the same zone as my bot, and my bot decides to walk towards you. You'll see me do a m="PLAYER_WALK_FORWARD" and then a lot of <u /> messages will show I'm moving towards you and when I stop you'll see me do a m="PLAYER_WALK_POST". All good. But what you won't see is that for most of the time I'm walking any Paragon Chat user will see my bot playing "PLAYER_WALK_FORWARD_CYCLE". That's because according to the CoH sequencer system, PLAYER_WALK_FORWARD_CYCLE should be played immediately following the completion of PLAYER_WALK_FORWARD and should itself be continuously looped until interrupted. Paragon Chat never sends such information because its not necessary due to its ability to execute the sequencer, and as a consequence my bot never sends that CYCLE MOV either. Because there are animations that get played but never sent on the wire, *if* your bot needs to know if a player ever performs a specific MOV, its not enough to simply check for it on the wire. You would technically need to make sure that no entity played an animation that eventually leads to that MOV also, and you would need to know how long each MOV takes from here to there, so you know *when* that MOV ultimately comes up.
If your bot is not going to react to specific MOVs played in its presence, then its not necessary to know if any specific MOV is being played at the moment, and thus its not necessary to understand how the sequencer affects future animations. If at some point you do want to do that, then you would need to know at least some of that information.
Separately, while you don't *need* to understand the sequencer itself, knowing which animations follow which is useful if your bot moves, for economy. Meaning, if I *tell* you that the walk sequence is PLAYER_WALK_FORWARD -> PLAYER_WALK_FORWARD_CYCLE -> PLAYER_WALK_FORWARD_CYCLE, PLAYER_WALK_POST, then without necessarily knowing anything about the *specifics* of the CoH sequencer you could still use that knowledge to make your bot walk by knowing you only need to send the first and the last and not worry about the middle. So if you want your bot to play animations with multiple sequence entries, knowing how that specific sequence works can be helpful. But that doesn't explicitly require that your bot understand the sequencer. Only that you do when you write the bot.
-
Yes, for starters the bot just needs to be able to walk around Atlas Plaza but the end game will need to be an emote database that knows the full sequence of any arbitrary animation it might play.
-
Yes, for starters the bot just needs to be able to walk around Atlas Plaza but the end game will need to be an emote database that knows the full sequence of any arbitrary animation it might play.
When you get there, there's a tiny little catch to being able to fully predict when an animation will play. If you simply take the sequencer dump Codewalker posted on the forums a while ago, you won't even need to decode the piggs to get a complete set of at least the MOVs that players can actually play (there are animations that players cannot play that aren't in there, but they aren't relevant to player-lookalike bots). The sequencers tell you to play this animation, give a start and end frame, and then tell you what to do next. Since an animation frame is 1/30th of a second, you can calculate at what precise moment the sequencer will start playing the next animation**.
Unless the sequencer says the end frame is zero. In that case, the sequencer plays the animation until the end, and then moves on. But how will you know how long that is? The answer is: you'll have to look up the actual .anim file in the piggs which contain that animation and figure out how long that is. And you cannot predict precisely from the size of the file. Fortunately, I discovered that if you look hard enough the actual frame count for the animation is in that file. Unfortunately, that information is located somewhere in an animation comparison script I can't find at the moment. When I get there, I will go looking for that. Theoretically speaking, all you need to do is find the .anim and read byte #XX, and that's the frame count. For the correct value of XX. This was something I needed to figure out how to do when I decided to figure out how long combat powers actually rooted the player for, and if that was consistent with what we were told (it was not in all cases).
At some point in this process, a technically proficient person will either guess something or experiment and discover something. The answer is yes I know, and I think for now its best that the only people that know this are the people that can figure it out for themselves. It serves no useful purpose for either Icon or Paragon Chat.
** I'm ignoring things like scale here for simplicity sake
-
The answer is: you'll have to look up the actual .anim file in the piggs which contain that animation and figure out how long that is.
It is my fervent hope that nothing I want to eventually do will ever come to that.
But then, deciding " to figure out how long combat powers actually rooted the player for, and if that was consistent with what we were told" and then actually *DOING* it is what makes you @Arcanaville. ;-)
Besides, I'm not really sure whether this:
At some point in this process, a technically proficient person will either guess something or experiment and discover something.
is a breadcrumb trail or a "Here Be Monsters" sign, heh.
Have you read _True Names_ by Vernor Vinge? If not, you should. I suspect you'd enjoy it.
-
Besides, I'm not really sure whether this:
is a breadcrumb trail or a "Here Be Monsters" sign, heh.
Have you read _True Names_ by Vernor Vinge? If not, you should. I suspect you'd enjoy it.
Sometimes slavery or greatness rests on the goodwill of one or two persons.
-
Oy, vey.
Excuse me a moment.
/headdesk /headdesk /headdesk /headdesk
I "get it" now.
I was toying around with assigning velocity vectors to the bot and watching what happened. It was the point where I thought, "I'll have to slow that down if it's going to walk around the plaza" that the first bulb lit up. Then when I changed velocity and saw my bot rubber band back to start point, because I was just setting velocity and not position, the other light bulb lit up. It's not enough to know that I want to go from A to B. I have to know the animation time so that I know what speed to assign and how long it will take to move from a to be while playing that animation.
Geez, this is almost like writing a game from scratch in a lot of ways. It's a good thing I didn't think this through very far before I started or I would have just gone off to play some Hearthstone instead, LOL.
-
So I decided to take "a look" at trying to integrate the simple idea of walking on the ground instead of being a ghost. I thought I might make some progress on that this weekend.
Yeah, ha ha. I'm probably going to spend most of that time just thinking about the problem. In the general, most expansive case, I'd say after thinking about it and seeing what the problem space actually is, this one problem dwarfs every single problem the bot construction has faced to date, combined, by several decimal orders of magnitude. In fact, I think an accurate description of Paragon Chat is that its a program that performs map and surface calculations on moving objects, that happens to incorporate every other feature as a side project.
That doesn't mean there aren't short cuts. The entire bot project has in a sense been one giant shortcut around the massive project that is Paragon Chat. I'm carving a starter key out of balsa wood that starts the Ferrari that is Paragon Chat. So hope is not lost. A lot of my weekend is lost, but there's always hope.
I could always just wait around for Codewalker to implement a surface relay API that tells you which surface is closest in the direction of gravity, and what the first collision is between the world and your travel capsule, but that seems lazy and like cheating. I'm not opposed to lazy, but lazy cheating offends my sense of honor.
When it comes to maps and geometry, I really, really, really wish it didn't. I'm actually impressed that Paragon Chat implements this as well as it does in only a few months of development work. Unless Codewalker has been writing 3D virtual reality software for the past decade and hasn't gotten around to mentioning it yet, this is actually one of the most impressive features of Paragon Chat. For example, I was thinking about this today when I got home and I began to think about surfaces. I was compelled to log into Paragon Chat to test something. You swim in Paragon Chat. If you jump into the water you swim.
It seems like a small thing, but from a programming perspective consider what Paragon Chat is doing. It understands gravity, it understands geometry, it lets gravity drop you to the water. It *then* understands that the water is not a collision surface, but the bottom of the lake is. It understands that that surface is a different kind of surface, and sets the correct modes that alter your character's default animations when standing still and moving. Some of this is happening "automatically" in the City of Heroes client, but a lot is not. Bots don't appear to tread water, so the game client doesn't apply that automatically: Paragon Chat has to know this and apply this. Multiply this by all the different kinds of surfaces there are in the game. Consider walking up stairs. Consider walking over a ground surface and under a ceiling surface. Its not like every 3D game doesn't do this, but reconstructing it is still non-trivial.
-
Oy, vey.
Excuse me a moment.
/headdesk /headdesk /headdesk /headdesk
I "get it" now.
I was toying around with assigning velocity vectors to the bot and watching what happened. It was the point where I thought, "I'll have to slow that down if it's going to walk around the plaza" that the first bulb lit up. Then when I changed velocity and saw my bot rubber band back to start point, because I was just setting velocity and not position, the other light bulb lit up. It's not enough to know that I want to go from A to B. I have to know the animation time so that I know what speed to assign and how long it will take to move from a to be while playing that animation.
Geez, this is almost like writing a game from scratch in a lot of ways. It's a good thing I didn't think this through very far before I started or I would have just gone off to play some Hearthstone instead, LOL.
Paragon Chat implements prediction. Or rather Paragon Chat passes the parameters to the game client which implement prediction. Which means if you set your position to 0,0,0 and your velocity to 0.7,0,0, you're telling Paragon Chat and everyone else's game client that your bot is moving at 0.7 feet per frame, or 0.7 feet every 1/30th of a second, or 21 feet per second. If you send nothing else, your bot will appear to take off at about 14 miles per hour in that direction. Everyone will guestimate that in one second your position will be 21,0,0. However, if you actually send 18,0,0, then you will "rubberband" to 18,0,0. And then continue to move at 21 feet per second in the x direction. So to make sure this all works, you have to make sure that every time you send a <u /> your advertised position is consistent with your advertised velocity, or you will rubberband.
It actually did take some thinking to make that relatively smooth walking/running/flying video. You need to implement velocity, position, and a clock to sync it all up. That's why I considered it a milestone.
-
It actually did take some thinking to make that relatively smooth walking/running/flying video. You need to implement velocity, position, and a clock to sync it all up. That's why I considered it a milestone.
Yes, that's what I mean by "I get it now". I had thought about individual pieces here and there in regards to motion but I'd held onto this stubborn belief that, at heart, what I was doing was really just a sort of lower level version of Architect Entertainment and that moving from here to there was just a matter of defining what "here" and "there" meant.
What we're actually doing is taking a graphics engine and a lot of premade assets and trying to build a game, or some sort of interative toy at least, on top of it. Not quite from the ground up but a lot closer to the ground than I'd ever really considered it to be.
So, yes, I have a much better appreciation now for what you accomplished there, in a relatively short period of time to boot.
-
Speaking of the game client doing prediction -- I was experimenting with velocity vectors some more this morning and found an interesting effect.
After some fiddling with small values and seeing results I hadn't expected, I gave the bot a command to !setv 0 6 3.
I knew that gravity exists and that flying is basically exerting a plus velocity along the Y axis. What I expected was that setting vY would cause the bot to fly away at a pace dictated by vY and an angle dictated by the interaction of vX and vZ.
What happened instead is that the vY was treated as an instant burst. The predictive code moved the bot's avatar as expected, but not at a constant vY velocity. Instead of telling my bot to fly, I shot it out of a cannon and the game client automagically applied gravitational pull to it after that as it moved forward along the z axis.
In effect, my bot super jumped from the feet of Atlas to a spot out near the park south of Ms. Liberty.
What this means is that any time there's natural force in play along an axis of motion, that it takes a constant assertion of counter force to resist it. Flying isn't just a matter of sending a <U> with upward velocity. You have to send a stream of <U>'s that reinforce your position and heading. If the game supported a zone with "head winds" or currents in water, you would have to actively resist those forces in the same way.
The question this raises is -- How is my bot supposed to figure out ahead of time where its landing point is supposed to be? I can't query your client to ask where it put my bot's avatar.
There are probably some standard physics equations that can handle that problem, but that really hilights a challenge that is facing any bot writer - namely, that every single bot basically has to carry around in its brains a collection of all of the stats and equations that would normally be held by a single central server in a more typical client/server architecture.
A bot ought to be a specialized kind of client but by necessity we're having to make them be more of a specialized kind of mini game server.
In an ideal world, I would tell the server to move my avatar from A to B using method C and the server would generate the movement stream that made it so.
Even in a world where my bot might need to send its own <U> stream to be properly tracked by other clients, a bot would want to be able to query the server for a path and the server would reply with a list of <U> stanzas that would implement the path.
At a bare minimum, it seems like a server ought to be able to define its "world rules" and respond to a service discovery query from a bot with a summary of the rules; like, for instance, "How strong is the pull of gravity in your world?"
Openfire would handle something like this through its plugin system. Instead of every bot carrying a bunch of physics around and reinventing the wheel a dozen times, the physics would be handled by the server and the bot would *ask* for something like "move me from point A to point B and tell me how long it will take travel between them and what special animation stream or movement stream I need to use while doing it".
-
In fact, I think an accurate description of Paragon Chat is that its a program that performs map and surface calculations on moving objects, that happens to incorporate every other feature as a side project.
Loading the geometry is the hardest part, but we've had a working geo viewer for some time (http://www.cohtitan.com/forum/index.php?topic=7467.msg116180#msg116180). That's from 2013, but the initial post of those images on our private development forum is dated March 27, 2012. Combined with the bin formats that had been fully decoded since discovering the Rosetta stone embedded in the client exe in late 2011, most of the information is there, but you're right that putting it all together is not trivial.
When it comes to maps and geometry, I really, really, really wish it didn't. I'm actually impressed that Paragon Chat implements this as well as it does in only a few months of development work.
I don't want to quote myself, but I did mention that Paragon Chat builds on work that has been going on for much longer. So while the idea for using XMPP as a communications medium and the work that went into making that a reality happened over a few months (closer to half a year), a lot of the other pieces you mention have been under more or less constant development for the last 2, 3, or in some cases even 4 years.
It also helps that the client itself has a fairly complete implementation of the same physics and collision processing, even though it only uses it for local player prediction, ripe for picking apart with a debugger to figure out exactly what formulas to use to come up with similar results.
-
What happened instead is that the vY was treated as an instant burst. The predictive code moved the bot's avatar as expected, but not at a constant vY velocity. Instead of telling my bot to fly, I shot it out of a cannon and the game client automagically applied gravitational pull to it after that as it moved forward along the z axis.
That's a Paragon Chat thing, intended to make jumping appear less jerky and more natural, since velocity updates only come in every second or so. It continuously applies gravitational acceleration to any nonzero Y velocity value. If the remote PC and game client are doing the same thing on their end, any updates should be relatively close to the predicted value, with some margin of error for inconsistent latency.
That's part of why flying requires extra protocol support, because it will need a tag to tell other clients to disable that prediction and use velocity as-is.
The question this raises is -- How is my bot supposed to figure out ahead of time where its landing point is supposed to be? I can't query your client to ask where it put my bot's avatar.
If you want to try to match the prediction, COH's default gravity is 0.065 units/frame^2 when moving upward, or 0.0975 units/frame^2 when moving downward or at rest.
You can also use 1.95 units/frame/sec or 2.925 units/frame/sec, respectively, to make the math a little easier if you're using seconds as your unit of time. Just don't forget that velocity on the wire is always in units/frame.
Yes, the COH universe has bizarre physics where the force of gravity is different depending on which direction you're moving. I have no idea why. It's probably one of the many, many things that the developers did by feel rather than by math. That's why I continue to say that any attempt to recreate the game in an off-the-shelf engine will never 'feel' quite the same.
Assuming I'm not screwing up the math completely, and assuming the usual convention of 1 world unit = 1 foot, that comes out to around 17.8 m/s^2 when rising and a whopping 26.7 m/s^2 when falling. That doesn't seem right to me as it doesn't seem like it's that much higher than real world gravity, so it's possible I'm missing a conversion factor somewhere.
I'll note that gravity does not affect flying entities at all; they get to ignore it completely and do not have to exert a balancing upward force. The effect you're encountering is only due to Paragon Chat's prediction and the fact that it doesn't know you're flying.
There are probably some standard physics equations that can handle that problem, but that really hilights a challenge that is facing any bot writer - namely, that every single bot basically has to carry around in its brains a collection of all of the stats and equations that would normally be held by a single central server in a more typical client/server architecture.
For non-vertical movement, the formula is exceedingly simple: deltaP = v * deltaT, where deltaT is in frames.
But yes, the XMPP protocol used by Paragon Chat is not really so much a protocol to talk to a game server but rather a means of synchronizing the state of many independent game engines.
-
I knew that gravity exists and that flying is basically exerting a plus velocity along the Y axis. What I expected was that setting vY would cause the bot to fly away at a pace dictated by vY and an angle dictated by the interaction of vX and vZ.
What happened instead is that the vY was treated as an instant burst. The predictive code moved the bot's avatar as expected, but not at a constant vY velocity. Instead of telling my bot to fly, I shot it out of a cannon and the game client automagically applied gravitational pull to it after that as it moved forward along the z axis.
Actually, I believe when a City of Heroes entity flies, setting the fly mode effectively makes them immune to gravity. Codewalker said somewhere else I believe that if your delta-Y is non-zero, Paragon Chat applies gravitational acceleration. If it is exactly zero, it does not. So the correct way to mimic flying in Paragon Chat today is to explicitly set deltaY to zero, and set your Y position to whatever. That way Paragon Chat won't attempt to predict your next Y position and won't apply gravity. What *will* happen is that your vertical motion will be significantly more jumpy than your horizontal motion, but so long as you confine flying to a relatively xz plane-ish motion, it won't be too bad.
There are probably some standard physics equations that can handle that problem, but that really hilights a challenge that is facing any bot writer - namely, that every single bot basically has to carry around in its brains a collection of all of the stats and equations that would normally be held by a single central server in a more typical client/server architecture.
A bot ought to be a specialized kind of client but by necessity we're having to make them be more of a specialized kind of mini game server.
That is both problem and opportunity in my opinion. For example, right now Paragon Chat doesn't support flying. But my bot can fly, in a way. It doesn't support teleporting, but my bot can teleport. Because the bot is required to enforce its own rules - like every other Paragon Chat instance does - and because Paragon Chat only loosely enforces certain rules on what everyone else can do, bots can do things that Paragon Chat *users* can't do, but Paragon Chat implicitly supports allowing.
At a bare minimum, it seems like a server ought to be able to define its "world rules" and respond to a service discovery query from a bot with a summary of the rules; like, for instance, "How strong is the pull of gravity in your world?"
That's an interesting idea, actually.
-
If you want to try to match the prediction, COH's default gravity is 0.065 units/frame^2 when moving upward, or 0.0975 units/frame^2 when moving downward or at rest.
You can also use 1.95 units/frame/sec or 2.925 units/frame/sec, respectively, to make the math a little easier if you're using seconds as your unit of time. Just don't forget that velocity on the wire is always in units/frame.
Yes, the COH universe has bizarre physics where the force of gravity is different depending on which direction you're moving. I have no idea why. It's probably one of the many, many things that the developers did by feel rather than by math. That's why I continue to say that any attempt to recreate the game in an off-the-shelf engine will never 'feel' quite the same.
Part of that bizarreness is almost certainly due at least in part to SuperJump. In beta, superjump worked in a particular way (I wasn't in beta, so I can't comment on the precise way it was different, but I know it was different in significant ways). Very early on, they changed the way it worked mechanically to change the way players controlled jumps and to affect how the leaps appeared. There was a lot of consternation by players already used to the previous superjump mechanics and the devs tweaked it to make sure superjump experts could regain the same level of control or better with the new version. That required tweaking with the physics of superjumping, and of course that means tweaking how gravity works; particularly how it works when you're going up and independently how it works when you are going down. In fact I'm not convinced normal gravity is actually working when you are in the upward half of a superjump and completely different physics are happening there. It doesn't *feel* like gravity - any kind of gravity - when I'm superjumping.
Assuming I'm not screwing up the math completely, and assuming the usual convention of 1 world unit = 1 foot, that comes out to around 17.8 m/s^2 when rising and a whopping 26.7 m/s^2 when falling. That doesn't seem right to me as it doesn't seem like it's that much higher than real world gravity, so it's possible I'm missing a conversion factor somewhere.
That doesn't feel right to me either: its twice normal gravitational acceleration in the upward direction and three times normal gravitational acceleration in the downward direction. Its worth some additional investigation. I think I have a way to do this in a controlled and relatively precise manner.
-
Interestingly, my low precision tests (putting my bot at various heights in the air and setting downward velocity to -0.0001 and letting Paragon Chat drop them and timing the result on a stopwatch) fairly consistently measure downward acceleration as being about 50-60 feet/sec^2, or about 16-20 meters/sec^2. There are potentially large margins for error on that measurement, but its interestingly consistent within +-20%.
Does Paragon Chat apply any frictional forces or velocity cap on downward motion?
I'll see about trying to create a higher precision version of this test (using video motion extrapolation) later this weekend.
Just as an FYI I added a new command to my bot: /experiment. Its a cutout that lets me invent functions to test, like /experiment fall XXX which causes the bot to teleport XXX feet higher than current position, set downward velocity to -0.0001, and let Paragon Chat drop them. Works with /comehere which teleports the bot to within five feet of my position (because you have to reset your position after every drop, because your bot doesn't actually have a consistent position with Paragon Chat after the fall).
-
PS: is this why I got a college degree?
-
Interestingly, my low precision tests (putting my bot at various heights in the air and setting downward velocity to -0.0001 and letting Paragon Chat drop them and timing the result on a stopwatch) fairly consistently measure downward acceleration as being about 50-60 feet/sec^2, or about 16-20 meters/sec^2. There are potentially large margins for error on that measurement, but its interestingly consistent within +-20%.
Hmmmm, interesting. In the meantime, I built a modified version of Paragon Chat with the gravity values set to what I calculated as appropriately representing 9.8 m/s^2. I then turned off clientside prediction so the client wouldn't get into a rubber-band fight. It felt like walking on the moon.
I suspect the issue may instead lie with the assumption that 1 world unit = 1 foot. Nothing has ever been particularly well scaled according to that, regardless of what in-game marker distances may say.
Does Paragon Chat apply any frictional forces or velocity cap on downward motion?
It does apply friction (with a coefficient varying based on the surface parameters) to the local player, but not to other players that are being received over XMPP. It treats those as frictionless and expects that the remote instance will reduce the velocity accordingly.
There is a speed cap (base of 1.0 magnitude IIRC), and it is applied to all three dimensions while flying, but only to X/Z when walking/falling. Again that is to the local player only and is not applied to XMPP-driven players; it would just be one more thing to have to synchronize. So bots should be able to exceed the speed cap.
In fact I'm not convinced normal gravity is actually working when you are in the upward half of a superjump and completely different physics are happening there. It doesn't *feel* like gravity - any kind of gravity - when I'm superjumping.
Correct, there is no gravity when you're jumping, and you maintain a constant upward velocity while the key is held down. Once it's released (or the timer runs out), the lower gravity kicks in first to counter your upward momentum, then it switches to the higher gravity as soon as you start moving downward.
-
PS: is this why I got a college degree?
You need a better reason? :-p
Codewalker - I wouldn't put this high on any "to do" list, but if a bot is focused on a particular player anyway as part of a "quest" or something, it might be useful to send an IQ to that player's client and ask it "Hey, where do *YOU* think I am located?" I don't have any specific reason to ask a client that question at the moment, but somehow it strikes me that it could be an interesting thing to know; particularly if a script was running that was based off of stuff happening relative to the player and not necessarily happening at an absolute position in Paragon space.
Okay, here's a made-up example. Suppose I have a NPC who patrols between three points, but the only time he ever sends out a<U> message is when he reaches a vertex of his patrol triangle and changes course. While Mr. NPC is walking between B and C, a player steps up and stops the bot by saying, "Hey, Mr. NPC! Tell me about X!".
My bot knows that he last emmitted a <U> at timestamp X, and he can determine that Y number of seconds have passed and from that he can calculate the distance he must have traveled in the meantime. Maybe, though, what he cares about more is to ask the player's client "Where am I right now, according to you?" and then setting his absolute location to be the same as that relative location. If there was any sort of data transmission lag, or even just a small amount of time required to processing things before the bot established its location, then setting its location by the client's coordinates would eliminate potential rubber-banding caused by jumping to the bot's more accurate idea of location.
Like I said, not something with any kind of priority but still maybe something that could have uses in certain situations.
-
Another feature request, along the lines you mentioned awhile back of "knowing when you've been clicked".
If a player has another player targeted when he sends a <U> stanza, can the <U> stanza include a "target=JID" or "target=CharacterName" attribute?
The cool consequence of that is that a bot can register a call back specifically for "<U target='myJID'>" and if, say, someone emotes "/em punch3" while the bot is targeted, the bot can respond by tipping itself 45 degrees, launching itself backwards 100 feet and shouting, "Curse you SuperduperMan! You haven't seen the last of The Claw!"
-
I suspect the issue may instead lie with the assumption that 1 world unit = 1 foot. Nothing has ever been particularly well scaled according to that, regardless of what in-game marker distances may say.
I usually set my characters to average height (under 6ft, maybe 5'9") and have often noticed that I seem to be much shorter than that as I go about the city.
-
You need a better reason? :-p
Codewalker - I wouldn't put this high on any "to do" list, but if a bot is focused on a particular player anyway as part of a "quest" or something, it might be useful to send an IQ to that player's client and ask it "Hey, where do *YOU* think I am located?" I don't have any specific reason to ask a client that question at the moment, but somehow it strikes me that it could be an interesting thing to know; particularly if a script was running that was based off of stuff happening relative to the player and not necessarily happening at an absolute position in Paragon space.
Okay, here's a made-up example. Suppose I have a NPC who patrols between three points, but the only time he ever sends out a<U> message is when he reaches a vertex of his patrol triangle and changes course. While Mr. NPC is walking between B and C, a player steps up and stops the bot by saying, "Hey, Mr. NPC! Tell me about X!".
My bot knows that he last emmitted a <U> at timestamp X, and he can determine that Y number of seconds have passed and from that he can calculate the distance he must have traveled in the meantime. Maybe, though, what he cares about more is to ask the player's client "Where am I right now, according to you?" and then setting his absolute location to be the same as that relative location. If there was any sort of data transmission lag, or even just a small amount of time required to processing things before the bot established its location, then setting its location by the client's coordinates would eliminate potential rubber-banding caused by jumping to the bot's more accurate idea of location.
Like I said, not something with any kind of priority but still maybe something that could have uses in certain situations.
I'm uncomfortable about this. As a source of information, it might be useful in some circumstances. But as a crutch, I'm less supportive. I think bots should track their own internal state well enough to not require asking clients what they think the bot might have been doing. For example, *who* do you ask? Every single Paragon Chat client is doing its own predictions. Its not necessarily kosher to ask a specific client for positional correction information even if they happen to be the one interacting with the bot. Its also vulnerable to poisoning: if I know your bot does that, I can write a bot that specifically interacts with your bot and then sends it to the moon.
If and when Codewalker adds a bot-API, that's the point where these kinds of questions can be better addressed. Because the correct thing to ask "where am I" if you happen to not know is not another player, but your own Paragon Chat instance. So maybe a future Paragon Chat will allow you to connect to it rather than directly to an XMPP server. You would start Paragon Chat, and then instead of launching the CoH client, you would launch your bot and connect to PC. PC would handle all of the XMPP, and your bot would talk directly to PC and send it entity commands.
-
Another feature request, along the lines you mentioned awhile back of "knowing when you've been clicked".
If a player has another player targeted when he sends a <U> stanza, can the <U> stanza include a "target=JID" or "target=CharacterName" attribute?
The cool consequence of that is that a bot can register a call back specifically for "<U target='myJID'>" and if, say, someone emotes "/em punch3" while the bot is targeted, the bot can respond by tipping itself 45 degrees, launching itself backwards 100 feet and shouting, "Curse you SuperduperMan! You haven't seen the last of The Claw!"
http://www.cohtitan.com/forum/index.php/topic,10956.msg187675.html#msg187675
-
A note about protocol. I just started using 0.98b which now supports diagnosing invisible characters. While testing with this, I discovered something that wasn't explicitly specified (as far as I can recall). If you change Presence in any way, you MUST immediately send a new <u />. It seems if you change presence Paragon Chat essentially considers you a new entity, and that means it won't know where you are until you send position information. I'm not sure if this is new to 0.98b, or if this behavior was ill-defined in previous versions of PC but I noticed that my costume-changing code started to glitch out in 0.98a, and 0.98b told me that the problem was no position information. I changed my code so that whenever a costume change occurred I immediately send pc:u after presence, and the problem went away.
-
That sounds like a bug. Presence should apply the new costume to the current entity. It doesn't reset the 'has received initial position' flag that was added in 0.98b. The only other thing I can think of is it's failing to find the current entity and sending a new one to the client, but that shouldn't be happening either.
-
Its also vulnerable to poisoning: if I know your bot does that, I can write a bot that specifically interacts with your bot and then sends it to the moon.
We're already breaking the "never trust the client rule" six ways from Sunday anyway. There's no point in someone maliciously doing that because it doesn't (yet) benefit them in any way.
For the rest of what you're saying, I actually do agree with you but, I have this nagging idea that something about that question "Where do you think I am?" is interesting but I can't quite formulate what it is yet.
Maybe it's just a bad idea. I'm never one for tossing away an idea just because it sounds bad, at least not until the people who know better confirm that it really is something better let drop.
-
That sounds like a bug. Presence should apply the new costume to the current entity. It doesn't reset the 'has received initial position' flag that was added in 0.98b. The only other thing I can think of is it's failing to find the current entity and sending a new one to the client, but that shouldn't be happening either.
If I use my "resize" command on my bot in 0.98b, it disappears. If I then move the bot any amount in any direction, it reappears. If I change my code to immediately resend position after sending updated presence, the bot never disappears in the first place. When I did a /whoall, the bot showed up as not received initial position after sending (updated) presence but before resending position. Just FYI.
Also, I just did an informal test of altitude by commanding my bot to move exactly 6 units upward. The bot moves slightly more than its own height upward, which is consistent with the height of the costume and one unit equaling one foot.
I too am getting the same "dance, bot, dance" feeling slickriptide got when he got his bot to obey commands. Its weirdly empowering. /t ArcanaBot, /comehere. /t ArcanaBot, /moveto 942.2,-32,-714.3. /t ArcanaBot, /experiment fall 300. /t ArcanaBot, dance like a chicken.
When the robot revolution happens, my head is going to be one of the first ones on a pike.
-
When the robot revolution happens, my head is going to be one of the first ones on a pike.
LMAO
Okay, so here's a question for everyone; in fact, it's one that even @Dyne is in a position to weigh in on.
It's this - How much of what we're doing is something that should be available to anyone as a service they can interface with and avoid having to repeat the basic implementation themselves?
Take the sequencer data, for instance.
Maybe, instead of five people handling that data five different ways in five different formats, it would instead be useful to create a repository of sequencer data that can be asked for by XMPP feature discovery and queried with IQ queries in a well defined namespace. Then everyone who wants to use the data is using the same data, and everyone who wants to talk about it is speaking the same language. The service could be provided by a standalone server that registers with a nearby XMPP server to advertise its services, or it could be a plugin to a modular server like Openfire. The advantage to the plugin route being that when someone sets up their own personal server, they just request the plugin to be installed and there's no need for any further management of the service by that person.
I guess that what I'm suggesting is that maybe there are some functions that can be designated as being functions that every bot/paragon-related-server needs to be able to handle, and codify those functions into a form that makes it so that future programs don't have to reinvent their own version of the wheel in order to interface with Paragon Chat.
.
-
Here's my take on that.
Right now, this is all experimental, but I do intend to document everything I do in a sort of shortest-path to making a bot. Its a bit fluid because I'm not 100% there yet and because I'm aiming at a moving target with Codewalker obviously improving and enhancing Paragon Chat itself.
Codewalker and his team have a concern and I believe its a valid one: can you make the software in a way that avoids as much legal entanglement as possible. I'm not operating under the same limitations they are, but I'm also cognizant of the same issues. I believe the solution is to implement in a similar manner: never distribute data, only code. For our purposes, that means while I'm happy to include elements of information in the bot like specific MOV names, I'm not prepared to distribute entire sequence databases with the bot itself. But in the long run, that won't be necessary for two reasons. One: Paragon Chat's architecture may make that redundant. Two, if it doesn't, then at some point I'll simply add the ability to read that information within the bot itself. That way, any bot writer that emulates or just plain rips what my bot does will have access to the same information mine does - provided they also have a copy of the I24 client to run Paragon Chat in the first place.
The irony here is that if you take Codewalker's sequencer data that he posted a while ago and stick it in your bot, no one will care. However, the more organized you attempt to make the sharing of that data the more you become an attractive target and a failure point of the system. So if you decide to make a public sequencer or map data internet accessible service that is designed to be used by people trying to replicate the CoH environment, you potentially become a take down target. And when you're taken down, everyone who is depending on you goes down with you.
Philosophically, I believe (one of) Paragon Chat's development principles is to avoid that situation, and even though nothing I'm doing will itself ever rise to a level that mandates it, I'm tending to follow that principle as well.
-
How not to measure gravity in City of Heroes:
1. Command an entity to change its location to a point XX feet above the ground.
2. Immediately start timer.
3. Wait for entity to hit the ground.
4. Stop timer.
Why this doesn't work:
1. When you command an entity to change its location, that *doesn't* instantly teleport the entity to that location. Instead, the game performs an interesting "warp" to that position, and more interestingly the game always tries to perform this act in about the same amount of time: about a quarter of a second. Moreover, there's a visual artifact: you actually *appear* to move to that new location at very high speed. Because of this, there's a correction factor of about 0.25 seconds before the entity actually starts falling.
2. If you do this while standing next to the entity, odds are it will actually fall *on* you. And that means there is a momentary collision which deflects the falling entity in one direction. At the falling speeds being tested, this has a minimal but measurable effect on the falling trajectory.
I discovered this with my "high precision" method of testing gravity, which is to analyze demorecords of the bot falling which require Paragon Chat (city of heroes, technically) to record the path of the bot with timecodes. Demorecords are still not perfectly precise and have timing glitches, but those can be averaged out. Given what I now know about motion** and observer collisions*** I will perform a more detailed analysis probably tomorrow.
** I actually knew about the space-warping thing because I had witnessed effects related to that in the past, but knowing and consciously factoring in are two different things.
*** Its interesting that it is possible to be standing next to an entity and obviously not colliding with them, and yet when they move in a mathematically precise manner straight up they can land almost on top of you. I'm not sure if that means CoH movement calculations have a lot of round off errors, or if gravity itself isn't 100% perpendicular to the Y-axis on Paragon Earth. I had a brief thought of making a gravity map of Paragon, but it took only three swift forehead strikes with a hammer to dispel that notion.
-
Assuming I'm not screwing up the math completely, and assuming the usual convention of 1 world unit = 1 foot, that comes out to around 17.8 m/s^2 when rising and a whopping 26.7 m/s^2 when falling. That doesn't seem right to me as it doesn't seem like it's that much higher than real world gravity, so it's possible I'm missing a conversion factor somewhere.
Analyzing demorecords and dropping entities from heights of 100 feet to 500 feet and only analyzing the segments of the demorecords when the entities are definitively tracking the free-fall portions of their descent, I calculate gravitational acceleration during downward falling as -87.3 f/s^2 or -26.6 m/s^2, plus or minus 0.3 m/s^2 (variations in demorecord timing). That basically confirms your calculations, and means gravity is about 2.7 times stronger in Paragon City than on Earth (at least when falling).
Also, on the general subject of speed caps, I can also confirm there appears to be no enforced speed cap in the Y direction when falling, at least no cap within the region of my testing. Maximum downward velocity measured was 286 fps from a drop height of 500 feet. That is 195 mph. Coincidentally, that's about terminal velocity for a human being falling while in roughly the standing position, and its achievable on Paragon Earth in about three seconds of free fall. By comparison, on Earth Earth it takes about three seconds to hit the pavement jumping out of a fifth story window, and you hit the ground moving at about 60 miles per hour - fast enough to probably kill you. In Paragon City, jumping out of a fifth story window causes you to hit the ground in a blink-and-you-miss-it one second, hitting the ground at about 90 miles per hour - fast enough to absolutely kill you if you were a normal Earth person. In fact, its fortunate Paragon City citizens are practically indestructible because just tripping on the pavement would cause you to hit the ground at speeds of 25 mph or better - significantly higher than the minimum collision speeds necessary to cause life-threatening injuries in auto vs pedestrian accidents (which I'm a lot more familiar with than I was two years ago).
The next interesting question is why gravity is so much stronger in City of Heroes than real world gravity, and why stronger gravity "feels" correct even though its enormously stronger than real world acceleration. I have suspicions, but there's probably more testing necessary to confirm them.
I'm probably just ducking making space trees.
-
Okay, so here's a question for everyone; in fact, it's one that even @Dyne is in a position to weigh in on.
It's this - How much of what we're doing is something that should be available to anyone as a service they can interface with and avoid having to repeat the basic implementation themselves?
I'm of two minds.
On the one hand, I'm a big fan of not reinventing the wheel if you don't need to.
On the other hand I am, in fact, re-implementing a lot of the same stuff (probably in stupid ways) that you and Arcana have already done. Often even bits for which a working solution was already posted (like loading a costume file). Mainly that's because I am more interested in writing my own bot than in copy-pasting someone else's (if for no other reason than because I'll understand it better that way, at least in theory. Which I'll probably need in order to make it do what I want.)**
Maybe, instead of five people handling that data five different ways in five different formats, it would instead be useful to create a repository of sequencer data that can be asked for by XMPP feature discovery and queried with IQ queries in a well defined namespace. Then everyone who wants to use the data is using the same data, and everyone who wants to talk about it is speaking the same language. The service could be provided by a standalone server that registers with a nearby XMPP server to advertise its services, or it could be a plugin to a modular server like Openfire. The advantage to the plugin route being that when someone sets up their own personal server, they just request the plugin to be installed and there's no need for any further management of the service by that person.
It's not a bad idea in theory, but I'd tend to agree with what Arcana said earlier. It might be better to simply distribute the code for loading such data, written as a generic module that others can use like a black box (as long as they are using python, anyway). They just import it in their own bot or a server plugin or whatever they want. That way you'd still get most of the benefits but less legal risk. And, while it may not be a huge issue, that way there's no need for a dozen bots to start eating bandwidth to look up information that they theoretically already have locally.
** Not that it means I won't welcome a guide or look at a solution to each problem before I start, and not that I haven't borrowed from the code posted earlier (I took the basic clevel structure from Arcana's costume loader, for example, but most of the actual actions it takes have been heavily modified because I'm stuffing the data into my own Costume class).
I expect I'll have to do more of this for sleekxmpp, because hoo boy did that library give me a headache when I first started fooling with it (registerHandler vs add_event_handler? stanza interface vs. stanza plugins, and stanza plugins vs. XEP-plugins? etc.) I did get to the point of successfully logging into my Openfire server (once I turned openfire's SSL off, because getting self-signed certificates working are SO not what I'm interested in right now), but it's not a particularly impressive feat considering that most of that was pretty much in the Echobot demo.
As seems to be par for the course, I more-or-less understand what needs to be done, but understanding how to get the library to do it is a different question.
-
My bot can walk around Atlas Plaza when told explicitly to "Walk this way. No, *this* way. Walk east at velocity 0.07. Turn right. No, *YOUR* right. Switch velocity vector, you're walking sideways!"
Yay me.
All of which just means that if it was following a script that it would be walking around like a champ. If it waved hello when it walked by you, it would be just like a real CoH street NPC.
So, there's a scripting interpreter that needs to be written.
There's also the thing I really want the bot to do, that's turning out to be a bitch to figure out a "right" way to do it - following a player. Plus, if I want to continue to do most of my coding at the ErrBot level rather than at the SleekXMPP level, then I'm going to have to bite the bullet and make a ParagonChat backend to ErrBot that is basically the existing XMPP backend modified to also recognizes all of the Paragon Chat data stanzas and pass them up to ErrBot along with the standard presence and message data structures.
Basically, a whole bunch of bot functionality tends to boil down to the bot having to know who is in the zone and how far away they are. Mostly, that's so that the bot can determine the answer to "Which players are within 20 feet of me?" or "How far away am I from player X and in what direction?" Even cheating on the "follow player X" bit by just replaying the target's movement stream at a bit of an offset, position-wise, means having to track that player's <U> footprints, which means, once again, making sure that the bot has access to those footprints in the first place.
So, basically, once again diving into a whole mound of foundational stuff in order to make one seemingly simple, but vital, effect possible.
I'm getting a better appreciation for what real game designers have to deal with when building these sorts of systems.
-
There's also the thing I really want the bot to do, that's turning out to be a bitch to figure out a "right" way to do it - following a player.
Last week I talked about refactoring my code to change it from a remote controlled drone using keyboard to a more goal-executing bot. That was specifically to implement /followme. Hypothetically speaking, you can rough-cut commands like "/faceme" and "/comehere" and "/emote XXX" by sending instantaneous commands: change orientation to this and then send it, for example. But /followme cannot work that way for obvious reasons. So that meant I could no longer have keyboard control and command execution coexisting as it did before. So I tossed that code and rewrote it to execute goals.
Now, keyboard control only sets movement flags. A separate goal-executing function checks those flags and decides what to do with them. In a sense, the goal exec only replicates what the original keyboard-brain-surgery function did: edit the bots positional/velocity/orientation values and send them on the wire. But it now *also* checks a goal queue. If there is at least one goal on the queue, it executes the first one. Also, whenever a goal needs to "move" the bot, instead of using brain-surgery to edit the velocity of the bot, it actually sets the same movement flags as keyboard control does. In a sense, the bot presses its own keys. Damn I'm clever (or would be, if that isn't basically how a lot of bot software does the same thing).
When I send "ArcanaBot: /faceme" to the bot, the message handler adds a goal to the goal queue: ['faceme','position'] where the first item in the goal entry is the command and the second is its parameters, in this case the location of the entity that sent the command. When the goal exec sees that, it figures out which direction to turn, sets 'CLOCKWISE' or 'COUNTERCLOCKWISE' and lets the normal movement routines do their thing (or rather, it mostly does that because I haven't really thought through the best way to "coast" to a final orientation quite yet, so there's a hacky thing where the rotational code will "force" a final orientation at the end of a turn).
The queue lets me do something I couldn't do before quite as easily: I can take actions to satisfy a goal that don't actually complete the goal. Instead, I can do something, check to see if the goal is complete, and if not I don't delete the goal. On the next clock tick the bot will check the queue again, see the goal is still there, do what its code tells it to do, check to see if done, and if not leave it on the queue. Rinse, repeat.
Now, the ['followme','entity'] goal can work like this: on each tick of the clock, if you are within five feet of 'entity' stop moving. If you are not, accelerate towards entity and face entity. Continue doing this indefinitely. A different command, ['unfollowme','entity'] deletes that goal from the queue.
Its not quite working yet because I got distracted by gravity, and also because I want to add some sophistication to the actual follow tracking algorithm. For example, if the target entity is within about 45 degrees of current facing angle, just swing towards it. But if its, say, suddenly behind the bot, I want the bot to do a more visually appealing stop, turn around, and start moving again in the opposite direction.
I also have "/orbitme radius", "/mimicme", and "/knockback" in the works.
All of this is prelude to the holy grail of this project: "/fightme"
Basically, a whole bunch of bot functionality tends to boil down to the bot having to know who is in the zone and how far away they are.
My presence and <u /> handling code keeps a pcRoster. Whenever I get a presence stanza, I check to see if I already have them. If I don't, I add them to the pcRoster dictionary. Whenever I get a <u /> I add those fields to the appropriate pcRoster entry if it exists. That allows me to do this:
def add_command(self,cmdstr,sender):
for rosterEntry in self.pcRoster:
if self.pcRoster[rosterEntry]['pc_jid'] == sender:
rosterID = rosterEntry
print "Debug: sender " + str(sender) + " converted to rosterID " + str(rosterID)
cmd = cmdstr.split()[0]
if cmd == 'faceme':
position = self.pcRoster[rosterID]['position']
self.cmdQueue.append(['faceme',position])
Yeah, there's no error checking. If I zone the bot in and execute that command too quickly, it gets there before the roster entry is created and it bombs. That's my failsafe for when the robot revolution happens. But whenever someone sends my bot a command, I can look them up in the roster and figure out things like where they are. That allows me to easily execute commands like "/faceme" which get converted to the command ['faceme',position] where position is the p= entry from the last <u /> I got from that entity. Technically, I should apply (velocity) prediction to that, but I'm not there yet. Mostly because I stupidly forgot to timecode my roster entries.
Also, why didn't I just key my roster entries to jid? That's complicated.
-
Its not quite working yet because I got distracted by gravity, and also because I want to add some sophistication to the actual follow tracking algorithm. For example, if the target entity is within about 45 degrees of current facing angle, just swing towards it. But if its, say, suddenly behind the bot, I want the bot to do a more visually appealing stop, turn around, and start moving again in the opposite direction.
I think this one is sufficiently interesting to bot writers it was worth amplifying. Here's a video of my current following algorithm:
https://www.youtube.com/watch?v=VFxbyLM4OTA
Notice it actually works not too bad at first. I was smart enough to create a following distance: once the bot reaches this distance it stops trying to get any closer. That prevents bots from following you right up your nose. And it turns and moves forward. You can also notice what I'm not doing yet: without motion prediction the bot is actually lagging behind and following where the character *was* rather than where it is. Which can seem oddly almost more sophisticated of a follow.
Then note what I do at about 40 seconds in: I start running in circles, and I specifically get to a very specific distance and orientation from the bot. Its tricky to do, but when I do it something remarkable, or at least remarkably weird happens. The bot begins to orbit my character (remember my previous post when I said I was working on an "/orbitme" command? Well, I didn't just decide to do that for no reason). Basically, because the follow me algorithm just keeps trying to point towards the character and accelerate forward in that direction, there exists a magic distance from the target where turn + accelerate = orbit, very much analogously to how actual objects orbit gravitational bodies. In this case, my following distance is 5 feet, and my algorithms intrinsic orbiting distance is about 10 feet, so it can just about work.
There's a couple of things you need to do to prevent this from happening. First, you need to make sure your bot isn't a rocketship: because I do not implement friction, my movement commands are accelerating the bot in a certain direction while it keeps all of its current velocity. Discounting the y-axis, in effect I'm ice skating on the XZ plane. Second, you need to make sure that when your direction of acceleration diverges from your actual velocity vector by a high enough angle, you stop the bot and have it make a turn, rather than try to accelerate at an obtuse angle (which real people can't do when not sitting on curling stone).
And then, of course, you need to figure out how to reproduce the effect within those parameters so you can actually do it on command, when you actually want to.
Also, notice that as long as you are on relatively level ground, map geometry is not a huge problem. Because the bot is "following" the character, it is always trying to move towards the character's ground level, which itself honors the ground surface. So long as your bot doesn't try to get too involved with sudden changes in altitude and interacts with players that are themselves standing on the ground, ground collision is not a mandatory requirement to work reasonably well.
I'm still going to do it, because I ultimately will need it. But for other bot writers, consider the fact that this one can, in many (but not all) circumstances, be ignored and still mostly work.
-
One more word to aspiring bot writers. It may seem like all of this is daunting, and its not exactly easy, but its worth noting that in my experience to date, development goes in cycles where first you are climbing what seems to be a very steep and unrewarding learning and coding curve, and then there's a payoff in being suddenly able to do a lot of things with the results of that curve. Followed by another curve.
First, you have to learn your tools. The language you're using, the frameworks you're using, enough to be able to write any meaningful code at all. Initially, it will seem like its difficult just to "Hello World" yourself out of a box. But once you get into the flow of things, your production rate will start to rise substantially.
The second hurdle to overcome is to understand and implement enough of XMPP to be able to connect to servers and send and receive messages. Until you get there, its difficult to get any feedback as to whether your code is actually doing anything or not. But once you can connect and send and receive messages, your ability to process those messages and do things will speed up dramatically. If you want to make a conversational bot, or a bot that interacts with the world but doesn't itself need to appear in that world, you can do a lot just from here. For example, the hypothetical "ski slope bot" that was discussed in another thread could have been written entirely from this point.
The third hurdle is visibility. It takes a lot of things to work simultaneously to make a bot visible to other players, and any one of them failing will make you invisible. Until you can see your bot, its difficult to really know if you can make it do anything at all beyond the chatting and other non-visible tasks discussed above. But once you make your bot visible, all sorts of things open up. You can get hacky things working like making the bot move to a spot, spin around, and emote without too much difficulty. Smooth locomotion - i.e. running, jumping, flying, walking - is trickier, but once your bot is visible, its a lot easier to experiment with this.
The fourth hurdle is to realize that your bot needs a way to execute a sequence of commands. Depending on how you've written your bot and what language and tools you're using, this could be easier or harder, but you will probably at some point need to put in at least some work here. And its entirely possible that the way you wrote your bot up to this point makes this difficult, because you didn't take this fully into account. My strongest recommendation here is, like you should have been doing up to now, keep every version and iteration of your bot. When you start refactoring code, make sure you never, ever, EVER delete any previous version, particularly versions that worked even if they didn't do exactly what you wanted. Nothing is more disheartening than "breaking" a working bot, and discovering you forgot how it used to work in the first place.
Also, there's two kinds of "sequences" you need to consider, and you need to consider if you're going to address them in the same way or different ways. First, there are animation sequences. When animating a visible bot, animations need to flow in certain orders. Some Paragon Chat will do for you automatically, some not. You need to learn how to do that, and how to write your bot so it can understand what state it is in, and what it should do next when it comes to animations. Separately, you will probably one day want to give your bot a script or sequence of commands, and each command may or may not require many different actions by your bot to perform them. Don't be afraid to stop and think about this problem carefully, taking a break from coding the bot if you need to, until you have a solution you like. This is probably a critical part of your bot development because it will influence how you interact with the bot from now on. My recommendation is to try to disconnect what the bot literally does from moment to moment and the higher level tasks it has to perform, if for no other reason than it becomes easier to try out ideas, and if they don't work to discard them and try something else without having to make major changes to the bot.
Above all, remember that if you break up the task into manageable components, each one with a payoff, you can better manage your own expectations and keep development moving without getting bogged down in trying to do too much at once. If you keep trying to swing for the fences, you'll never get on base.
Here's a partial list of my intermediate bot goals as I was developing the bot (much of which has been documented in this thread):
- Log into XMPP correctly, see my chat in pidgin.
- Send and receive chat on the Atlas Park broadcast channel between bot and a player
- Receive command by tell, interpret as a query to City of Data website, send response to sender
- Connect to meta channel, see other player data
- Become visible using costume of logged in player, doesn't require costume inquiry code
- Add keyboard controls so can move bot, turn bot.
- Implement walk, run, fly animations in conjunction with keyboard controls
- Implement costume code, allow bot to appear as any costume
- Implement grow, shrink with costume code
- Implement basic in-game command processor to send commands to bot, implement "comehere" and "faceme"
- Refactor controls, add goal queues and basic goal processing
- Implement basic "followme"
Small steps, each one with an achievable and demonstrable goal. And remember that from the very beginning, the goal was to make a shadow boxing combat bot. These little goals are all achievable and fun in and of themselves, but they are also heading in that ultimate direction.
At the moment my bot is 990 lines of very organic (i.e. messy, ugly, ill-documented, poorly error checked, highly unoptimized) python code with an additional 320 lines of costume stanza library code. Technically speaking, I started actually writing code about June 30th (http://www.cohtitan.com/forum/index.php/topic,10947.msg185173.html#msg185173), which means at this moment in time I've been working on this, not every single day, for about 63 days. I've been averaging 21 lines of code a day. In other words, on an average day, I sit at my computer, think about the bot, write this:
elif self.cmdQueue[0][0] == 'comehere':
distance = 5.0 # how close to get
x1 = self.position[0]
y1 = self.position[1]
z1 = self.position[2]
x2 = float(self.cmdQueue[0][1].split()[0])
y2 = float(self.cmdQueue[0][1].split()[1])
z2 = float(self.cmdQueue[0][1].split()[2])
xx = x2-x1
yy = y2-y1
zz = z2-z1
k2 = math.sqrt(xx*xx + yy*yy + zz*zz)
xk = xx/k2 * distance
yk = yy/k2 * distance
zk = zz/k2 * distance
self.position[0] = x2 - xk
self.position[1] = y2 - yk
self.position[2] = z2 - zk
print "Debug: popping comehere, moving to " + str(self.position)
self.send_pcu()
del self.cmdQueue[0]
and then I go out and see Ant-Man. I'm not saying everyone should be able to code, or everyone should want to do this, but if you can code, and you want to do this, I'm here to tell you its doable, and doable without a herculean level of effort (although everyone has different skill levels with code, I'm talking relatively speaking).
Go. CODE. MAKE BOTS.
-
My strongest recommendation here is, like you should have been doing up to now, keep every version and iteration of your bot. When you start refactoring code, make sure you never, ever, EVER delete any previous version, particularly versions that worked even if they didn't do exactly what you wanted. Nothing is more disheartening than "breaking" a working bot, and discovering you forgot how it used to work in the first place.
I would highly recommend using a source code repository for this reason. Depending on your preferences, you might like Mercurial, Git, or Subversion better (leaving aside things like Perforce). Each has different strengths and weaknesses. On Windows, you can grab the Tortoise version (TortoiseHg, TortoiseGit, and TortoiseSVN respectively) to integrate into explorer and get a nice GUI, not only for the repository itself but also to tools like Diff and Merge.
I have all three, but I mainly use Mercurial, because that's what I am used to. It's what I use for other projects (including UE4 game dev and non-programming projects like 3D models).
In fact, all of my COH costume files, keybinds, screenshots, chatlogs, Sentinel+ files, and demos are also kept in mercurial (it's primarily for version control, so it's not strictly useful for some of these files unless I want to see if another copy is different, but e.g. character mugshots sit alongside my character backgrounds and archived wiki pages, so why not). So are my Paragon Chat config files and database (I copy the current database over to the repository working folder only after making major changes, because I don't feel the need to make a commit every single time I change coordinates or zones).
For certain repositories, I keep a copy on Google Drive or Dropbox for offsite backup; you can easily push or pull changes from one repository to another to keep them in sync if your primary repository is not in the local Google Drive or Dropbox folder.
My biggest problems are 1) remembering to commit and 2) remembering enough of what I changed to make a good commit message. My repositories are littered with commits to binary files (so I can't easily see diff) with comments like "Probably nothing important" or "Vaguely remember X, or maybe Y"
-
Technically speaking, I started actually writing code about June 30th (http://www.cohtitan.com/forum/index.php/topic,10947.msg185173.html#msg185173), which means at this moment in time I've been working on this, not every single day, for about 63 days.
Oh wait, I forgot that in odd numbered years July has 31 days, not 61. That should read 33 days, not 63 days. Wow, that means I've been writing 40 lines of code per day. I'm awesome.
-
I would highly recommend using a source code repository for this reason.
I started using git a short while ago, but I'm not a frequent git user so I can't comment yet on its efficacy firsthand. I find checkin-checkout to be a little more overhead than might be reasonable for such a small and single developer project like this, but I wanted to see if there was some reason for doing it anyway that might compensate for the overhead.
-
Ooooh, this is a nasty one. For those using python and sleekxmpp, I noticed what appears to be a reproducible and horrible bug. I was trying to extract channel information from message stanzas, because I only want the bot to obey commands sent as tells, as opposed to things it might overhear on local, say. Long ago, I wrote this:
class pcChannelStanza(ElementBase):
namespace = 'pc:message'
name = 'channel'
plugin_attrib = "channel"
interfaces = set(('id'))
registerStanzaPlugin(Message,pcChannelStanza)
However, when I try to extract the channel id, nothing happens:
print msg['channel']['id']
u''
I couldn't figure out what was going on, until I dumped the value dictionary out of the msg stanza. Incredibly, if I just do a x=pcChannelStanza() and then x.values, I get this:
[u'lang':'', u'i':'', u'd':'']
That's right, the interface name "id" was being exploded into its constituent characters, and the stanza was being defined as having two attributes, "i" and "d". Apparently, this happens whenever you define a stanza with exactly one interface. I'm not quite sure why yet. But this fixes it:
class pcChannelStanza(ElementBase):
namespace = 'pc:message'
name = 'channel'
plugin_attrib = "channel"
interfaces = set(('id','pcfoo')) # there is a weird bug where if there is only one interface, its name
# gets exploded into its individual characters as interfaces
The extra unused attribute fixes the problem. Maddenly insane. Fortunately, <channel /> is the only stanza that has one and only one attribute (at the moment) so its the only attribute affected by this (so far as I've discovered so far).
Oh, and also, this fix won't work until you delete your pyc files. Because if you radically change your code python updates those, but apparently if you change this one definition the optimizer knows you don't use that extra attribute ever, and doesn't update the pycs. Only took me three hours to figure this out.
(https://images.weserv.nl/?url=www.authoritydomains.com%2Fblogs%2Fwp-content%2Fuploads%2F2010%2F09%2Fangry-computer-large.jpg)
-
I started using git a short while ago, but I'm not a frequent git user so I can't comment yet on its efficacy firsthand. I find checkin-checkout to be a little more overhead than might be reasonable for such a small and single developer project like this, but I wanted to see if there was some reason for doing it anyway that might compensate for the overhead.
If awesome were a goose, Git would be the sauce.
It's light-years ahead of Subversion. In theory, Mercurial has feature parity, but I never could get the feel for it. IMO Mercurial tries to simplify things too much for the average user, and makes it difficult to do interesting things with, like rebasing (rewrite history on your local branch before merging it back to master). The git ideology of branch early, branch often, merge often I've found works well in small groups of people. Mercurial doesn't offer lightweight branching out of the box (all branches in Hg are permanent), though IIRC there are addons to do it.
Like many awesome things, it's targeted at programmers, so there is a steeper learning curve than most SCMs before you figure out how to make it do awesome things. There comes a moment with Git where the architecture behind it just 'clicks' and you realize that while its primary interface is a version control system, it's not just a version control system. It's a content-addressable filesystem with mutable pointers that represent the head of a branch. Once you come to that realization, the reason why it works the way it does and how to leverage that makes much more sense.
But most importantly, it's damn fast when it comes to large projects, and it doesn't need a whole ton of dependencies to install. No scripting language interpreters or Webdav libraries or whatever required.
-
There comes a moment with Git where the architecture behind it just 'clicks' and you realize that while its primary interface is a version control system, it's not just a version control system. It's a content-addressable filesystem with mutable pointers that represent the head of a branch. Once you come to that realization, the reason why it works the way it does and how to leverage that makes much more sense.
It'll probably take a bit more usage for me to reach that point with git. I can kind of see what you're describing already, but the live-wire connection hasn't been made yet.
Actually, I've heard ZFS described in terms very similar to how you describe git, and there are strong analogies between the two technologies, some probably not entirely coincidental.
-
In theory, Mercurial has feature parity
As I said, I'm not an expert in either, but I am passingly familiar with both. One thing I did after sticking my sources into git was realize I didn't need to keep the long names around anymore: i.e. pcbot1.02-refactor-controls.py. I could just call that pcbot.py and let git handle version comments in commits. And *then* I realized I wanted to import my entire previous source history, from pcbot-0.1.py to the latest versions. So I just recreated my repository and threw those files into it one at a time, scanning and committing as I went along**. Then deleted the last one and added the latest pcbot.py source. Bingo, automatic history with automatic name tracking.
I don't think Mercurial does that in quite the same way. Which is to say, Mercurial has its own way of tracking such renaming histories, and because reasons, I'm pretty sure it takes more work (philosophical difference in design, as I understand git's history).
That's a totally unfair comparison point, but life's not fair.
** Even that's not really the most efficient way to do that in git, but I'm on Windows so sue me.
-
The goal queue idea is brilliant and I will probably steal from that idea at some point.
I suppose I'm going to have to get setup with Git just because my framework software is already setup to load modules directly from Git.
I've got my Paragon Chat back end to ErrBot going. With a decent working knowledge of all of the moving parts at this point, it was a trivial operation to define a new backend instance based on the XMPP backend and override the event handling stuff to make it send the XML payload up to ErrBot along with the basic presence and message events.
So trivial, that I was tempted to contact the ErrBot guys and ask, "Why didn't you guys just make this an option in the first place?"
That got me thinking about my goals for this project and the whole concept of "frontend" and "backend".
Ideally, what I want is for the person coming behind me to be able to be someone who wants to just say, "Load this costume file", or "tell me who is near me", or "tell me who is in the zone with me", or something similar. That person won't need to know about stanzas except as a means to perform "advanced" programming of the bot.
Another thing, though, is the whole concept of the "bot" as dispatcher and manager of a bunch of NPC's and the implications of doing that, where a lot of the work is going to have to happen at the "backend" side of things as far as managing multiple connections for multiple avatars.
The upshot being that maybe I *don't* want to just push the stanzas up a level, but instead make a separation between tasks that clearly belong at a "low level" compared to tasks that don't need to understand the underlying protocols to generate an effect.
I might actually have to take this seriously enough to draw up some kind of design document for it, if only to solidify for myself what it is that I actually want this thing to do beyond walk around Atlas Plaza and pretend to be knocked back if I pretend to hit it.
I actually have some specific things in mind but first I need to be able to follow someone and that means defining where all the pieces of that function fit, which puts me back at this point, heh.
Which leads me to some questions that I'm going to put in a seperate post just for the sake of clarity.
-
Here's the basic question - What are the implications of managing a group of NPC's?
Imagine this:
A group of five Hellions are lounging around in front of City Hall. A hero walks up. Four of them scatter and the fifth one interacts with the hero in some way.
What are the implications from the standpoint of the XMPP server, and from the standpoint of Paragon Chat?
We've established, I think, that all five of these Hellions need to have presence in the atlaspark and atlaspark_meta rooms. They all need costumes. They each need to have their own XMPP connection in order to communicate their presence and position. They each, therefore, need to have their own thread or their own process that is controlling their personal XMPP connection.
What does this mean for the server? Do the individual Hellion NPC's each need a unique JID? That is, does the bot login as mybot@chat.cohtitan.com with password "mybot" or does each Hellion require its own Titan account? Can one account join a chat room five times under five different nicks?
How is this many connections going to affect the server from a performance standpoint? Obviously, five isn't a lot, but five today in Atlas Park could scale up to 100 per zone in twenty different zones at once. (We should be so lucky as to be that popular, but dream big.)
From the "bot" standpoint, this encounter means forking and managing five connections while the Hellions are standing around, and then closing and releasing four of them after they flee, while sending the fifth one into whatever encounter handler takes care of its interaction with the player.
There are probably more implications that need thinking about but those are what come to mind initially.
If I create this encounter, am I going to be "hogging" resources on the server? What sort of resources should a bot owner be priveleged to use and to be considered to be "playing nice" with the other people who might be doing the same thing with their own resources?
-
If I create this encounter, am I going to be "hogging" resources on the server?
All good questions, but I think that's something that can only be answered through experimentation. Which is why I strongly advocate bot writing pioneers to run on their own private server during development, so that these questions can be characterized and answered.
At some point, Codewalker will allow NPC spawning in some fashion. I think the most efficient means of ultimately making a large number of NPCs function is to give bots spawning abilities and let them remote control actual NPCs. In effect to make the bot a remote mapserver of sorts. Until then, bots are going to potentially burn more resources per entity than desirable. But its difficult to be certain: it depends on what you want the entities to do, and how interactive they need to be. The devs had to deal with this kind of problem as well: the difference between the resource costs of a power like quickness vs a power like invincibility were enormous: it was possible for a single player to burn as much CPU resources as dozens of others under the right circumstances. Optimizing that, and in some cases simply not doing certain things, were all part of the development requirements.
Dream big, but start small. Honestly, federation might solve many problems in this area: GMs could host their own XMPP server and therefore spam it with as much traffic as desired to effect the kinds of gameplay they want, without hurting anyone else. You'd zone into that XMPP room, and then zone out. Doing this in *public* zones on *public* servers is where you have to be a lot more careful.
-
Because my experiments suggest that actually interpreting maps is something that might not be completely necessary at this stage, and because its kind of daunting, I've temporarily moved on to what might finally be the holy grail of my bot experimentation: implementing sparring. And when I think about shadow boxing, I realize that my bot needs another system.
With each iteration of the bot, I've been adding layers of abstraction and command and control. Its gone from literally doing exactly what I told it to do, to executing a set of instructions to do what I'm telling it to do, to finally trying to achieve a goal. You could say its gone from "do A' to "execute function B" to "execute function B until condition C." Do, do{}, and do until. Turns out I need one more: if/then.
Goal queues make sense when trying to execute a script of instructions, one at a time, one after the other, each one requiring performing a set of actions until a condition is met. But its not capable of emulating, in effect, the algorithmic equivalent of if/then/else, or alternatively "switch." And combat needs that, because combat is about reacting in different ways to different conditions. I need to be able to tell the bot "if you are more than seven feet away from the target, try to move to the target." "If you see the target facing you and executing an attack, at the right time react as if the attack hits (or misses)." "If you see this specific attack executed, fall down." The problem with the cmdQueue is that its designed to execute its instructions one at a time in order. But this type of thing doesn't happen in order, it happens as necessary. Basically I need to give the bot a set of "reactions" and the best way to perform those reactions is with a set of if/then style event driven commands.
So I'm making a new goal-like structure similar to the cmdQueue, which I'm currently calling "triggers."** The triggers list will contain individual elements similar to command queue goals, but instead of having a command and a set of parameters (which implicitly define a goal, like "turn towards this target" or "follow that entity"), it will have a conditional and a goal (identical in nature to the original goals). And unlike the cmdQueue which is explicitly intended to be executed in order, with command #2 only executing when command #1 completes, the triggers queue will be processed by having all entries checked on every clock tick to see if they should fire.
"Combat" would then be setting the right set of triggers: if distance > 7 then follow target. If distance <=7 and target facing bot and target animation in (attacks) then execute bot.reaction. If distance <= 7 and target not animating and random.chance then execute bot.attack_animation. The bot would now have both the ability to execute commands and have a stimulus-response set of behaviors that is also programmable. In fact, what I see happening is that combat will be enabled by command: "/t ArcanaBot, /fightme" would cause the bot to execute "/fightme sender" which would itself add a set of triggers to the bot's trigger list. "/t ArcanaBot, /nofightme" would delete those triggers. "/t Arcanabot, /fight target" would cause the bot to add the same triggers, but with a specified target rather than sender. With this command, a player could send the right commands to two separate bots that would then fight each other until commanded otherwise.
At that point, I will have achieved near proof of concept***. And then I can start bothering Codewalker to add FX spawning. I wonder if I can have this working by the weekend.
Also, one other thing. I started thinking today about authentication; well, I've been thinking about it since the beginning, but I started thinking about the specifics today. There are some ways in which I want the bot to respond to anyone that interacts with it. And then there are some commands I don't want just anyone to be able to send in some cases. So I've been thinking about a way to perform an authentication with the bot to put yourself on its whitelist, whereupon it will accept any command from you. If you don't, it will not respond to certain commands being sent its way. I could just hardcode a whitelist into the bot, but having the ability to do this dynamically has certain advantages. For one thing, you can support opt-in. Meaning, some things the bot will do to anyone, like maybe responding to a tell saying "hi." Some things the bot will do to anyone, provided they first opt-in to the bot's behaviors, like telling the bot "/opt-in" could cause it to allow you to say "/faceme." But if you try to teleport it away with "/comehere" the bot won't necessarily obey that command. For that, you'd have to login with "/admin password" and then the bot would start obeying all of your commands.
I'm not quite sure how I want to do this yet. But the time to start thinking about it is before I implement too many commands.
** "Triggers", "behaviors", "reactions" - I often spend an inordinate amount of time thinking about what to call something, because it bothers me later if I give something a bad name or I end up using it in a way completely contrary to its name. Even the cmdQueue is something I wished I had called the "goalQueue" but I'm not going to find/replace that one just to fix it at this point.
*** I won't be completely satisfied until the bots can brawl in Pocket D while being knocked off the walls and kicked into the furniture. So at some point, geo processing will become necessary. Its a small benefit requiring a huge amount of effort, but I refuse to let that mental image go.
-
Here's the basic question - What are the implications of managing a group of NPC's?
(snip)
We've established, I think, that all five of these Hellions need to have presence in the atlaspark and atlaspark_meta rooms. They all need costumes. They each need to have their own XMPP connection in order to communicate their presence and position. They each, therefore, need to have their own thread or their own process that is controlling their personal XMPP connection.
I am intending to use this exact structure, except it would be for contacts scattered across various zones, not random street mobs in one cluster. I'm not intending to replicate COH combat, just some narrative semblance of missions (i.e. "something to do"). And yeah, from the player's standpoint, they are talking to five different characters, but really they are talking to the same bot wearing five different masks in five different places.
What does this mean for the server? Do the individual Hellion NPC's each need a unique JID? That is, does the bot login as mybot@chat.cohtitan.com with password "mybot" or does each Hellion require its own Titan account? Can one account join a chat room five times under five different nicks?
Obviously it is possible to join multiple rooms with one account, because Paragon Chat is doing exactly this for the zone room, zone meta room, and every global chat channel you are on. But a single account can also join a single room multiple times. That's how running multiple copies of PChat and COH works when both clients are in the same zone. I've even been logged into the same zone three times on a few occasions already. One via Pidgin, and two via PChat.
For example, in the screenshot here (http://www.cohtitan.com/forum/index.php/topic,11192.msg189200.html#msg189200) the two characters on the left are me. If I also happened to be in the main galaxycity room via Pidgin at the time, that'd be a single account joining that room three times, and the galaxycity_meta room twice (one less because Pidgin doesn't care about the meta, though I could join it too if I wanted). Technically, I was also on various global channels multiple times as well, especially paragonchat.
Unless the server is set to cap the number of simultaneous logins on one account, there's no reason it can't be more than three. And there's certainly no reason a bot can't do the same thing that I do manually.
What does this mean for the server? Do the individual Hellion NPC's each need a unique JID?
The full JID differs, but only by resource name. The bits without the resource name are the bare JID, which I believe remain constant as long as the XMPP server name you log into and the account you use to log into it remain constant. The resource name can be set to anything (Paragon Chat puts your character name there). It gets auto-assigned by the server if you don't bother (which in openfire seems to gives the bot a hex number as a resource).
Which reminds me that the bare JID is one way that players can trust NPCs, at least to the extent they can trust anything. Imagine a mission that has a "go talk to Statesman in Independence Port" step, and the player walks up and sees three different Statesmans lounging around. Rather than being the City of Heroes equivalent of a "Reign of the Supermen" scenario, it probably just means there are missions run by three different bots that each have a "go talk to Statesman in IP" step. So which one do you actually talk to?
If the Statesman for your mission had a separate account from whatever bot you have that sends the player there to see him, the player would have no way of knowing which Statesman was correct other than trying to interact with each one until they get an interaction that makes sense (if any, given that bot writing skills do not equal story writing skills).
On the other hand, if your set of NPCs all use the same bot account, the bare JID will be identical for each NPC it controls, so players sent from one NPC to another could distinguish between NPCs that happened to have the same character name and costume. So there's a legitimate reason (aside from convenience) to code multiple NPCs as a unified system rather than as a bunch of independent bot accounts.
Another (probably more likely) scenario is a griefer version of Arcana's MirrorBot, cloning your Statesman to distract players from the "real" one.
If I create this encounter, am I going to be "hogging" resources on the server? What sort of resources should a bot owner be priveleged to use and to be considered to be "playing nice" with the other people who might be doing the same thing with their own resources?
My rule of thumb is always whether it, as a whole, improves the experience for players. There's no hard line, and I tend to assume members of the bot community are generally well-meaning and care about that. There is obviously potential for griefing, or even a DOS attack, but that potential exists already. Bots didn't create it.
The first bot on the public servers giving players "stuff to do" will be very beneficial. For bots 2 through N (where N is some unknown number), the drop due to declining novelty is way more than offset by the variety. Still, each is offset a little less than the previous one. Eventually there are too many bots using resources, and adding another one on the pile is more likely to subtract from the overall experience, even if the newest bot is the greatest thing since sliced bread.
Also, bear in mind that server resources aren't the only consideration here. XMPP is inherently less efficient than the normal COH protocol (as was discussed back when Codewalker posted the protocol), so a hundred bots in one area in Paragon Chat are going to be much worse on network bandwidth on the client end than a hundred mobs were in the same situation when the game was live. It uses higher resources strictly on each client machine as well, since there's an extra XMPP -> COH translation step going on as well as communication between PChat and COH. As Codewalker said awhile back, performance was not the primary design consideration.
Among the implications, it behooves you to default to NOT put any given bot or NPC into Atlas Park without a reason, as that will inevitably be the zone that sees the most traffic. My intent was to have a single avatar in each of the starting zones to direct people to "contacts" in less well-traveled zones, like blueside initial contacts, and even that was more for visibility than out of necessity. (I was never a huge fan of AP, but I recognize the practicality of it being the primary starting zone.)
Of course, all of this presupposes that I can ever wrap my head around sleekxmpp. The official docs are ... special (http://sleekxmpp.com/howto/stanzas.html). Fritzy's are a little better, but the info here (https://github.com/fritzy/SleekXMPP/wiki/Event-Handlers) didn't help. Turning to code, the demos are enough to let me figure out how to log in but don't seem to handle enough of what I need to know to implement PC's protocol, and the example code that ostensibly does is missing some context. I'm also apparently missing something important in the code posted here. I resorted to looking at the XEP plugins, but so far it hasn't helped me get any further than a more elaborate version of the MUC bot demo.
But damn have I nailed using Python's argparse and logging modules. I got your timestamps. I got your fancy loglevels. I got your multiple log streams, so I can have both console and file debugging without duplicating every single message as both a print statement and a call to a logging method. I can even have different loglevels for the logfile and the console.
Based on the debug messages sleekxmpp itself generates in my logs, the bot is receiving u stanzas and such; it just isn't reacting to them. So it has to be something I've done wrong.
I did quickly put in a quit command, though, so I can stop the bot more easily.
-
so a hundred bots in one area in Paragon Chat are going to be much worse on network bandwidth on the client end than a hundred mobs were in the same situation when the game was live.
The XMPP server that we're running is set to limit each zone to a max of 30 players in order to keep the exponential scaling of network traffic under control. Pocket D may be set higher (50 I think) as many people stand in one place for long periods of time there.
That's implemented as a cap on the number of occupants of the meta channel. The broadcast channels are set a bit higher so that people can join them from other XMPP clients.
To reference an earlier example, if you logged in 5 times as players pretending to be Skulls, then you'd be taking up 5 slots in the zone. If you created 29 Skulls that way, only one player could actually get into King's Row to see them. Any subsequent players would be unable to join the room, and their client would fall back to joining "King's Row 2" instead.
-
The XMPP server that we're running is set to limit each zone to a max of 30 players in order to keep the exponential scaling of network traffic under control. Pocket D may be set higher (50 I think) as many people stand in one place for long periods of time there.
Yeah, there's that, too (I'm running Openfire as well).
I just mentioned a hundred as an arbitrary large number to emphasize the point.
To reference an earlier example, if you logged in 5 times as players pretending to be Skulls, then you'd be taking up 5 slots in the zone. If you created 29 Skulls that way, only one player could actually get into King's Row to see them. Any subsequent players would be unable to join the room, and their client would fall back to joining "King's Row 2" instead.
Which just made me notice a wrinkle for bots ... they will not automatically appear in extra zone instances.
-
Which just made me notice a wrinkle for bots ... they will not automatically appear in extra zone instances.
Good! If they did auto-spawn in new instances, then given CW's scenario, we'd have hundreds of instances with only 1 person in each!
I think perhaps we'll need bots to be authorised by the server somehow, so that there aren't too many running and taking up people-slots...
-
Good! If they did auto-spawn in new instances, then given CW's scenario, we'd have hundreds of instances with only 1 person in each!
I realize that. I wasn't arguing they should; I was just pondering the implications for bots operating as a mission chain.
When the bot sends someone to Skyway to talk to another NPC contact, it can't rely on the player actually being able to locate the NPC there; the bot might be in a different instance. Ok, its really unlikely for that zone, but still.
Of course, a bot can check which instance its avatars are in and direct the player accordingly, but that doesn't help when the player can't actually get in a specific instance.
I suppose one possible solution is to give the player the ability to tell the bot which instance they are in and have the bot try to move the npc to meet them.
-
Good! If they did auto-spawn in new instances, then given CW's scenario, we'd have hundreds of instances with only 1 person in each!
I think perhaps we'll need bots to be authorised by the server somehow, so that there aren't too many running and taking up people-slots...
Unfortunately, *these* bots are people, or rather operate identically to people. At some point Paragon Chat could have a bot API, which means anyone that anyone using it would be distinguishable from actual players. But you'd always have to deal with the possibility that some of the players logged in are not actual people. Even without XMPP bots, there are ways to bot the City of Heroes client itself.
The presumption is that in a chat system with little gaming infrastructure and no rewards, there is little incentive to do anything that isn't of at least some nominal community service. But its one of the reasons my particular project is focused on programmers and not users. I have no intention of making an end-user friendly bot, so to speak, because I'm more interested in helping programmers write their own. I'm currently operating under the compromise position that people with skin in the game, so to speak, are less likely to abuse the privilege.
-
I realize that. I wasn't arguing they should; I was just pondering the implications for bots operating as a mission chain.
When the bot sends someone to Skyway to talk to another NPC contact, it can't rely on the player actually being able to locate the NPC there; the bot might be in a different instance. Ok, its really unlikely for that zone, but still.
Of course, a bot can check which instance its avatars are in and direct the player accordingly, but that doesn't help when the player can't actually get in a specific instance.
I suppose one possible solution is to give the player the ability to tell the bot which instance they are in and have the bot try to move the npc to meet them.
Or you can make sure that the bot is always in an instance that has a low population count, even going so far as to deliberately spawn a new instance to make sure its empty (except for the bot) and directing players to go there. In a sense, you'd be instancing the zone into a pseudo-private map.
None of these problems will have complete solutions because we are dealing with a system that currently has many technical limitations. However, there's two ways to approach those limitations. You can try to technology your way past them, or you can try to invent gameplay that works around them. If you are writing a bot for a specific gameplay purpose, then in a sense you're a game dev, and your task is to do both and try to meet in the middle. You have to simultaneously try to create the technology you need, while accepting the fact the technology you have will never be perfect and your gameplay ideas have to accommodate those limitations.
Incidentally, I believe players should announce their instance in their presence stanzas along with their map location. I'm pretty sure that's part of the map attribute, although I haven't had to deal with that yet directly (since I've never had more than three entities on a map simultaneously and I'm on my own server).
-
How about something like an extra meta channel, for bots only, that can only be joined programatically and that would sidestep the zone limit, but have it's own limit?
-
How about something like an extra meta channel, for bots only, that can only be joined programatically and that would sidestep the zone limit, but have it's own limit?
The reason why zones have limits in the first place is because XMPP can, in some circumstances, increase its traffic quadradically with number of people in the room. That's because I say something and everyone else has to hear it, given n players, the number of XMPP messages scales as some factor k * n * (n - 1). Actually, its worse than that because XMPP sends you back your own messages, so really its k * n * n or kn^2. And this happens most strongly in the meta channel. So long as bots participate in the zone meta channel, they would be contributing to the reason for the zone limit. If they didn't participate in a zone's meta channel, they wouldn't be able to track the locations of other players in that zone and would be invisible.
If there was a way to segregate bots that would still allow them to functionally exist within the zones, we could generalize that solution to break up players into subrooms as well and allow more players per zone instance without suffering from the quadratically scaling message issue. That's actually a technical idea that was discussed earlier, but its complicated and unlikely to arrive anytime soon.
-
Well, we work with the system we're given. There are a ton of mission maps as well as zone maps, and if I need to tell someone to "Go to Jacaranda Vista ((/mapmove 1134))" and make it known ahead of time that people need a zone.cfg that has the latest collection of maps in it, well, that's the price of admission for directing someone to a place where you can reasonably expect that running twenty NPC's at once is going to be acceptable from the standpoint of any live people visiting the zone.
A considerate bot would keep a list of possible mission maps and first check whether a "_meta" for the proposed mission currently exists and if it does, try another one down the list.
-
Okay, I knew there was a reason why the "Where do you think I am?" question was interesting.
Our bots are blind. They live in Paragon Space where they are surrounded by impassable barriers but they cannot detect them.
The only way for my bot to know what is in front of it is to either know how to read and interpret a map of the city's geometry or to be able to ask some central authority, "I am at point X, Y, Z. What is around me?"
Or, at a bare minimum, tell the client my bot is focused on to "Hey, be my eyes."
Sure, in an ideal world, I'd load up a map of the city and analyze it for collisions. The plain fact of the matter is, though, that I not only don't know how to do that, I just don't care about doing that. All I care about is getting my bot from point A to point B following reasonably close to the path taken by the player.
I'm as impressed as Hell with Arcanabot, but I wonder what happens if you hop a fence and then tell her to /followme?
With no knowledge of the existence of the fence, Arcanabot is either going to get hung up or she's going to end up teleporting past the obstacle when, by her own calculations, she ought to be much farther along her path and she spits out a <U> that puts her where she "ought to be".
I've called it a cheat up until now, but in a world where a bot is blind, becoming the bloodhound that sniffs along the player's trail isn't a cheat, it's the only practical way to travel along a route that you know will take you where you want to go without teleporting all over the place or simply moving through large obstacles like buildings and trees and mountains and mid-air for that matter.
-
There's another alternative, that goes back to the running on rails thing.
You could create your own map of the city that has rails running around the major obstacles. When a bot finds itself near a rail, it has the option to hop onto it, ride it to the end, and then hop off and resume its goal seeking.
That won't help with flying into a building but it would cover the basic ground-pounding follower scenario.
-
I'm as impressed as Hell with Arcanabot, but I wonder what happens if you hop a fence and then tell her to /followme?
With no knowledge of the existence of the fence, Arcanabot is either going to get hung up or she's going to end up teleporting past the obstacle when, by her own calculations, she ought to be much farther along her path and she spits out a <U> that puts her where she "ought to be".
This is why experimentation is important. Actually, neither of those things happens. First of all, collisions occur between player capsules and surfaces. There aren't really any solid objects in City of Heroes: when you collide with a building, you collide with one of its walls (or rather a polygon that's part of the wall). But if you get past the wall, you are now within the structure and there aren't any more collisions until you hit another wall. You don't get stuck on one side of a building until you suddenly teleport to the other side. At worst, you momentarily collide with the outer surface of the building for all of one frame of animation, and then when you send a <u /> that places you past that surface your bot "teleports" the tiny distance it takes to get from one side of that 2D surface to the other side. Its something you can't even see. See this: https://www.youtube.com/watch?v=JWj3IXMwzNo
Note the bot basically just walks smoothly through that building and the short wall.
However, that's not to say that the combination of "ghosting" and client-side prediction can't make strange things happen. Take a look at this: https://www.youtube.com/watch?v=SKOdqohZSiI
I should point out that at no time *ever* does my bot *ever* touch roll. I only touch pitch and yaw. In every single <u /> my roll angle is zero. So what's happening here? Obviously, the bot's defiant belief its intangible combined with a client side predictor that doesn't believe that combine to create some very odd effects which are not obviously emergent. Because the bot is visibly "rolling" over the wall in a weird sort of almost-hop over, and yet its only being told to walk straight through the wall. The only place that behavior can come from is client-side prediction. Why prediction predicts that is quite frankly inexplicable to me, but I'm sure there's a good reason. But without full knowledge of the precise algorithms for prediction, you'll probably have to experiment to figure out what can happen.
Bottom line, though, is that without the bot calculating collisions it will always ultimately get to where you tell it to go, without getting "hung up" on geometry and it won't get there by jumpy teleports. It might get there with some odd movement prediction, but that seems to be a corner case.
-
There's another alternative, that goes back to the running on rails thing.
You could create your own map of the city that has rails running around the major obstacles. When a bot finds itself near a rail, it has the option to hop onto it, ride it to the end, and then hop off and resume its goal seeking.
That won't help with flying into a building but it would cover the basic ground-pounding follower scenario.
City of Heroes uses a technique called "beaconing." The maps have invisible "beacons" placed at strategic locations and paths laid out between them. When a critter needs to get from point A to point B, it locates the nearest beacon and travels to it. Then it finds the beacon path to the beacon that is the closest to B. Then it runs along that path to B. Then it goes from that beacon to B.** The beacons and beacon paths act as safe "highways" for the critters to navigate the map, knowing someone else has "cleared" those paths. The only problematic parts then become the path to get from A to the nearest beacon, and the path to get to B from the nearest beacon.
You could build your own beacon system for each map you want the bots to nagivate, placing*** beacons and paths strategically to the only locations you care about. If you choose your beacon locations carefully enough, you don't even need to worry about ground height, because you can ensure that the path from beacon Alpha to beacon Beta is always parallel to and close enough to the ground between those beacons (sticking to roads and sidewalks helps).
Beacons don't help me for what I want to do, although the alternative for me is a do-it-yourself simplified space map of boundary boxes that roughly approximate the map volume I care about. But I think I have a legitimate shot at figuring out the basics of geo-mapping; at least enough to compute volumetric maps out of them. It can't be all that much harder than trying to figure out the animations sequencer or power database starting from nothing.****
** At least that's how I understand the system to work. I've never really messed around with the beacons to be honest, because my travels through the City of Heroes game system never required I master NPC pathing. Until now, of course.
*** Understanding that you don't literally need to "place" anything anywhere, you just need the bot to know where you decided they should go.
**** Yes it can.
-
Ah, I was still stuck in my "set it and forget it" frame of mind. Arcanabot has a fast clock. Fast enough, at least, that the duration between ticks, and therefore the duration between <U> footprints is so short that she isn't noticeably impeded by the barrier.
That "rolling over the top of the fence" effect is pretty interesting. I have a theory about that based on what I've seen happen when two bots collide, or when I've blocked my bot with my own avatar.
I'd theorize that on occasion Arcanabot emits a <U> that causes her collision box to intersect the collision box of the fence, requiring that they repel each other until they're both outside of each other's bounds. Since Arcanabot already has "forward" velocity, the game engine helpfully repels her "up" instead of repelling her "back" or in some other direction. The temporary pitch/roll alteration might even be baked into the fence - I seem to recall that there were NPC's that appeared to put their hand on top of a fence or wall and vault over it as opposed to just jumping like a player would do. A change in pitch to the angle demonstrated by Arcanabot would accommodate a change in animation that involved kicking your legs out to vault over a wall.
At any rate, the fence can't move. By altering Arcanabot's apparent orientation and velocity, the game gets her out of the "red zone", so to speak, and then Arcanabot's own movement code puts her back on the correct path and orientation.
This still leaves us with the problem that Arcanabot is Kitty Pride. She just phases through anything that gets in her way.
With no map server to tell you where surfaces exist, though, I suppose that's going to be an unavoidable fact of bot life.
I haven't had a lot of time to work on my bot lately but I think I'm going to go ahead and try out the bloodhound approach just to see if it looks "unnatural" and to what extent it appears that way. I'm thinking that the CoH devs probably had the best idea with the beacons and rails assisting what was otherwise a seeker that was probably similar to what Arcanabot uses.
-
Arcanabot has a fast clock.
By default, 30 ticks per second, adjustable:
while pcbot_keyboardTracker.processPyEvents():
pcbot_meta_key.processKeyState(pcbot_keyboardTracker)
pcbot_meta_key.take_action(pcbot_meta)
pcbot_meta.doGoal()
pcbot_meta.move_bot() # goals set movement flags, so move must be after goal
clientClock = time.clock() * 1000
clientLoop = clientLoop + 1
#print "Debug: clock loop: " + str(clientClock)
clockAlign()
clockAlign does this:
def clockAlign():
miltick = (time.clock() * 1000 % 1000)
nexttick = (int(miltick / 33.33) + 1.01) * 33.33 + 1
#print "Debug: sleep:",miltick,nexttick,nexttick-miltick
time.sleep((nexttick - miltick)/1000)
Yanked right out of my "MMO 0.001" experiment. Also, "clientClock" is used to timestamp every roster update with a millisecond timestamp, which I'm going to use for prediction when I implement prediction. I could adjust this up or down by adjusting the clockAlign calculations to 1 frame per second, 8 frames per second, or whatever.
I'd theorize that on occasion Arcanabot emits a <U> that causes her collision box to intersect the collision box of the fence, requiring that they repel each other until they're both outside of each other's bounds. Since Arcanabot already has "forward" velocity, the game engine helpfully repels her "up" instead of repelling her "back" or in some other direction. The temporary pitch/roll alteration might even be baked into the fence - I seem to recall that there were NPC's that appeared to put their hand on top of a fence or wall and vault over it as opposed to just jumping like a player would do. A change in pitch to the angle demonstrated by Arcanabot would accommodate a change in animation that involved kicking your legs out to vault over a wall.
You're probably close. City of Heroes has a way of calculating which surface is "the ground" by performing a calculation that determines which surface is the closest to the entity in the downward direction. Meaning, given the entity - in this case the bot - position(x,y,z), calculate a vector pointing in the downward direction and see what is the first polygon that vector intersects. That is "the ground." Generally speaking, its the ground. However, if I set position to a point within a "solid" object like the wall, that calculation could generate a normally nonsensical result: it could end up pointing, say, right at a funny tessellation in the wall. Finding "the ground" is important because it tells City of Heroes things like what friction to apply to the character, and of course also what the surface normal is - what angle the ground currently slopes. If City of Heroes picks up a weird surface as "the ground" at the instant I phase through the wall, it could cause it to believe "the ground" is sloped at a weird angle and force my bot to roll in the direction of the slope. Its also possible that the collision code could bump me in a strange direction at the same time, which could combine to produce the effect.
Its still a strange effect because while that all makes sense on the surface, there's still a few unanswered questions about why it does it quite the way it does. There are still some things about the client I don't fully understand, if not what it does then why it does it. For example its interesting to me that if you change an entity's location dramatically, the original devs decided the entity should move there at supersonic speed rather than just teleport there. I know that it happens, but not what the thinking was. Maybe they just thought it was cool.
-
For example its interesting to me that if you change an entity's location dramatically, the original devs decided the entity should move there at supersonic speed rather than just teleport there. I know that it happens, but not what the thinking was. Maybe they just thought it was cool.
The interesting thing is that it's intangible while it's doing it. I first encountered that effect when I was launching my bot across Atlas Park and then recalling it to its "origin". I ended up pretty much shrugging and saying, "Well, okay then." Despite it being a visual artifact of the bot's motion, there was *not* any of the collision that ought to have happened if the bot was really moving along that line. ErrBot had to move from a lower elevation to a higher elevation across a hundred yards or so of ground that included fences, sidewalks, walls and all kinds of obstacles that it ignored entirely.
Basically, the bot did teleport but it still left this visual effect connecting its start and end points. I guess it means that if you're going to literally jump from point A to point B all at once that you need to turn invisible first unless you want to appear to be moving so fast that you vibrate right through any solid objects in your way.
-
Obviously, the bot's defiant belief its intangible combined with a client side predictor that doesn't believe that combine to create some very odd effects which are not obviously emergent. Because the bot is visibly "rolling" over the wall in a weird sort of almost-hop over, and yet its only being told to walk straight through the wall.
I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?
-
For example its interesting to me that if you change an entity's location dramatically, the original devs decided the entity should move there at supersonic speed rather than just teleport there. I know that it happens, but not what the thinking was. Maybe they just thought it was cool.
That's the client doing that. All position updates are subject to interpolation by the client. It's not as obvious with small updates, but it does happen, because the server doesn't send 30 updates a second for every entity. Even clientside prediction is limited to 30 "frames" per second, and many systems can render faster than that. With larger changes, the interpolation gets spread out over multiple frames, though I haven't sat down to hammer out the exact details.
In the network protocol, there's a bit you have to explicitly set when sending the update to indicate that the client should not interpolate, but rather instantly move the entity.
Paragon Chat currently only sets that bit in a few circumstances, mostly when first creating the entity and moving it from 0,0,0 once the costume arrives and the first update is received. I plan to add an attribute to the update stanza to indicate other circumstances when this is needed; such as when taking an intra-zone door (it already sets it for the local player but doesn't communicate that over XMPP).
Speaking of which, I've been unable to reproduce the issue you mentioned where sending presence again causes it to think the initial position hasn't been received. I've been testing with PC doing costume changes and could not recreate it, even when being careful to hold still so that a <u> stanza is not sent right away.
-
Speaking of which, I've been unable to reproduce the issue you mentioned where sending presence again causes it to think the initial position hasn't been received. I've been testing with PC doing costume changes and could not recreate it, even when being careful to hold still so that a <u> stanza is not sent right away.
Hmm, I will try to see if I can come up with a reproducible way to generate that issue. I might have to deliberately break the client to do it since I refactored that code to eliminate the possibility in the first place.
-
I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?
It seems unlikely because that would cause dramatically amusing errors in motion at other times but I can't say for certain such a bug isn't in there somewhere. I will retry those tests making sure the bot isn't attempting to follow me while I am not in the same xz plane, just to see what happens.
-
Also, there's an important distinction that I think should be noted. When I'm talking about clientside prediction, I'm talking about the player. The COH client does motion prediction only for its own player. It never speculatively predicts the location of other players, NPCs, etc. Those are effectively physicsless and are not even processed except to draw them. It relies on the mapserver to send regular updates that it can interpolate between, but introduces some intentional time lag so that those updates seem to be happening in real-time, even when they're not being continuously received.
That's probably the cause of the latency that you're seeing even when the XMPP server is local. The movement 'buffer' as it were seems to be integrated into the client. I haven't yet investigated all the murky corners of the protocol to see if there's any way to disable it within the protocol itself assuming a near-infinite speed connection.
Now, Paragon Chat does do speculative prediction as well as apply collision mechanics to the simulated players (mostly so falling doesn't go through the ground), which is why you often see people rubber-banding back when they stop moving. Eventually that will probably go away and be replaced by a model much more similar to how the game handles it natively, but more research is needed in that area to make it work with real latency conditions.
-
Also, there's an important distinction that I think should be noted. When I'm talking about clientside prediction, I'm talking about the player. The COH client does motion prediction only for its own player. It never speculatively predicts the location of other players, NPCs, etc. Those are effectively physicsless and are not even processed except to draw them. It relies on the mapserver to send regular updates that it can interpolate between, but introduces some intentional time lag so that those updates seem to be happening in real-time, even when they're not being continuously received.
That's probably the cause of the latency that you're seeing even when the XMPP server is local. The movement 'buffer' as it were seems to be integrated into the client. I haven't yet investigated all the murky corners of the protocol to see if there's any way to disable it within the protocol itself assuming a near-infinite speed connection.
Now, Paragon Chat does do speculative prediction as well as apply collision mechanics to the simulated players (mostly so falling doesn't go through the ground), which is why you often see people rubber-banding back when they stop moving. Eventually that will probably go away and be replaced by a model much more similar to how the game handles it natively, but more research is needed in that area to make it work with real latency conditions.
Multiboxers probably noticed this more than other players. If you multiboxed, movement in one session took a lot longer than people might expect to register in the other session, completely disproportionately to actual network lag. Client-side prediction works because we can usually only detect the difference between our own inputs and our own character's response to those inputs, but we're usually poorer judges of "world lag" because we don't really know when the world "wanted" to do something verses when it did it.
-
Well, my bot is finally reacting to PC stanzas.
And today's episode of Stupid Dyne Errors shall be called:
String Formatting Considered Harmful
When it came time to register the handlers for the custom stanzas, both Arcana (http://www.cohtitan.com/forum/index.php/topic,10956.msg187143.html#msg187143) and this example (https://github.com/fritzy/SleekXMPP/wiki/Event-Handlers) used a matcher like:
MatchXPath( '{%s}presence/{%s}pc' % (self.default_ns, pcPresenceStanza.namespace) )
That uses python's string formatting. See, in the versions of Python 2 that I am familiar with, the %s patterns get expanded to the values of the variables after the % operator. In Python 3 (and 2.7), this old way still works, but the preferred syntax for this has changed to "{}".format(variable) and -- having learned my lesson with the Byte slicing earlier -- I was just habitually converting this sort of thing as I went along.
Can you guess what I did? Look closely at what is present in both the XPath call above and in Python 3's syntax, yet absent from my description of the old Python 2 syntax.
Yep. It probably took you about 1/100th the time it took me. :-[
I saw the braces in the original version. But, thanks to their presence in exactly the same spots in the new syntax, I didn't see them see them. When I converted to the new syntax in situ, they were there already so I just called it a day.
My version ended up like:
MatchXPath( '{}presence/{}pc'.format(self.default_ns, pcPresenceStanza.namespace) )
Instead of "{jabber:client}presence/{pc:presence}pc", it was trying to match "jabber:clientpresence/pc:presencepc". Until I managed to get around the blind spot induced by the syntax change, I couldn't even figure out how the original version could have produced anything valid. Bad Dyne, no cookie.
(In my defense, I was distracted by other things that seemed more likely. I kept glancing at that MatchXPath, frowning, and then checking something else. After all, two different sources had used string formatting, and one would have to be a complete idiot to mess up a syntax conversion that simple, right?)
Unrelated, but I also discovered that joining the meta channel from Pidgin while also testing an invisible bot that chokes and dies (for a different issue) can confuse either Paragon Chat or Openfire, not sure which. It resulted in a duplicate of my character; it looked like me and followed my movements, emotes, etc. It's almost like I had a bot that would mirror my appearance and movements, without doing any of the work to get there. 8)
(https://i.imgur.com/GiKlcrml.jpg) (http://imgur.com/GiKlcrm)
-
Unrelated, but I also discovered that joining the meta channel from Pidgin while also testing an invisible bot that chokes and dies (for a different issue) can confuse either Paragon Chat or Openfire, not sure which. It resulted in a duplicate of my character; it looked like me and followed my movements, emotes, etc. It's almost like I had a bot that would mirror my appearance and movements, without doing any of the work to get there. 8)
That's odd. If I had to guess I'd say that Paragon Chat put an entry in an entity table for the pidgin client when it announced presence but because it didn't actually announce anything in the paragon chat namespace it wasn't visible. Then when you bot tried to become visible and crashed it somehow convinced Paragon Chat that the pidgin entity was a Paragon Chat entity, and cross linked it with the position and animation data for your actual Paragon Chat character. Whereupon you'd now have two entities with identical position and animation information. Then capsule bumping would separate the two and you'd have two identical entities echoing each other's movements and separated by a few feet.
-
Did you use the same nickname in the channel by any chance? I've noticed that while Openfire normally disallows identical nicknames from separate resources as per the XMPP spec, it allows them if they differ by case only. Once you do that, the room occupant status gets really weird and both hang around as long as one of them stays in the room, and don't become "unavailable" until the last client on that base JID leaves.
As Paragon Chat treats JIDs as case-insensitive, I could definitely see that confusing it and causing it to only recognize one of them as "self", then creating a proxy player for the other.
-
I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?
Now this is strange. I knew that my orientation calculations were working, because earlier I posted the results of my "faceme" command which actually worked. Today, /faceme doesn't work, it rolls the character. So you're right, the funny roll is due to the character trying to pitch up and rolling instead when running through the wall as my character jumps over it.
Except, I can't see what I'm doing wrong. In fact, Paragon Chat itself says I'm sending the right thing. When I do a /faceme where I'm above and in front of the character, instead of pitching upward the character rolls clockwise. However, this is what Paragon Chat says it received when I do that:
xmpp DEBUG RECV: <message lang="en" type="groupchat" to="arcanacoh@core3770/Violet" from="atlaspark_meta@conference.core3770/ArcanaBot_meta"><u p="906.5038956 0.0818596208137 -490.958855515" xmlns="pc:u" v="0 0 0" o="-0.393764839539 -1.58960360374 0.0"/></message>
Notice roll is zero. As far as I can tell, Paragon Chat confirms my roll in o= is always zero. Did Paragon Chat change PYR order in orientation at some point? I know this worked back when I posted this: http://www.cohtitan.com/forum/index.php/topic,10956.msg189244.html#msg189244. The code to do that is literally identical between that version and my current one.
No, it can't be an order thing. I just added roll controls to the bot that allow me to roll and nullify rolls. Even when roll around and then force roll to be zero, when pitch is nonzero roll won't nullify. I think there's something wrong with Paragon Chat's matrix calculations that are intertwining pitch and roll. Its almost as if pitch is always zero and roll is always pitch+roll, if that makes sense.
-
Gotcha. I added *pitch* controls to my bot so I can move freely. The problem is that while roll seems to be working correctly, pitch is not. Pitch doesn't factor in yaw. Meaning, if your orientation is 0,0,0 and you try to pitch up and down, that works. But if you turn 90 degrees, roll still works correctly about the axis you're facing in, but the pitch axis doesn't rotate with yaw, pitch now *becomes* roll because the pitch axis is now aligned with the roll axis, as if it were still pitching up and down relative to 0,0,0. And I think this bug has been there since the beginning of time, since when I test with older version of PC I see the same behavior. My pitch test from long ago worked only by coincidence: I'm facing close enough to 0,0,0 for pitch to appear to be working correctly.
-
Did you use the same nickname in the channel by any chance? I've noticed that while Openfire normally disallows identical nicknames from separate resources as per the XMPP spec, it allows them if they differ by case only. Once you do that, the room occupant status gets really weird and both hang around as long as one of them stays in the room, and don't become "unavailable" until the last client on that base JID leaves.
I'm pretty sure I did end up in the room as dyne and Dyne, yeah. In fact, I was just looking at a global chat muc that was doing exactly what you described (Paragon Chat was no longer logged in, but Pidgin saw two of me there). I should change Pidgin's settings.
-
And I think this bug has been there since the beginning of time, since when I test with older version of PC I see the same behavior. My pitch test from long ago worked only by coincidence: I'm facing close enough to 0,0,0 for pitch to appear to be working correctly.
Sure enough, the function that recomposes the matrix from the "o" attribute is using PYR order instead of YPR. Well, easy enough to fix, I just swapped it and the fix will be in the next minor release. Just goes to show about how often COH uses pitch that it's gone unnoticed until now.
That probably explains the odd rotations I had seen when dual boxing and looking at myself "flying" around with nocoll turned on.
-
Sure enough, the function that recomposes the matrix from the "o" attribute is using PYR order instead of YPR. Well, easy enough to fix, I just swapped it and the fix will be in the next minor release. Just goes to show about how often COH uses pitch that it's gone unnoticed until now.
That probably explains the odd rotations I had seen when dual boxing and looking at myself "flying" around with nocoll turned on.
Even I didn't notice anything amiss until now, and I wrote it off. When you're testing to see if something is working, you don't usually say "lets try that again, but facing in a different direction" although I really ought to know better, because that's *exactly* the test you should perform when testing orientation calculations. But since I was mostly working in the XZ plane trying to get "normal" motion working, 99% of the time both pitch and roll are zeroed out.
At least now my bot can do something my City of Heroes characters could never do: an Immelmann. I'm thinking of making it a flying primitive.
-
<frederick xmlns="FRONK-un-steen" age="young">From that fateful day when stinking bits of slime first crawled
from the sea and shouted to the cold stars, "I am man.", our greatest dread has always been the knowledge
of our mortality. But tonight, we shall hurl the gauntlet of science into the frightful face of death itself. Tonight, we shall
ascend into the heavens. We shall mock the earthquake. WE SHALL COMMAND THE THUNDERS, AND PENETRATE INTO
THE VERY WOMB OF IMPERVIOUS NATURE HERSELF!</frederick>
(https://i.imgur.com/QnuS72km.jpg) (http://i.imgur.com/QnuS72k)
(https://images.weserv.nl/?url=www.quickmeme.com%2Fimg%2F67%2F6759ff044c6366116a82b8507ccecdd4480d40b0af18462342931a05622791dc.jpg)
<igor>He's going to be very popular with the ladies.</igor>
And now I'm going to bed. *flop* :)
-
(https://i.imgur.com/QnuS72km.jpg) (http://i.imgur.com/QnuS72k)
Gratz!
-
Congrats. ;-)
-
Dynebot is now serving a (hardcoded) costume in response to requests instead of just parroting a hash. I wasted a bunch of time fighting the library, trying to get it to let me use the obvious solution of storing my desired appearance/part xml elements in a string and then stuffing that into the costume stanza. The closest I came was using the iq._set_sub_text method, which solved my previous problem -- how to put the description into my iq.reply().
Unfortunately, the underlying objects appear to escape the text contents of an element whether you want them to or not ... which is fine for a description, but as far as Paragon Chat was concerned, it turned the costume to gibberish. I could have delved into ElementTree or whatever, but at that point I just gave in and separated my string into objects. I'd have been better served hooking Dynebot up to my actual costume loader, but I was just trying to get the thing to supply an independent costume ASAP.
Currently, Dynebot is at 745 lines; building the stupid costume manually is about 6% of that, and a large minority of the rest probably consists of either comments, whitespace, or logging.
It has multiple commands (help, restart, quit, resend presence, resend position, and speak), the latter of which is mostly pointless because it only joins the meta channel right now. It keeps a list of nearby users for later use with local chat/text emotes whenever I get around to implementing them. When someone joins the meta channel, the bot emulates Paragon Chat's random delay before refreshing presence/position (since it doesn't move or change animations yet) via an asynchronous thread.
While I was held up by the costume thing, I added classes and handlers for the chat bubble's <scale> and <dur> pseudo tags on a whim, even though Paragon Chat doesn't actually support them yet, and I'm not sure why the bot would care. I suppose I could make it ask people using scale != 1 why they are whispering or yelling, and those with non-default durations why they're talking so fast or slow. Considered going ahead and templating team, sg, league, etc. chat, but then I realized that these would probably need to be implemented as completely separate rooms or whatever, rather than messages in the zone room using the channel element (because they used to work across zones).
I haven't decided what I want to do next. I really need to factor out as many bits of code that control the bot's behavior and appearance from the bits that are "generic PChat/XMPP stuff" as I can, but I expect that'll be fairly tedious.
-
Dynebot is now serving a (hardcoded) costume in response to requests instead of just parroting a hash. I wasted a bunch of time fighting the library, trying to get it to let me use the obvious solution of storing my desired appearance/part xml elements in a string and then stuffing that into the costume stanza. The closest I came was using the iq._set_sub_text method, which solved my previous problem -- how to put the description into my iq.reply().
That's not necessary. If you're got a stanza object defined, and you're setting the attribute values via its defined interfaces, then it's already an XML object. The costume stanza is just an appearance stanza and a bunch of part stanzas - Create a new costume stanza variable, make an appearance stanza, append it to the costume, make each part stanza and append it to the costume, then append the costume to the IQ reply stanza.
The stanza methods take care of the xml magic. You can even do like I did and save out myCostume.xml to a file on disk. If you build the stanzas at the same time you are building your hash string, it all happens organically. Your costume stanza gets generated along with your hash, and then you stick it in a variable where you can pull it out whenever you need to append it to a presence or IQ reply.
You're going to want to be able to load a .costume file sooner rather than later, so you might as well make that your next order of business.
I haven't decided what I want to do next. I really need to factor out as many bits of code that control the bot's behavior and appearance from the bits that are "generic PChat/XMPP stuff" as I can, but I expect that'll be fairly tedious.
I'm at this point myself. I have some other things going on that are demanding my attention and I'm having to consider some sort of direction that I'm taking this thing. I've identified three goals of increasing complexity that I believe I can accomplish with just the technology we currently have, though the implementation details will require some math that I haven't used since high school (and you don't want to know how long ago that was).
Pushing towards something that gives a tangible result gives me more motivation. At the moment I'm taking a break from the bot code proper, and I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up. I actually found that I could pull up practically an entire five or six year archive of the old CoH forums from the Wayback, but I don't think I've got the drive to write a parser that would spend the time to download each page, download all the individual posts, and then extract just the text of each post. What I do have is all of the lore bible docs that were released after the game closed, and all of the Paragon Times, and Know Your Adversary and other lore stuff that was on the official web pages. The initial results are sometimes rather hilarious but there's enough non-useful garbage in the story bible docs that I'm taking the time to edit out all the crap and make a "clean" data file for training the chatbot's "brain".
Once that's done, then I'll go back to the bot and add the code it needs to track everyone in whatever rooms the bot has joined, track their positions, respond to input on /Local or /Tell, and insure that local chat is only responded when the recipient is within 15 feet or so. Once all that works, I'll probably throw it online and see if anyone finds it amusing. ;)
-
The costume stanza is just an appearance stanza and a bunch of part stanzas - Create a new costume stanza variable, make an appearance stanza, append it to the costume, make each part stanza and append it to the costume, then append the costume to the IQ reply stanza.
That is precisely the solution I was trying to avoid, as it took a lot of fiddly reformatting to turn my already valid xml string into a bunch of code ... which just told the library to put my original xml back out. It was kind of overkill for what amounted to a short-lived test on the road to supporting full costume loading. But needs must.
The stanza methods take care of the xml magic.
No real xml magic should have been necessary. I just needed the library to let me use the xml that I already had. And it would have ... as long as I was content with having that xml escaped automatically, rendering it useless.
Naturally, I quickly found out how to do this as soon as I no longer needed to. Sleek may or may not provide support for what I was wanting, but python's xml library (which sleek is built on) does. It's actually pretty simple, so for posterity:
from sleekxmpp.basexmpp import Iq
from xml.etree import ElementTree
bob=Iq() # Just for a demonstration of the idea, since there's no actual stream here
tom = "<costume><appearance nosejob='1' /><part n='1' tex='Costume Data!' /></costume>"
# this works as long as the xml string has a root element. take out <costume></costume> and it complains.
tomxml = ElementTree.fromstring(tom)
bob.setPayload(tomxml)
print(bob)
The snippet prints:
<iq xmlns="jabber:client" id="0"><costume><appearance nosejob="1" /><part n="1" tex="Costume Data!" /></costume></iq>
(Which is obviously not full-fledged Paragon Chat format, but there's nothing that I know of preventing me from sticking pc:namespaces or whatever in the string.)
The reason I couldn't figure it out before was that I was looking in the wrong place; I was expecting sleek to have this capability. While I had a suspicion that I could get around it, as previously mentioned, I didn't want to go Standard Library delving in detail to sort out how. I figured I'd end up having to do something like reimplement _set_sub_text, only without the escaping, and that was even more work than the method that sleek wants. I wasn't expecting it to take literally five minutes to just dodge the entire issue, including the time to type the above test code.
You can even do like I did and save out myCostume.xml to a file on disk.
Are you serializing the xml objects or dumping the xml text to a file? If it's the latter, I suspect you'll need to do some parsing like the above to make use of the files.
You're going to want to be able to load a .costume file sooner rather than later, so you might as well make that your next order of business.
That is true.
I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up.
That sounds like fun.
-
Naturally, I quickly found out how to do this as soon as I no longer needed to. Sleek may or may not provide support for what I was wanting, but python's xml library (which sleek is built on) does. It's actually pretty simple, so for posterity:
from sleekxmpp.basexmpp import Iq
from xml.etree import ElementTree
bob=Iq() # Just for a demonstration of the idea, since there's no actual stream here
tom = "<costume><appearance nosejob='1' /><part n='1' tex='Costume Data!' /></costume>"
# this works as long as the xml string has a root element. take out <costume></costume> and it complains.
tomxml = ElementTree.fromstring(tom)
bob.setPayload(tomxml)
print(bob)
The snippet prints:
<iq xmlns="jabber:client" id="0"><costume><appearance nosejob="1" /><part n="1" tex="Costume Data!" /></costume></iq>
(Which is obviously not full-fledged Paragon Chat format, but there's nothing that I know of preventing me from sticking pc:namespaces or whatever in the string.)
The reason I couldn't figure it out before was that I was looking in the wrong place; I was expecting sleek to have this capability. While I had a suspicion that I could get around it, as previously mentioned, I didn't want to go Standard Library delving in detail to sort out how. I figured I'd end up having to do something like reimplement _set_sub_text, only without the escaping, and that was even more work than the method that sleek wants. I wasn't expecting it to take literally five minutes to just dodge the entire issue, including the time to type the above test code.
The sleepxmpp documentation does note this in the documentation for Stanza Objects: http://sleekxmpp.com/api/xmlstream/stanzabase.html. If you search that page, at the bottom you'll see the listed methods for Stanza objects, including setPayload(). It probably complains if you don't have the root element because without it the string might be valid XML but it cannot be converted to a valid (single) sleekxmpp XML object. You'd then have two independent XML objects (attribute and part) that need to be passed as a sequence (with each object in a different element), not a single string.
-
Pushing towards something that gives a tangible result gives me more motivation. At the moment I'm taking a break from the bot code proper, and I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up. I actually found that I could pull up practically an entire five or six year archive of the old CoH forums from the Wayback, but I don't think I've got the drive to write a parser that would spend the time to download each page, download all the individual posts, and then extract just the text of each post. What I do have is all of the lore bible docs that were released after the game closed, and all of the Paragon Times, and Know Your Adversary and other lore stuff that was on the official web pages. The initial results are sometimes rather hilarious but there's enough non-useful garbage in the story bible docs that I'm taking the time to edit out all the crap and make a "clean" data file for training the chatbot's "brain".
You're making a UniqueDragon bot? That's very meta.
I myself was forced to take a break this week, owing to work detonating on me. I'll probably have dug myself out of vendor-hell by mid-week next week and be able to resume work on the bot. In my never-ending search for ways to duck trying to make a map parser and space tree search algorithm, I'm thinking of building up my library of bot animation primitives. Given what I have, and given lack of support for things like ragdoll, I'm going to make a list of animation and animation combinations that replicate some of the behavior I want the bot to be able to perform. I "know" what I want and mostly how to find it, but I still need to track it all down and test it. I suspect it won't all exactly work the way I want without some kludging. I'm thinking of a way to make animation "libraries" composed of sequences of moves correctly timed, so my bot can call them without having to "think" about how to do them once I've hand-tested them. That's time consuming, but has immediate payoffs as I work, which I kind of need after the past week.
-
-
UniqueDragon! Wow, that takes me back!
I might have to train that into the chatbot, LOL.
-
Yeah, in hindsight, I can see what they were going for. It wasn't so helpful from the other side, though; I read that at least a dozen times while trying to work this out. (Mostly it read like a reference to StanzaObject.xml, which I already knew that method took.)
It's quite possible that I'm actually losing the ability to make connections like this, probably due to lifestyle factors.
Honestly, I'm still figuring out sleekxmpp. Some things work the way I expect, some things don't, and its therefore difficult at time to tell the difference between a bug and library goofiness. For example, as I mentioned upstream, don't ever do this:
interfaces = set(('id'))
if you want to preserve your sanity. Sleekxmpp will decide that interfaces is not a sequence with one entry but a single string sequence of entries, and will happily make interfaces with names "i" and "d". That has to be a bug in the library since its a set**, but not a bug I particularly want to hunt down at the moment.
** If I had to guess, they make it a set to prevent dups, then try to make the entries into an array one element at a time, and when there's only one entry that happens to be a sequence (string), the code gets confused and starts derefing the string.
-
UniqueDragon! Wow, that takes me back!
I might have to train that into the chatbot, LOL.
/jranger
No, just kidding, go for it.
-
Why no updates lately (from me):
1. Work got busy and complicated, slowing me down from a crawl to whatever it is snails do.
2. I'm currently at a point in my bot research that involves more research and less coding, so there's less interesting things to talk about. However, I'm hoping to have something interesting to show in maybe another week. Thread not dead yet.
-
This should be obvious to my fellow bot writers in this thread, but for any new bot writers and for the purposes of completeness, there's a new stanza to support the hacks necessary to get some of the travel powers working. Codewalker can confirm or correct, but the new stanza is:
<hack xmlns="pc:hack"><power name="powername" state="[off|on]" /></hack>
powername currently is either fly or superspeed. It is almost certainly used to communicate to other Paragon Chat users that entities with these powers enabled should have the appropriate VFX attached to them. Walk and Sprint do not send hack power state messages, probably because neither power requires any other Paragon Chat user to know it is turned on, because there's no visual difference for characters with these powers toggled on (except the animations they play which is already handled with standard <u /> stanzas.
The current apparent intent is to use <hack /> stanzas to signal to other Paragon Chat users when a particular continuing state is true or false for a particular entity to signal to them that Paragon Chat should visually turn something on or off for that entity. Unless this model changes, expect eventual extensions to the hacks stanza to support enabling state for anything other than the sender - for example, to turn on continuing VFX for spawned entities (when those become available) or to signal environmental changes (if that ever becomes a possibility).
-
The current apparent intent is to use <hack /> stanzas to signal to anyone who wants to interoperate with Paragon Chat that anything inside of it is a temporary stopgap at best, and the protocol is subject to change as soon as something better is designed.
Fix'd.
-
Fix'd.
LMAO
I have been busy with some other projects, and then Frontier decided that when my internet service broke that they didn't need to fix it for the better part of two weeks. I'm comfortable with all of the tools by now but I still need access to online reference material and using my phone for that was less fun than playing some of the games I bought from Steam and GoG over the last year and never bothered to play before my Internet went out. (Beyond Good and Evil is pretty good, if you're curious.)
I might get back to the bot project sometime this week.
-
Out of curiosity - is "oldcostume =" going to be the byteswapped version of the costume hash after a version of Paragon Chat comes out that doesn't byteswap it any more?
-
Yes. 0.98 and later prefer oldcostume if it's present, so that when the switch is made of the 'costume' attribute, they will continue to work so long as newer versions still send oldcostume.
Given that the travel capabilities in 0.99 are a strong incentive to upgrade to at least that version, I think we're on track for performing the switch to the new hashing algorithm (which I'll post details about shortly for review / comment) with the 1.0 release.
-
So its been another week, and I'm still plodding along. Sorry, nothing much to show yet, but I can say what I've been working on. I decided to see what it would take to write a generic animation engine for bots (which would have uses beyond bots). In other words, given a set of inputs, the engine would tell you what animation if any you should be playing/sending to Paragon Chat. For example, feed it a bunch of sequence bits and a time index, and then for any future time the engine will tell you based on the game sequence database what animation should be playing now. That kind of thing. I actually once had an almost fully functioning set of code to do that, but the last time it worked was in I20 and the code is also a mess. I mean literally a mess. Take a look at this:
def fixedpoint4byte(s):
fp1 = ord(s[3])
fp2 = ord(s[2])
fp3 = ord(s[1])
fp4 = ord(s[0])
fpsign = 1.0
if fp1 == 0 and fp2 == 0 and fp3 == 0 and fp4 == 0:
return 0
if (fp1 & 0x80) != 0:
fp1 = fp1 & 0x7f
fpsign = -1.0
## sys.stderr.write("Debug: negative sign triggered in fixedpoint routine\n")
fpE = fp1 * 2
if (fp2 & 0x80) != 0:
fpE = fpE + 1
fpE = fpE - 0x80
fpM = (fp2 & 0x7f) * 65536.0 + fp3 * 256.0 + fp4
fpS = 2 ** (22 - fpE)
# sys.stderr.write("Debug: fpE:"+str(fpE)+" fpM:"+str(fpM)+" fpS:"+str(fpS)+"\n")
# print "Debug: fpE:"+str(fpE)+" fpM:"+str(fpM)+" fpS:"+str(fpS),
fpval = ((2 ** (fpE + 1)) + fpM / fpS) * fpsign
return fpval
Seriously. That's the function I wrote, way, way, WAY back in I9 when I was first trying to reverse engineer pigg files. All I remember was trying to figure out how those numbers were encoded and trying out different theories until I found one that worked (trying to do this before you know what the numbers are supposed to represent yet makes it a bit more interesting). Of course, this is doing things the really hard way, treating the problem like code breaking. There are vastly simpler ways to do this, if you're willing to, say, trace the program itself (which didn't occur to me at the time).
This function was the one I was looking for, for a while:
def loadAnimTables(piggdict,subdir):
# subdir = "player_library/animations/male" for male players
global global_dver
piggfilename = piggdict['PiggFileName']
piggfilenameshort = piggfilename.split('/')[-1]
pf_debug = open('C:/CoH/decode/working/'+piggfilenameshort+'.sizes.'+global_dver+'.txt','wb')
anim_dict = {}
piggf = open(piggdict['PiggFileName'],'rb')
for pfile in piggdict.keys():
if pfile[0:30].lower() == subdir:
piggf.seek(piggdict[pfile]['foffset'])
fbz = piggf.read(piggdict[pfile]['fsizecomp'])
fraw = zlib.decompress(fbz)
fcount = fixedpoint4byte(fraw[520:524])
sys.stderr.write(pfile+","+str(fcount)+"\n")
anim_dict[pfile[26:].lower()] = {'frames':fcount}
pf_debug.write(pfile+": "+str(fcount)+"\n")
#sys.stderr.write(pfile[0:30].lower()+"\n")
piggf.close()
pf_debug.close()
return anim_dict
What does this do? This scans the pigg files looking for all of the *.anim files, which are animation files. This part: "fcount = fixedpoint4byte(fraw[520:524])" - that's some very hard-fought for knowledge. It took about a week for me to figure this out back in I11ish, but that's how long that animation takes to play from beginning to end, in frames. Why is that important? Because many animation sequence entries basically say "play until the end, then move to this sequence." That means the data in the sequence tables itself is insufficient to know how long to play sequence X before moving to sequence Y. But if you know how long the animation associated with sequence X is, you can time that accordingly**.
With that piece of information, plus a sequence decoder, you could write a set of functions that could load the sequence data and animation data from the CoH client piggs, then perform basic calculations on animation sequences. Paragon Chat itself must have the ability to do this to do what it does, more or less. Once I untangle all my old (really, really horrible) code from my pigg diving days, and make it more readable and less embarrassing, I should be able to replicate that functionality in a way that would be useful to bots. And also other things***. But its been slow going, first because my pigg decode code is organic, includes code for every version of pigg from I9 to I23 in a giant mish-mash, and I forget what the purpose of some of it was. And my free time is still fairly limited on what I can spend on it. But I'ma keep cranking away, and we'll see if I can't make something useful eventually.
** Actually, for things like emotes this is often not necessary because those kinds of animations usually contained explicit start and stop frame counters. Interestingly it tended to be powers-related animations that had stop=0 meaning play to the end. Which is why I needed that bit of information to calculate animation-based root times and compare them to powers-based cast times, which oddly no one before me ever seems to have thought to do.
*** Hypothetically speaking, it could be used to spit out custom emote.cfg files for people, based on what sorts of animations they wanted to have. With a little extra work, it could be used to spit out an emote.cfg that contained every animation connected to every power in the powers database (whether the animation actually worked or not). Stuff like that.
-
Well, engaging in a bit of thread necromancy here.
Life happened, I put my bot aside back in August, and never really got back to it. Judging by the state of this thread, that may have happened to some other folks also.
A few days ago I updated to PChat 1.0. I had a bit of time this morning, so I fired up the bot. After re-familiarizing myself with the right way to launch errbot, I fired it up and of course it was invisible.
However, a bit of fiddling to set oldcostume="TheByteswappedCostumeHash" and costume="" set matters right.
Now I just have to re-learn everything I knew about SleekXMPP and Paragon Chat. :-p
Anyway, the reason for the post - Are there any other technical details that a bot should be aware of, given the advent of PChat 1.0?
On another note - I was reading the thread where someone asked about how Null the Gull worked, and I was intrigued.
Maybe this would be crossing a fine line into behavior that constitutes "acting like a game server", but I'm wondering about the difficulty of modifying the NullClick() function to broadcast a message to the metadata room for a clickable object that identifies the object and requests a response to the click.
A bot configured to listen for such requests could respond with an ACK and an XML packet describing what the player should see as a response.
I have no clue about whether this is something that would be able to leverage Null's existing code or if it would require a new "dialog interpreter" module added to Paragon Chat. In the latter case, I'm sure that Codewalker has more than enough on his plate already.
-
I noticed that my bot is sending its version number as "0.98a". Are there any pluses or pitfalls to updating it to whatever the current protocol version number is? (Probably 1.0?)
-
The version number in the presence is only for troubleshooting and for display purposes (on the player info tab). It is not used by Paragon Chat to make any decisions about how it communicates with that player. That's what the "protocol" attribute is for, and it is incremented whenever non-backwards compatible changes are made to the XMPP protocol that other clients need to be aware of.
Probably the best thing would be to put something to uniquely identify your bot in there instead of a real paragon chat version number.
-
Small victories -
Firstly, having fallen two major versions behind on errbot, I updated my installation and it predictably broke the bot five ways from Sunday.
After a day of tracing down error messages and altering code to accomodate changes in naming conventions and functionality, I once again had my bot hanging around in Atlas Park. In the end it was not a picnic but it was easier than I feared.
Secondly, after another day of debugging both my code and the assumptions the code was originally based on, the bot is now able to load a costume on command.
I need to try loading a costume that is not in my client's cache to really test it, but for the moment, it looks like one more feature checked off of the list. Next up is tracking room occupants and their position relative to the bot.
-
So, I'm not surprised to learn that a NPC costume is an illegal costume for a bot, but I had my hopes.
Question - Is it well understood how the Halloween costume powers functioned? Is there a possibility of duplicating that function in Paragon Chat? I'm only suggesting this if it's more or less a switch inside of the client where the server sent a message that said "replace your current model with this NPC model".
On another note - I learned two things while experimenting with NPC costumes (from the collection that is linked elsewhere in the forum in the thread about hacking NPC costume bits into the sqlite character database).
One is that launching the bot with a NPC costume as its initial costume caused it to be invisible, but commanding it to change to a legal costume made it visible. Not a surprise, really. I was sort of hoping that if it was invisible that I could still see its chat bubbles in local and have it pretend to be a static NPC, heh. No dice there.
Two, and more interesting, is that starting with a legal costume and changing to an illegal costume caused no visual change for the bot. I would have expected the bot to become invisible with an illegal costume exception in the /whoall. Instead, it retained its previous costume, from a visual standpoint, even though it's presence was being advertised with an illegal NPC costume.
I'm not sure if this is a bug or a feature? (I wouldn't eliminate the possibility that I'm causing it somehow but my review of the bot logs looks like it's sending presence for illegal costumes identically to the way it sends presence for legal costumes.)
It means that if the bot is currently visible and I want to deliberately turn it invisible, that I should leave the zone entirely (leave atlaspark_meta and maybe also leave atlaspark channels) and rejoin with no costume or with an invisible one at any rate.
-
I suspect that your inability to turn invisible by announcing an illegal costume is due to the various things Codewalker did to try to explicitly prevent weird errors from making players invisible. My guess is that if you announce a legal costume and everyone in the zone acquires it, announcing an illegal costume causes all of the other players to simply ignore that illegal command. That means you effectively continue to look like what you last looked like to them. But if you start by zoning in with an illegal costume, no one has a previously legal cached costume for you, so you have to be invisible.
Unfortunately, I've had to shelve my own work on this project. I'm currently trying to figure out why FreeBSD's LSI driver (both of them) stall their writes to Intel 3510s under high random load. Work has me chasing a number of really complex systems problems at the moment.
-
So, I'm not surprised to learn that a NPC costume is an illegal costume for a bot, but I had my hopes.
It doesn't care about NPC pieces per se, but the same rules that apply for player costumes are in effect. A costume will only be accepted if the scales are in the valid range for players, and if any referenced FX are associated with costume pieces that are available to players.
The geometry and textures are not verified (intentionally, so that some NPC stuff can be used). The FX are, in order to prevent people from using things like power effects, screen space FX, and spamming oneshots in a context designed for maintained FX only (that can eventually lead to a client crash).
Question - Is it well understood how the Halloween costume powers functioned? Is there a possibility of duplicating that function in Paragon Chat? I'm only suggesting this if it's more or less a switch inside of the client where the server sent a message that said "replace your current model with this NPC model".
Changing models to an NPC is easy. The only reason there isn't a method in the protocol to do it currently is because some of the NPC models are a griefer's dream, and I've already had to boot a couple people for spamming the deactivate chrono whatever MOV in pocket D.
Protocol support for changing models is dependent on the permissions framework. Once that's in place, it can be limited to private instances and people who have GM rights. A whitelist of known-good models for use in public zones may be in the cards if there is enough volunteer support to do the legwork for it once the system is ready.
One is that launching the bot with a NPC costume as its initial costume caused it to be invisible, but commanding it to change to a legal costume made it visible. Not a surprise, really. I was sort of hoping that if it was invisible that I could still see its chat bubbles in local and have it pretend to be a static NPC, heh. No dice there.
Two, and more interesting, is that starting with a legal costume and changing to an illegal costume caused no visual change for the bot. I would have expected the bot to become invisible with an illegal costume exception in the /whoall. Instead, it retained its previous costume, from a visual standpoint, even though it's presence was being advertised with an illegal NPC costume.
I'm not sure if this is a bug or a feature?
Neither? Being "invisible" isn't a supported state that something can be in. It's the result of incomplete or inconsistent data that Paragon Chat is unable to reconcile, such as not having a costume at all. If the location data is otherwise good, Paragon Chat creates a placeholder entity to track it, but due to the lack of a valid costume, does not send it to the client yet. This half-formed entity is good enough for the local chat mechanism to work and is what is commonly referred to as "invisible". Only once the entity is fully complete is it sent to the client and becomes visible.
Once the entity is fully realized, trying to switch to an invalid costume won't undo that. It'll just reject the costume change and leave the costume that is currently applied.
So, using an invalid costume doesn't make you invisible. The best way to explain it would be to say using an invalid costume results in "undefined behavior".
What you might be able to get away with is using a valid costume that has no parts (or rather, empty parts). Or just send the bot to some far-off coordinates.
-
What about sending <presence type='unavailable'/>? Does that have an effect on Paragon Chat?
-
I'm currently trying to figure out why FreeBSD's LSI driver (both of them) stall their writes to Intel 3510s under high random load. Work has me chasing a number of really complex systems problems at the moment.
When I was in college far too many decades ago, I made an attempt (only partially successful) at writing a device driver so that a Unix v6 (Yeah, not even System 7) machine could read an IBM RL01 disk pack. The fact that I didn't know anything about writing device drivers was okay, since nobody else at the time really knew anything about it either.
Personally, I'm glad there's an Arcana to take care of these things in the modern world because I sure haven't got the chops for it today.
-
What about sending <presence type='unavailable'/>? Does that have an effect on Paragon Chat?
Sending it where?
If you send it to the meta channel... well, sending unavailable presence is how you leave a room. So that'll make the bot disappear, but also make it unable to see what is going on in that zone.
Sending it unqualified just disconnects you from the server. Mostly anyway, it doesn't close the stream but most servers will not forward you any stanzas while you're unavailable.
As for directed presence... I don't know, that's kind of a murky area of the XMPP spec. Paragon Chat itself won't do anything with it IIRC, other than maybe make the bot display offline in the global friends list.
-
What you might be able to get away with is using a valid costume that has no parts (or rather, empty parts). Or just send the bot to some far-off coordinates.
I was thinking the most logical way to be somewhere kinda, but be invisible, would be to change your location to a point directly under the map. You'd be "there" but impossible to see. You might also need to set yourself to fly so gravity doesn't yank you to the bottom of the map if you accidentally move downward.
-
Alright, that's about what I expected.
One last thing for today - another of those, "I'm not sure if this is strictly a bug or more a matter of the bot requiring sane bounds checking" things.
I was experimenting with some costumes that I made straight out of the Paragon Chat costume creator and none of them were loading. Not when I issued a command to the bot, and likewise not even when the costume was the bot's default costume.
After examining the offending costumes, I found they all had something like this in their opening stanzas:
{
CostumeFilePrefix male
Scale -3.934e-006
BoneScale -1.192e-007
ShoulderScale -1.192e-007
ChestScale -1.192e-007
WaistScale -1.192e-007
HipScale -1.192e-007
LegScale -1.192e-008
HeadScales 0, 0, 0
BrowScales 0, 0, 0
CheekScales 0, 0, 0
ChinScales 0, 0, 0
CraniumScales 0, 0, 0
JawScales 0, 0, 0
NoseScales 0, 0, 0
SkinColor 255, 178, 155
NumParts 28
The scales turned out to be the source of the problem. The bot didn't do any checking on the values in the scales, it just accepted them on faith. Technically, that makes it a bug in the bot. It should be treating 0.0000000197 as zero rather than blindly accepting it as meaning something.
Essentially, these values are what happens when you skip past the scale sliders without changing them. Adjusting the scales changes the values enough to make them meaningful.
I *think* (but am not positive) that this is new behavior since v0.98a of Paragon Chat. I could be wrong - it might be that I just never made a costume without nudging the sliders a bit.
In any case - bots need to be aware of the possibility that the scales need to be rounded off to some reasonable number of decimal points rather than accepted verbatim.
-
I was thinking the most logical way to be somewhere kinda, but be invisible, would be to change your location to a point directly under the map. You'd be "there" but impossible to see. You might also need to set yourself to fly so gravity doesn't yank you to the bottom of the map if you accidentally move downward.
I'll have to try that. I have a feeling that you'd see the bot actually sink in to the ground as it moved from A to B, at least based on my other experiments with what happens when I tell the bot to "go to loc X, Y, Z".
.
-
The scales turned out to be the source of the problem. The bot didn't do any checking on the values in the scales, it just accepted them on faith. Technically, that makes it a bug in the bot. It should be treating 0.0000000197 as zero rather than blindly accepting it as meaning something.
That shouldn't matter. Did you look at the XMPP output to verify that they didn't get turned into something crazy in the XML by the floating point formatter?
I *think* (but am not positive) that this is new behavior since v0.98a of Paragon Chat. I could be wrong - it might be that I just never made a costume without nudging the sliders a bit.
I don't remember exactly when the scale verification went in, it might have been after that. But values near 0 should be perfectly fine, since the baseline is all 0 and valid ranges extend in both the positive and negative direction.
I'll have to try that. I have a feeling that you'd see the bot actually sink in to the ground as it moved from A to B, at least based on my other experiments with what happens when I tell the bot to "go to loc X, Y, Z".
I don't think I remembered to implement it yet, but I have on my todo list a movement flag for instantaneous, for teleport as well as entering doors.
-
That shouldn't matter. Did you look at the XMPP output to verify that they didn't get turned into something crazy in the XML by the floating point formatter?
According to my logs, the XML and hash string were just copying the slider values verbatim. That is, the literal string "-1.92e-007". There didn't seem to be any funky formatting happening. Is there a way to tell Paragon Chat to save the contents of the debug window? Next time I do some testing, I'll set a close watch on what Paragon Chat is logging itself as receiving, but I don't anticipate a problem in that area.
I did try setting the scales to "-0.000000192". That had no effect, though I realized later that the scales in a pristine costume file never have a leading zero and that might have invalidated the test.
I also tried loading up the costume in the CoH costume editor and playing with different combinations of slider values. Basically, if any of the sliders reads "-1.192e-007", meaning that it was untouched and set to the default value, then the costume is undisplayable by the bot.
Are those sliders allowed to be zero? Is "-1.192e-007" is just a floating point artifact like 2 + 2 = 3.9999999999, and the actual value is supposed to be zero? If I need to, I can have the bot convert the text string to a float value and then drop anything past six decimal places.
-
On a different note, I thought I'd toss out a use case for true invisibility via a switch:
Imagine a roleplay session where the GM wants the players to interact with Ms. Liberty. Instead of describing the conversation, the GM can turn his assistant-bot invisible, move it behind Ms. Liberty on the map, change its nick to "Ms. Liberty", and then tell it what to say in local chat so that it appears as if Ms. Liberty is doing the talking.
On a more amusing note, consider this - A wiki-bot stands in Atlas Park, that listens for questions about City of Heroes and responds with wiki entries related to whatever triggered it. On any response, the bot has a 1% chance to "overload" and explode! It plays appropriate dialog and animation, then turns invisible. It then assumes the form of a whistling workman who "arrives" on the scene to effect "repairs" and install a new wiki-bot on the spot. The workman walks off, turns invisible, becomes the wiki-bot again, and then resumes visibility in its normal listening spot.
Granted, that the second case would work just as well using Arcana's suggestion of moving below the ground, as long as the movement was instantaneous. Likewise, there's not really any intrinsic reason to avoid leaving the room to effect "invisibility", though doing that could incur some otherwise unnecessary overhead if your bot was tracking the occupants of the room and had to re-load a bunch of data that it already had before it became "invisible".
-
According to my logs, the XML and hash string were just copying the slider values verbatim. That is, the literal string "-1.92e-007". There didn't seem to be any funky formatting happening. Is there a way to tell Paragon Chat to save the contents of the debug window? Next time I do some testing, I'll set a close watch on what Paragon Chat is logging itself as receiving, but I don't anticipate a problem in that area.
Oh, that's right, Paragon Chat does truncate the slider values to 4 decimal places when hashing/sending the costume, specifically to prevent floating point jitter from causing the hash to not match. It also rounds them to the nearest thousandth when saving/loading costumes, though that's just a safeguard against 0.5 turning into 0.4999 and isn't strictly required.
I need to finish implementing the v2 costume protocol. With that I'm dropping the global costume cache, so it will no longer be required that the hash be generated a specific way. So long as the bot can generate unique identifiers for its costumes in a consistent manner, it won't matter if it isn't doing it the same way Paragon Chat itself does.
-
Groovy. I'll tell the bot to do the same sort of truncation.
Allowing the hash to be any unique identifier (as simple as MyBot-some_unique_string) would make things much easier on future bot developers, for sure. :-) Not to mention that some_unique_string could be an identifier that's meaningful to bot itself.
-
Out of curiosity - Given that Openfire (and by extension, I assume most other XMPP servers) can handle thousands of concurrent connections, why are the zones of Paragon Chat set to only a few tens of users before spawning a new instance?
-
There are a couple of things that make Paragon Chat difference.
1. XMPP servers handle thousands of concurrent connections, but these are users who occasionally communicate with each other and idle most of the time. They're not in one big chat room together.
2. Even in big MUC rooms, on a regular XMPP server, typically only a few people are talking at a time. The meta channel traffic from Paragon Chat is considerably more frequent than a regular conversation, and depending on how active people are, everyone in the room could be "talking" at once.
3. The above characteristics mean that meta channels are N^2 amplifiers. If 4 people in a zone are sending 1 update per second, then the server is processing 16 messages per second. If 30 people are in the zone, it's processing 9000 messages per second...
4. The exact performance and bandwidth usage was not known when we launched, so I opted to be conservative and set the zone limits rather low. At this point, we can probably safely up it to 50, which would put it on par with the live game.
It's worth noting that Atlas Park and Pocket D are both set to unlimited currently.
-
What are the odds that the permissions framework you mentioned earlier includes a way for a client to authorize acceptance of a "travel cookie"?
I assume it's obvious that I'm thinking here about a mission door or a temporary train menu destination.
A "travel cookie" wouldn't operate on a map ID #, necessarily. It would supply a map name, a XMPP room name, a map file, and a spawn location. Effectively, a private instance of a map, though it could just as easily point to an existing "global" zone instead.
If I'm handing out a mission, in an ideal world I want to tell the player which room to join and which map she'll be joining, not depend on she and I to agree in advance on room names and map ID numbers, or that the player even knows that map X exists.
-
Getting back to NPC's - in determining why my costumes weren't working, I learned that there are more "legal" body types than just the male/female/huge. BodyType 2 and 3 seem to be generic NPC's, while BodyType 5 is unmistakably a WarWolf.
Experimentation showed that trying various numbers other than 0-5 either resulted in no visible difference from male/female or they crashed the game client.
Rather than brute force I figured I'd just ask about other known usable body types. It's of particular interest to me because I want to make a rikti monkey bot. If the WarWolf has a unique BodyType complete with its own "ready" animation, then do any other NPC's have the same?
-
What are the odds that the permissions framework you mentioned earlier includes a way for a client to authorize acceptance of a "travel cookie"?
What you're describing wouldn't be part of the permissions framework. It would be part of the private instance support - which is itself part of the larger work to decouple the XMPP public zone backend from the client communication frontend. The permissions framework is a prerequisite for private instance support.
-
What you're describing wouldn't be part of the permissions framework. It would be part of the private instance support - which is itself part of the larger work to decouple the XMPP public zone backend from the client communication frontend. The permissions framework is a prerequisite for private instance support.
That makes sense. I was envisioning the "travel cookie" more as a case of a player setting a permission that read, "I will accept a travel beacon from the following people: [X] Anyone [ ] The following list of JID's... [ ] Nobody".
Well, I'll file any similar questions as "wait for private instances".
-
It'll probably be more of an invite based system - the invite includes the info about meta channel name (probably a long hex string) and map file. Although for people who want public areas we'll have to figure out some way to advertise the relevant info to access the instance.
The permissions framework is more along the lines of someone trying to do something like forcibly move a player, change to a restricted costume, spawn an NPC (though that also depends on the pubsub implementation so that people zoning in afterwards can see it), etc. For those privileged actions it has a list of possible ways it can be authorized, such as:
- Being listed as an administrator or GM in the shardwide config
- Being inside of a private instance and also owner of that instance
- Being inside of a private instance and having select abilities delegated by the owner
- etc.
It's a bit more involved than that, since the permissions framework isn't just about what you can do, it's about giving other players the ability to validate incoming data from untrusted sources. If you receive an XMPP message about something happening that involves a privileged operation, the permissions framework gives the recipient the ability to verify that the sender of the message is authorized to do that, and ignore the message if they're not.
-
It's a bit more involved than that, since the permissions framework isn't just about what you can do, it's about giving other players the ability to validate incoming data from untrusted sources. If you receive an XMPP message about something happening that involves a privileged operation, the permissions framework gives the recipient the ability to verify that the sender of the message is authorized to do that, and ignore the message if they're not.
Once such a framework is in place - I can't help but notice that XEP-323 and XEP-325 seem to be just the sort of thing that a Gamemaster-bot might use to touch buttons in the client to do things like move an XP meter, create a dialog window and set some contents in it, activate a "BADGE AWARDED!" banner, etc...
-
Direct access to the client like that isn't something I was planning to expose over XMPP. The vision is that XMPP is used not as a control protocol, but as a synchronization protocol.
Paragon Chat is not directly translating the COH mapserver protocol to XMPP. That would be impossible, there's far too much disparity between the two. PC has no central server. Everybody is running their own server, and those servers synchronize globally visible state over a shared communication bus (XMPP). XP and dialog boxes make no sense to synchronize, since those things are specific to each player. Nobody cares about your exact XP total but you.
Things like you're talking about are a much better fit for the plugin / lua scripting architecture. With that, a plugin can hook directly into the local server state and update it, such as by changing the amount of XP that the player has, or using functions to show contact dialogs. Whether those plugins are local-only or if they coordinate with other players who have the same plugin installed is up to the author.
Since a hypothetical plugin would have access to the XMPP stream, I suppose if you really wanted to, you could write one to accept XEP-325 messages and take action based on them.
-
I can see what you're saying, but I was imagining it more like X-Windows, where the bot (or the XMPP server itself with some sort of game management plug-in) is the "client" and Paragon Chat is the "terminal server". It wouldn't know anything about game state. It would only know how to display the graphical representations of game state as instructed by the GM-bot. That would be in keeping with the "keep Paragon Chat from being a game in and of itself" premise of it.
A smart, LUA-enabled client would be capable of a lot more stand-alone functionality, which is also exciting and interesting. ;-)
-
The CoH client has a built-in web browser.
Is it possible to present a http:// link in chat or in a character's Info window that invokes the web browser?
-
An update and a question.
I'm currently reorganizing and re-writing the core of my bot, for a few reasons.
I want to move all of the SleekXMPP code into a specialized Paragon Chat backend that can be divorced from the "front end" so that someone receiving the bot as a package doesn't have to worry about the details of low level communication protocols.
I want to move most of the bits and pieces into a single consolidated Avatar() class that holds all of the character presence data as well as the methods to update and publish that data.
I want to leverage the plug-in capabilities if Errbot to modularize function so that, for instance, a bot that stands in place doesn't have to keep movement code loaded and taking up resources.
One thing the bot does is load the zone.cfg into memory to know when a a room is a "zone" with a "_meta" room. What zone.cfg does not provide is a hint for default coordinates. I've looked at a few maps from geobin.pigg. All of them seem to have player spawn points listed as coords 0,0,0, even though the CoH game client seems to know the real coords of the label in the zone.cfg.
Is there a way to find the x, y, z coords of the playerspawn points short of brute force visiting every map and recording the spawn in /loc?
-
Well, it wouldn't be a week in Paragon City if I didn't have some new questions and desires.
Today's topic is chat text color.
Let me state up front that I am assuming I'm going to be disappointed but nothing ventured, etc...
The short version - is it possible to alter the colors of the chat dialog text or are the colors hard-coded to correspond only to the channel colors listed in the channel selection list? When the game was live, the channels expected that Pet Dialog, Battle Damage, and Help messages, for instance, would each be different colors and you'd identify the source of the text in your current chat tab by its color. In Paragon Chat, the text in the current "channel" is colored the same as the channel title.
On a related note, is it possible to color individual bits of a chat balloon or is it hard-coded to be one foreground color and one background color for the entire chat balloon?
The attached picture shows what I want to do:
(https://dl.dropboxusercontent.com/u/18795363/CoH/botchat.jpg)
PCBot is sending this message body: "<color blue>Hello, %s</color> <color green>How's things?</color>"
As you can see, the color has no effect on the "Broadcast" channel text. PCBot's chat balloon defaults the whole sentence to the last color tag it received.
In an ideal world, what I want to do is arbitrarily color individual words, like so: "Hello, %s. There's been a lot of <color red>gang activity</color> recently and we could really use your <color red>help</color>." (The idea being that someone would say, "What gang activity?" and the bot would respond with more info about activities it wants the person to participate in.)
This actually works in non-chat related dialogs, as the Description in the upper-left illustrates.
Is there any way to make this work in chat as well, or does the client programming make this a no-go?
What if you marked up your chat text as if it was from one of the built-in channels, like "Pet Damage Received"? Would the game client give it the appropriate colors, and would it apply them to just the marked up text or would it apply to an entire message?
-
I don't know and I'm not actively developing in Paragon Chat at the moment, but I do know that the colors in the chat box are due to formating hash tags: they actually show up in the combat/chat logs. You might try experimenting with sending those and just see what happens.
What the game can do and what we were allowed to do are different things, and Paragon Chat may sometimes bypass limitations that were imposed upon us. I'm thinking specifically about the chat-pocalypse that happened during one of the betas when someone discovered you could trick the client into embedding graphics within chat dialog. The devs decided to remove that feature, but it was only there in the first place because the chat rendering could do some interesting tag processing in the first place.
Experimentation might be the only way to authoritatively know what can and cannot happen in this area. Or I suppose Codewalker can figure it out as well.
-
The short version - is it possible to alter the colors of the chat dialog text or are the colors hard-coded to correspond only to the channel colors listed in the channel selection list? When the game was live, the channels expected that Pet Dialog, Battle Damage, and Help messages, for instance, would each be different colors and you'd identify the source of the text in your current chat tab by its color. In Paragon Chat, the text in the current "channel" is colored the same as the channel title.
The colors come from the channels, not the chat tags. That's handled by the client - the 'server' piece just sends the channel ID, it doesn't set the colors. Even the customizable channel colors are controlled by the client; the server just acts as dumb storage for the chat settings.
Internally the text drawing may use the tags to do it, but that's an implementation detail. Probably the client strips off any formatting tags from the incoming text and encloses it in one to set the correct color for the channel that the text arrived over.
On a related note, is it possible to color individual bits of a chat balloon or is it hard-coded to be one foreground color and one background color for the entire chat balloon?
Pretty sure the balloon text can't switch formatting mid-stream. You can't even have different font sizes in a balloon together; it changes the whole thing.
What if you marked up your chat text as if it was from one of the built-in channels, like "Pet Damage Received"? Would the game client give it the appropriate colors, and would it apply them to just the marked up text or would it apply to an entire message?
That isn't controlled by markup at all. For it to show up in that channel, it has to actually be sent over that channel.
-
That was poor language choice on my part. By "marked up as a channel" I meant embedding a <channel> substanza with an ID of "pet_damage_received" or whatever the actual label would be.
-
devs decided to remove that feature, but it was only there in the first place because the chat rendering could do some interesting tag processing in the first place.
Well, if I'm being honest I'd already figured that out by examining the various *message bin files and I was hoping that I could get it to display at least a badge icon or other graohic. From what you've said, that may be unavailable (though it may work in the description dialog).
-
Channels are not working the way I would expect them to work.
Here's a sample of a message being sent to Paragon Chat, as copied from the Paragon Chat debug log:
xmpp DEBUG RECV: <message type="groupchat" to="slickriptide@cjj-pc/Slickriptide"
lang="en" from="atlaspark@conference.cjj-pc/ErrBot"><body><color blue>Hel
lo, Slickriptide</color> <color green>How's things?</color></b
ody><channel id="request" xmlns="pc:message"/></message>
I would expect that PCBot's responses would come out colored as Request chat.
That isn't what happens.
There are four channels that can communicate with another player: Local, Broadcast, Request and Global.
Example 1: No channel stanza.
Local - Bot reply formatted as a /tell (due to the way local is implemented, the bot sees a private message and replies in kind)
Broadcast - Bot muc reply colored and labeled as "Broadcast"
Request - Bot muc reply colored and labeled as "Broadcast"
Global - Bot reply received as a /tell, and client prompt labeled and colored in the Global tab as Local.
So, anomaly #1 - Global is behaving as an alias for Local. Is this a bug? It remains true for all of the following examples, so I'll omit Global from here on out.
Anomaly #2 = Request defaults to Broadcast as its default formatting.
Anomaly #3 - the fact that the channel the client sends from even has an influence on how it interprets the channel a message is received on.
Example 2:
A channel stanza is sent but no ID is set.
Local - The bot reply is labeled and colored as Local. The existence of a <Channel> stanza is enough to cause the private tell to be treated as "local chat".
Broadcast - The bot reply is labeled and colored as Broadcast
Request - The bot reply is labeled and colored as Broadcast
Example 3:
A "legal" channel is set as the channel ID, where "legal" means one of Local, Broadcast, Request, or Global
Local - The bot reply is labeled and colored as Local, regardless of the ID
Broadcast - Broadcast and Request are displayed as specified, Local is displayed as Broadcast
Request - Broadcast and Request are displayed as specified, Local is displayed as Broadcast
Example 4:
An "extras" channel is set as the channel ID; for instance, Friends, Arena, or League
Local - The bot reply is labeled and colored as Local
Broadcast - The bot reply is labled and colored as Broadcast
Request - The bot reply is labeled and colored as Broadcast
The patterns are consistent for other values of channel ID.
Given how a lot of this comes down to how the CoH client handles the information that Paragon Chat is passing to it, I don't know if the above is strictly a bug. It's just not the straight across "set a channel ID stanza at one end and get that channel display at the other end" correlation that I expected.
-
Here's a sample of a message being sent to Paragon Chat, as copied from the Paragon Chat debug log:
[-snip-]
I would expect that PCBot's responses would come out colored as Request chat.
That message should be coming in over request, unless there's some subtle typo that I'm not seeing. Since you're also trying to manipulate the color, do you have broadcast and request split out to different chat tabs so you can tell which channel it's actually arriving over instead of looking at the color?
There are four channels that can communicate with another player: Local, Broadcast, Request and Global.
What is "global"? There is no "global" channel. Do you mean global channels? Each one is a separate channel that is associated with the JID of the room for it.
I think you're misinterpreting the purpose of the channel stanza. It isn't there to say which in-game channel the message is supposed to be sent over. It's there to distinguish between different sub-channels that are multiplexed over the same XMPP communication method.
There is not a 1-to-1 mapping between COH chat channels and methods of communicating in XMPP. The semantics of various channels are different and don't always map to similar XMPP capabilities. Some channels share the same underlying XMPP transport method with others that have the same semantics. Request and broadcast are a good example because they are both zone-wide and prefer character names over globals. So they share the zone's broadcast room instead of having two separate rooms. Incoming messages from that room default to broadcast and the presence of the channel stanza signals that it's request instead.
Ultimately the transport that the message arrived over (type=chat/groupchat, which room it came from, etc) determines where it is routed.
The way it should work:
Groupchat messages arriving over the room associated with a global channel -> That channel
Groupchat messages arriving over the zone's broadcast room that have a channel tag with an id of 'request' -> Request
Groupchat messages arriving over the zone's broadcast room -> Broadcast
Chat messages containing a channel tag with an id of 'emote' -> Emote
Chat messages containing a channel tag -> Local (Paragon Chat sends 'local' for this, but the presence of the channel stanza triggers routing of the message into the local chat handler. That handler only checks for emote and defaults everything else to local. This behavior may change so it's best to not depend on it and always use 'local')
All other chat messages -> Private tell
-
What is "global"? There is no "global" channel. Do you mean global channels? Each one is a separate channel that is associated with the JID of the room for it.
In the CoH chat window, there is a row of buttons for selecting the channel you want to chat in. L = Local, B = Broadcast, T = Team, etc... The last one is labeled 'A' and when you click it, it changes the chat channel to 'Global', which does not start with 'A' and which behaves like 'Local' in Paragon Chat.
I think you're misinterpreting the purpose of the channel stanza. It isn't there to say which in-game channel the message is supposed to be sent over. It's there to distinguish between different sub-channels that are multiplexed over the same XMPP communication method.
And here is the heart of the matter. You're absolutely correct. I was treating <channel> as "use this client channel", especially since Broadcast and Request appeared to be working just like that.
Okay, so that pretty much invalidates most of what I posted and once again my desire to send some sort of hilighted text is stymied, heh.
On the other hand, if Emote is its own channel then maybe I can use that for "out of character" text. I'll give it a try and see how it looks.
So, the only real reason for a bot to care about channels is make sure that Local chat and maybe emotes get handled properly.
So, what's the point of the Request channel? Does it have an intended use other than being broadcast in purple?
-
In the CoH chat window, there is a row of buttons for selecting the channel you want to chat in. L = Local, B = Broadcast, T = Team, etc... The last one is labeled 'A' and when you click it, it changes the chat channel to 'Global', which does not start with 'A' and which behaves like 'Local' in Paragon Chat.
Oh, 'A' is Active, which means to use the default channel of whatever chat tab has focus (it should be a different color).
I think the name of one of the default tabs created by the client is Global. That's probably where it's coming from. That tab must have local set as its default channel. The default channel can be set in the edit chat tab window.
-
So, what's the point of the Request channel? Does it have an intended use other than being broadcast in purple?
*shrug* No idea, really. Sometimes it would get used by Hamidon / mothership raid leaders because of it being a different color.
Since it's functionally identical to broadcast, implementing it in paragon chat took no time at all, so I figured why not.
The emote channel is also identical to local, with only cosmetic differences - but the messages are formatted in italics and the chat bubbles show up as thought bubbles.
-
So, what's the point of the Request channel? Does it have an intended use other than being broadcast in purple?
I think it was originally intended to be a kind of trade request channel. Hey, anybody wanna trade a Mutant Acc SO for a Tech Dmg SO? That kind of thing. Needless to say it was almost never used for that purpose, or as far as I'm aware, any purpose.
-
After a bit more experimentation, I've determined that Arcana is, sadly, correct that the devs must have disabled format tags entirely in chat text. OTOH, formatting in other dialog windows seems to work just fine. See the Description window below:
(https://dl.dropboxusercontent.com/u/18795363/CoH/channel%20bug/sample2.jpg)
The chat text at the bottom contains the same tags as the Description window.
So, chat pictures and formatted chat text are out, but if Paragon Chat someday acquires a LUA engine that can launch dialog windows, those windows will render formatting tags.
-
***edit***
Moved this post to a topic of its own.
-
Does Paragon Chat currently do anything with the <show> element?
Getting back to this question of hiding a bot. Let's say I want to emulate the old CoH thankyou behavior. I want the bot to run to a player, say "Thank you", run a few feet away and fade away.
Accomplishing that fadeout by leaving the metadata MUC puts you in for a whole load of overhead when you rejoin the room; especially if you're tracking all of the users in the room.
If Paragon Chat treated <show>XA</<show> as "Don't render this avatar" then the NPC could fade out without having to leave a metadata room that it intends to immediately rejoin.
A bot could move from point A to point B instantly without Paragon Chat rendering the motion, by turning invisible before relocating.
I guess that however it's accomplished, it would be useful to have a bot capable of becoming invisible without leaving or creating a deliberate costume error status in the process.
-
The only things it does with that element are to make a player appear AFK, and also update their status in the global friends list.
-
Putting aside my normal semi-unrealistic wish list -- A person who uses Paragon Chat solely as a chat client will occasionally want to "lurk" and hear what's going on even though she isn't actively participating. That's why text-only clients have options like "away", "do not disturb", and "extended away" instead of just forcing the person to disconnect.
A person in "lurk mode" may prefer that his avatar be hidden to avoid the appearance of inviting conversation, or of ignoring someone who doesn't realize they are AFK.
It's a valid answer to say, "Then they should just use Pidgin for that" but you ordinarily don't want to tell someone that they have to use two different clients to do one job.
Likewise, putting the responsibility of typing "/afk" on the user is a valid answer. I'd just suggest that sometimes a better answer is to remove the illusion of presence entirely. At least, the option would give a user choice in how to make themselves be "afk".
Something for the feature suggestion list.
-
Personally, I'd rather CW -not- waste time implementing a feature anyone can do just by logging in with a regular xmpp client, and use it to bring more functionality to PC.
-
Putting aside my normal semi-unrealistic wish list -- A person who uses Paragon Chat solely as a chat client will occasionally want to "lurk" and hear what's going on even though she isn't actively participating. That's why text-only clients have options like "away", "do not disturb", and "extended away" instead of just forcing the person to disconnect.
A person in "lurk mode" may prefer that his avatar be hidden to avoid the appearance of inviting conversation, or of ignoring someone who doesn't realize they are AFK.
It's a valid answer to say, "Then they should just use Pidgin for that" but you ordinarily don't want to tell someone that they have to use two different clients to do one job.
Likewise, putting the responsibility of typing "/afk" on the user is a valid answer. I'd just suggest that sometimes a better answer is to remove the illusion of presence entirely. At least, the option would give a user choice in how to make themselves be "afk".
Something for the feature suggestion list.
You may need to wait until Codewalker implements stealth and perception attributes, whenever that might happen, or whenever he implements teleport which requires implementing an alternate method of becoming temporarily invisible the client supports.
Separately, there are ethical concerns regarding lurking in a graphical chat environment. The reasoning is that in normal XMPP chat there's really usually just the notion of broadcast chat (across the entire chat room) and private chat. But Paragon Chat by virtue of supporting other kinds of chat like emotes has a theoretical way of "localizing" chat to only nearby participants. There's no analog to "nearby" in text-based communications. Is it ethical to allow someone to become invisible and yet listen to group chat intended only for avatars within a certain distance (and presumptively visible) from the participants?
-
A person in "lurk mode" may prefer that his avatar be hidden to avoid the appearance of inviting conversation, or of ignoring someone who doesn't realize they are AFK.
The easiest way to do that is to hide in some far-off corner, or use a custom zone config to park in a uniquely-named zone.
Players going invisible while still interacting normally isn't going to happen; and even the invisible costume issue may get changed to use a default so that the player name is at least visible. The local chat system is built on the expectation that if someone is close enough to hear your chat, you should be able to see them to know someone is listening.
The closest you can come is to send someone the "I am ignoring you" IQ. That serves as a hint that you have removed someone from your game universe and are not interested in interacting with them in any way, so they might as well remove you from theirs as well. Once you do that, you will also no longer receive their local chat, since it basically asks them to put you on (non-persistent) ignore.
-
Players going invisible while still interacting normally isn't going to happen;
Roger that. If I want a bot to "fade out" then it'll be the hard way, then.
For the record, I'm pretty much always aware of the fact that I'm asking for things that outside the bounds of Paragon Chat's primary function as a graphical chat room. My interests are in pushing it beyond the limits of the vision and seeing just how far it can be bent that way. That is not a criticism of anything to do with Paragon Chat or the vision driving it. I just want to see what we can do with it if it happens that this is the most that is ever done in the way of a CoH "environment".
Yes, I'm aware that other things are happening, including the hinted-at scripting interface. I'm also aware that projects end without notice all the time due to life events, changing interests, and people simply getting tired of working together. While I hope that there will be much more coming in the next year or two, my own vision is based on what's in front of me at the moment.
Here's an example of the sorts of things I'm talking about:
The first time I put a velocity on a bot and discovered that it "jumped", I immediately considered the idea of making a "Duck Hunt" game. One bot would listen for the command "Pull!" and launch a "duck" in a semi-randomly determined arc. Another bot would listen for command "Punt!" and "punt" a Red Hat into the air in whatever direction the player avatar was facing. Since actual targeting is not available at the moment, the command would probably be more like "Punt X" where X is a force between 1 and 10 or something. In any event, if the "duck" and the "projectile" intersect then a third bot scores a point for the player. The scorekeeper would keep track of individual scores as well as a leaderboard of the ten highest scores or something similar.
These three bots might all be one bot, or specialized individuals.
I'm a ways off from doing anything of the sort - I am still finding plenty of things I need to change about my current bot to make it handle player tracking properly, and I'm just now reaching that point where I can say I really understand XMPP, SleekXMPP, ErrBot, and CoH fundamentals and how each plays with the others. The last thing I need right now is to get sidetracked into learning cannon physics.
But maybe that gives you an idea why I'm constantly asking for things that appear to have nothing to do with chatting with people, heh.
Diary time:
Having got as far as I am, I'd recommend that anyone considering this pursuit do two things. One is choose a coding environment that you are familiar with. The other is start with the fundamentals of XMPP, master that, and then move up the "stack" to your XMPP API/Library and then to Paragon Chat and the internals of the CoH client.
I started down this road knowing zero about any of it, and assuming that I could take a "top down" approach to learning just whatever bits and pieces I'd need to get a chatbot up and running and then let the chatbot do the fundamentals while I did the stuff I actually cared about doing.
Arcana was both right and wrong about the follies of taking this approach. She was right, from the standpoint that despite my hopes and desires, there was simply no way to actually accomplish interactivity, especially if it was done in anything like a correct manner, without first learning the fundamentals of how communication in XMPP happens at all.
That meant that saddling myself with a chatbot framework from the outset was just adding yet another thing to the stack of things that I didn't know anything about and would have to learn thoroughly before I could use it effectively. I ended up handicapping myself to a certain extent, especially since I was forcing myself to struggle with data structures in the chatbot framework that were never intended do what I wanted them to do.
My unfamiliarity with Python only aggravated the problem. I didn't know how inheritance worked in Python, and even simple tasks required not only education in Python methods and modules but also in how SleekXMPP implemented certain tasks and then how Errbot implemented other things on top of those layers.
Has it paid off in the end? Yes, to some extent. Example - When I was working out how to correctly handle channels (and then working out that my understanding of channels was somewhat flawed) I was able to rewrite part of the bot and command Errbot to reload the affected module on the fly and test it immediately. When I wanted to write a "voice command" for the bot to respond to, it was a simple task using Errbot's existing interfaces.
On the other hand, it also cost me some pain. With no real knowledge of Python, I mistakenly approached it as if it was C or C++ and treated everything about it as if it was "sacrosanct", meaning that data structures were immutable and functions very strongly typed. I ended up spinning my wheels rather a long time trying to accomplish things that turned out to require no special accomplishment at all, because I was following examples written by other people who treated Python as if it was strongly typed and assuming that meant that it must BE so. It took some input from the Errbot people to point out how I was making things hard on myself to realize that I was, in fact, making things much harder than they needed to be.
As a beginner in all of this, I'd have been better off if I'd started out writing in C and using the Libstrophe library or something like it to learn to communicate in XMPP. Once I had that down, then moving on to interacting with Paragon Chat and expanding to other things once that was mastered.
I'm far from a "master" of all of this but I've reached the point where I can say I understand each of the pieces and how they interface, and where to place the new interfaces between Errbot and Paragon Chat so that the back end of Errbot can deal with the metadata and the front end can query and manipulate the metadata without having to understand where it came from.
It'll be worth it in the end but I made it a lot harder row to hoe than it needed to be, at least from the perspective of just learning to communicate with Paragon Chat in the first place.
If you're thinking of making a bot, then start with the fundamentals and work up from there. Like most other activities in life, short cuts aren't all they appear to be unless you're already so well versed in the subject that you didn't really need the short cut in the first place.
-
For the record, I'm pretty much always aware of the fact that I'm asking for things that outside the bounds of Paragon Chat's primary function as a graphical chat room. My interests are in pushing it beyond the limits of the vision and seeing just how far it can be bent that way. That is not a criticism of anything to do with Paragon Chat or the vision driving it. I just want to see what we can do with it if it happens that this is the most that is ever done in the way of a CoH "environment".
I get that, and it's fine. I'm not trying to jump on you for experimenting or coming up with crazy ideas -- just being honest about what's possible and what the goals/limitations of the project are.
Ideally I want to encourage creative uses of the platform, but as you said time is limited so some things need to be prioritized.
-
At the risk of being a thread necromancer, I'm resurrecting this thread because the answers to questions can be found here.
A year ago, roughly, I lost my hard drive and my bot code along with it. Rather than start over, I threw up my hands and went and played Hearthstone.
A couple of months ago, I ran across a backup of the costume code from the old bot, and was inspired to have a go at it again.
The break was good, as it turned out. This time I was in a learning frame of mind rather than a shortcutting frame of mind.
Anyway, my bot learned how to walk around this week so I thought I'd share a video of the bot doing some tricks, to celebrate.
https://youtu.be/xOhfslGTZ08 (https://youtu.be/xOhfslGTZ08)
-
So, velocity vectors. I'm having some difficulties wrapping my head around how to figure out what they should be.
My initial implementation of bot avatar movement was to emulate Arcana's approach and send a position every tick. That worked okay - The bot moved where it was supposed to at approximately the speed I wanted it to move. It was a little herky-jerky but I expected it to need some fine tuning. One thing I noticed was that Arcana wasn't computing any velocity for her "come here" command. Out of curiosity, I had Rover stop sending velocity and just send position and orientation. He started running smooth as silk. It didn't matter that his velocity between ticks was zero because the updates were coming in so fast that he was effectively moving at desired velocity just from the constant position updates.
So, I was getting the velocity wrong; hardly a surprise. Unfortunately, when I took a look at the debug logs I could see that Rover was spamming the Heck out of the metadata channel. It was good that he was running around on a private server because if there was more than a couple of people in the zone with him he'd be DDOS'ing the server!
No matter how accurate it might be, it was an unacceptable amount of traffic to be hitting the server with(or would be if there were multiple Rovers in the zone).
Then I took a good look at Paragon Chat and affirmed that it only sends position and velocity updates when something changes. Otherwise, client prediction handles things.
Which brings me back to velocity vectors.
I've got the math down for moving from point A to point B. Rover can respond to "come here" or "face me" or "follow me", but somehow when he reaches his goal, he always either over shoots or under shoots and gets rubber banded to the point he was trying to reach.
That is, he does this:
Compute one tick of movement.
If the any of the new velocity values are different from the current velocity values, update current and send a u stanza
If computed PYR is different from current than update current and send a u stanza.
If distance is less than 5 then set velocity to zero, update current and send a u stanza.
So, it sends the bare minimum number of messages, which is great, but it always overshoots or undershoots when it reaches its goal position.
The 64-million-dollar question, then - When I set my avatar to "walk" and I turn 45 degrees and press the "forward" key, how does the client and/or Paragon Chat figure out that the velocity vector is "-.01 0 -.12" and why does that add up to a different vector length than, say, a vector like "0, 0, .15"?
What are the velocity values for "walk", "run", and "sprint" and how does the client apply them to calculate the position in time from any given unit vector?
-
At the risk of being a thread necromancer, I'm resurrecting this thread because the answers to questions can be found here.
A year ago, roughly, I lost my hard drive and my bot code along with it. Rather than start over, I threw up my hands and went and played Hearthstone.
A couple of months ago, I ran across a backup of the costume code from the old bot, and was inspired to have a go at it again.
The break was good, as it turned out. This time I was in a learning frame of mind rather than a shortcutting frame of mind.
Anyway, my bot learned how to walk around this week so I thought I'd share a video of the bot doing some tricks, to celebrate.
https://youtu.be/xOhfslGTZ08 (https://youtu.be/xOhfslGTZ08)
Nice, Slick! Are you aiming to get mobs into the game with this, is that the purpose?
-
Nice, Slick! Are you aiming to get mobs into the game with this, is that the purpose?
That's the end-goal, yes. OTOH, Codewalker's "ideal world" goals for Paragon Chat include eventually building a scripting interface that would do a better job of pretending to be mobs in the game than a bot that is essentially a player pretending to be a mob in the game, so for now Rover is mostly experimental.
I'll decide what to do with him next after I've solved the "move somewhere without flooding the server with move commands" problem.
-
As a consequence of logging my movement computations, I confirmed not only that my math was correct, but also that my clock is slow.
Whether it's the overhead of Errbot or the overhead the movement computations, the clock is running at 21 ticks per second instead of the 30 ticks it's specced for. The game client is happily running along at an actual thirty ticks per second and so it's sending the bot further along than the bot believes itself to be.
So, in the short term I need to scale things between the standard and the actual clock and in the long term I need to have the scaling to be adjustable and have some sort of profiling that can adjust it for the performance of the machine running the bot.
-
That's the end-goal, yes. OTOH, Codewalker's "ideal world" goals for Paragon Chat include eventually building a scripting interface that would do a better job of pretending to be mobs in the game than a bot that is essentially a player pretending to be a mob in the game, so for now Rover is mostly experimental.
I'll decide what to do with him next after I've solved the "move somewhere without flooding the server with move commands" problem.
Cool, what happens if you toss him a frisbee while he's moving?
-
Nothing right now. That is, Paragon Chat might show Rover catching it but Rover wouldn't know it. Maybe I'll see if I can trick or treat up some candy canes to buy a frisbee power and figure out how to throw it.
-
So, here's an unexpected interaction/challenge that, in retrospect, ought to have been totally expected.
Once I got Rover's velocity scaled properly for his actual clock, he started arriving at his destination spot on the money. The next issue to deal with was a "bug" that caused him to frequently sit in one spot for a period of time, before taking off and following the current position of his target.
It was while I was watching the live debug logs playing on both Rover and Paragon Chat that the light bulb went off.
Rover didn't have a bug. He was doing exactly what he was programmed to do.
The issue is Paragon Chat's economy of traffic.
If I ran my character backwards in a straight line, the client didn't send any motion stanzas beyond the initial stanza that tells nearby people that Slickriptide is at position x,y,z with velocity a,b,c. As long as I maintained a constant velocity, Rover was blind to my current position. As far as he was concerned, I was still standing at x,y,z. Meanwhile, from my standpoint and from the standpoint of anyone watching via their own Paragon Chat client, my avatar was sliding away from x,y,z at whatever my run velocity was set to.
Once I changed velocity (by slowing or by curving in a new direction) then my client sent a position update and Rover realized, "Hey! He's really way over there!" and started running towards that new position.
That also explained why Rover seemed to always be chasing my back trail instead of chasing my current movements. His idea of "now" is based on the last position update he received from his target. That could be reasonably accurate if I was making a lot of turns and shifts or terribly inaccurate if I was moving in a straight line at a constant velocity (and therefore sending few or zero position updates).
That means that if Rover is going to follow someone that he needs to change from a "follow" behavior to a "pursuit" behavior, where he tries to anticipate where his target will be instead of going to the last place his target actually was. The biggest challenge with such a behavior being that he can't know how accurate he is until the target finally relents and sends a position update.
Basically, Rover is a victim of General Uncertainty, heh.
-
Well, you probably don't want to anticipate too much. If you're too accurate, the bot will try to occupy the same space as the player and trigger collision + pushing. Ideally you want to be directly behind the player.
What it doesn't sound like you're currently doing: the bot needs to know how to figure out where the player is right now, rather than where the player was when the last update was sent. In order to do this, you'll need to retain one more piece of information - the timestamp of the last velocity data that was received.
From there it's as simple as scaling the velocity vector by time elapsed and adding it to the last received position:
posCur = posLast + velLast * dT
Where dT (? Time) is the time elapsed in ticks. There are 30 ticks per second, so if your time interval is measured in seconds, just multiply by 30.
Paragon Chat does this exact calculation in order to determine if it needs to send an update or not. It compares the position that everyone else will predict (based on the last sent position and velocity) with the player's actual position. If it's within a reasonable margin of error, it knows it doesn't need to send an update. That's why it doesn't send anything at all when you're running in a straight line.
There are a couple things that will help. The first is that when PC sends a velocity change, it always sends a position update in the same stanza. That means you don't need to track the last received value/time for position and velocity separately.
The second is that while PC has a minimum threshold for when things have changed "enough" to warrant sending an update -- and may delay sending for a fraction of a second to try and batch up multiple changes -- the player coming to a complete stop (velocity becomes 0 after previously having a value) always triggers an immediate send.
-
The base run speed is 0.7 world units/tick (or 0.525 when moving backwards).
The following multipliers are applied by travel powers:
Sprint: 2.0
Super Speed: 6.46
Walk: 0.216
Ninja Run & Beast Run: 2.4
For walk, 0.7 * 0.216 = 0.1512, which is pretty consistent with the vector length that you're seeing.
There are a couple things that may cause the velocity vectors seen on the wire to not always match that exactly.
- When you press a movement key, velocity doesn't immediately jump to the maximum. There is a ramp-up for the first second or so that you hold the key down before you hit the max.
- The velocities above are the input velocity that get fed into the physics engine (which is a
poor ripoff close-enough reproduction of the client's physics engine to not cause rubber-banding). Other factors like surface friction or becoming airborne from elevation changes can change the final velocity. It's the output from the physics that is ultimately sent over xmpp.
Are you sure you were seeing -0.01 0 0.12 for a 45 degree angle? It should be closer to -0.11 0 0.11.
-
Are you sure you were seeing -0.01 0 0.12 for a 45 degree angle? It should be closer to -0.11 0 0.11.
Oh, no, sorry, that was pulled out of memory and not from an actual log. I'm sure it probably was more like -0.11, 0, 0.12 (since the 45 degrees was eyeballing).
Thanks for all the movement info. That will help a lot.
I'm not too concerned with being deadly accurate. I chose a 5-world-unit stopping zone, like Arcana did to avoid the kinds of collisions you mentioned. I've also been reading some interesting articles about steering forces steering forces (https://gamedevelopment.tutsplus.com/series/understanding-steering-behaviors--gamedev-12732) that I'm going to try to incorporate into Rover to make his following/escaping behaviors more interesting.
Of course, now Rover doesn't walk through walls any more, heh. He gets hung up on terrain until he decides it's time to update his own position. I'm going to have to revisit the whole conversation about beacons and paths before all is said and done.
-
Eventually some sort of bot API on the paragon chat client itself rather than speaking XMPP directly will be the way to go, for two reasons:
- At some point in the medium term I'm going to be reworking how movement works. Meta channel position broadcasts will be reduced to only happen every few minutes - or after moving a large distance - just enough for everyone to get a very rough idea where other players are.
A compressed binary stream containing timecoded movement data will be sent to a private pubsub node. Clients are expected to subscribe to the pubsub node of other players that they care about getting fine-grained movement/animation data for (i.e. are in visual range).
Not only does playing back timecoded data result in much smoother movement than the velocity prediction system, it avoids everyone spamming the meta channel with their position 1-2 times per second. No reason to waste bandwidth getting movement updates for someone on the other side of the zone you can't see anyway.
For at least a few versions the old protocol will still be supported as a fallback if clients older than a certain protocol version are in the zone.
That's not impossible for a bot to learn to speak, but a lot more work. Using an API would free the bot author from having to deal with it.
- It would be a lot easier to write bots that move around convincingly if collision info was available. An API would allow bots to get a feel for their surroundings, using either complex methods like pulling geometry, or simple ones like a call to cast a ray and see where it collides with the world.
That's still some distance off, so for the moment what you're doing may not be the best way - it's just the only way. :)
Oh, no, sorry, that was pulled out of memory and not from an actual log. I'm sure it probably was more like -0.11, 0, 0.12 (since the 45 degrees was eyeballing).
That sounds about right. -0.11 0 0.12 is a vector with length of 0.162 (a2 = x2 + y2 + z2), which is within the expected error range since the XMPP values are rounded to hundredths.
Since you said you got the velocity otherwise working I'm assuming you've got the math for moving around in a plane down (cos(a), 0, sin(a)) and just needed the right velocity for different types of movement.
-
I'm happy to see this project is still alive. It seems my brain can only hold one dev project at a time without exploding these days, and I've been continuously distracted by work stuff which actually pays money so I can make a big pile of it and retire.
At some point I would like to return to this stuff, but it might not be until after Codewalker releases an API for it. But keep on keeping on: I will be following along, vicariously part of the robot revolution.
-
Since you said you got the velocity otherwise working I'm assuming you've got the math for moving around in a plane down (cos(a), 0, sin(a)) and just needed the right velocity for different types of movement.
So far, so good. We'll see what happens the day Rover straps on a jingle jet. :-p
-
Rover is following nicely now, though he's still a little jumpy at high speeds and occasionally he forgets to turn off his velocity when he comes to a stop. Not sure what that last is about, yet. The scaling between Rover's clock and the real clock is probably part of the issue. I might have to look into what's causing the Errbot clock to run slow and see if I can correct it for my needs. Errbot already has thread management in place and I'd rather leverage that than have to roll my own to handle timer interrupts.
Next step is to take all of my ad-hoc movement code and turn it into methods on my avatar class. That will make it easier to assign movement instructions and follow/flee targets to individual avatars, and it will put movement processing where it really belongs - on the avatar that's doing the moving. When that's done, I'll add in methods to implement steering behaviors. The end game here is being able to assign Rover a seek target AND a flee target, so that he moves towards one thing while avoiding another thing (presumably a player avatar) as part of a single movement computation.
Looking towards the future - Navigation is the big problem to be solved.
Do we have schemas for the city maps, like city_01_01.bin? Never mind - that's geobin.xml, yes?
I'm conflicted about having to roll my own virtual file system to read the map and costume files. On the one hand, there are open source pigg viewers that can provide examples of what to do, but ultimately the question is whether it matters? There'll never be an issue 25. Short of modding the game client, the data in the pigg files is never going to change, so why not just extract the data I care about and put it in a SQL database where I can deal with it on my own terms?
Courtesy of bindump, I find the following info in city_01_01.bin:
294 groups named 'Omni/_NAV'. Each of them has beacon data embedded that parses out of bindump like so:
Beacon =
-------- Beacon 0 --------
u080 = BasicBeacon
u081 = 35.6909
I have no idea what that u081 number represents. Codewalker? Arcana? Any ideas? Distance to another beacon, maybe, or some bit-coded data? None of them has position data listed for the beacon; these are just flags that say that the beacon exists (and whatever u081 represents).
Defs 415 seems to be the list of paths and beacons. It's a list of nearly 1400 "groups" that looks like this:
-------- Defs 415 --------
Name = grp3953
Group =
-------- Group 0 --------
Name = Omni/NAV
Pos =
X = -909.5
Y = -0.447678
Z = 1235.75
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
-------- Group 1 --------
Name = grp105167
Pos =
X = 644
Y = 41.4434
Z = -746.75
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
-------- Group 2 --------
Name = Omni/NAV
Pos =
X = 643.5
Y = 41.4434
Z = -773.75
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
-------- Group 3 --------
Name = Omni/NAV
Pos =
X = 633.5
Y = 41.5527
Z = -803.75
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
"grp105167" is a basic beacon. "Omni/NAV" is, presumably, a point on a path from one beacon to another. How the paths run between beacons isn't immediately obvious. Individual basic beacons are sometimes referenced multiple times at different absolute positions, so it seems that those can't actually be "absolute" positions for any given beacon.
Any hints on how to make sense of the paths is appreciated. Did I read someplace that there's a dev command in Icon that will make the paths visible?
-
Beacon =
-------- Beacon 0 --------
u080 = BasicBeacon
u081 = 35.6909
I have no idea what that u081 number represents.
That means it's the 81st field in the file and that I didn't know what it was. When I was building the library that powers bindump, I numbered the fields u001, u002, u003, etc., the 'u' meaning "Unknown". Then as I figured out what each field was, the name was changed. Only the fields that didn't have a text name I could extract from the exe and didn't have an obvious purpose retained the 'unknown' designation.
Now I know that u080 and u081 are Name and Radius, respectively. Radius for a beacon depends on the context of what the beacon is used for, but generally just indicates the proximity in which the beacon is relevant.
Individual basic beacons are sometimes referenced multiple times at different absolute positions, so it seems that those can't actually be "absolute" positions for any given beacon.
There are no absolute coordinates in groupdefs, only relative to the parent group. The only place you'll find absolute coordinates in a map file is in the Refs section at the very end where groups are instanced into the scene graph.
How the paths run between beacons isn't immediately obvious.
They don't. Beacons are just location markers.
Any hints on how to make sense of the paths is appreciated.
If you're looking for an AI path network, you won't find one. NPC movement was all serverside, and whatever pathfinding info that it used does not exist in the client files.
Most beacons are for things like doors (the rationale for this is a little complicated, but door properties are assigned to beacons that were placed near the door rather than the door itself), navigation points for the minimap, special markers for things like a minimap swap when going indoors, script markers, etc.
Basically beacons are things that the devs manually placed on a map. AI pathfinding was most likely done through automatic analysis of the geometry, perhaps using some of the manual beacons as hints in tricky places.
The exception there is that a few maps have waypoint beacons for patrol routes where enemy groups followed a fixed path. There are also traffic beacons for cars (though most of these are part of the library pieces so you won't see them in bindump, but you can see them in Icon). Both of those are close enough together that they could probably be joined into a path network. However they would only work in certain places that those beacons exist, and wouldn't be generally useful for a bot to find its way around.
-
I guess that saves me worrying about parsing the map file. If I want a bot(s) to move around a map I'll have to map it out myself ahead of time and create the paths for them to move along. NPC's on rails, it is!
-
So, today I've been exploring Icon with /map_dev and /mission_control turned on.
Neuromancer. Woah!
(https://images.weserv.nl/?url=www.cjhunter.com%2F%7Escott%2Fscreens%2Fneuromancer.jpg)
I'm revising my opinion about parsing the map files. There may not be a specific AI pathing mechanism in the maps but there are definitely some paths built into the maps and there are lots of markers that can be abused as path markers, especially in the main city zones.
For instance:
(https://images.weserv.nl/?url=www.cjhunter.com%2F%7Escott%2Fscreens%2Fscreen1.jpg)
The big arrows in the street are directing traffic. The green upside-down tetrahedrons are marking an obstacle-free path that a running NPC can follow. The thin green arrows that are hard to see in that pic are marking a path for pedestrians to follow and the purple thing with the number on marks where the pedestrians change direction.
(https://images.weserv.nl/?url=www.cjhunter.com%2F%7Escott%2Fscreens%2Fscreen2.jpg)
Here you can see the path up the stairs laid out in steps that prevent a NPC from accidentally sinking into the stairs.
(https://images.weserv.nl/?url=www.cjhunter.com%2F%7Escott%2Fscreens%2Fscreen3.jpg)
Here the tetrahedron's light the way around the sidewalk that runs around City Hall.
Even in places where there aren't any paths, per se, there are LOTS of encounter spawn points and those spawn points can be abused repurposed into being "beacons" that can lead a bot past most nearby obstacles.
So, it seems that if I want to have some kind of general way of navigating any map in the game, that I'm going to have to learn to read map files, after all. I'm also going to have to come up with some quick way to search for "beacons" that are not too close, and not too far, and to make that "best beacon to aim for" computation in a reasonably short amount of time.
That's not too much to ask Rover to do, right? :o
-
I'm revising my opinion about parsing the map files. There may not be a specific AI pathing mechanism in the maps but there are definitely some paths built into the maps and there are lots of markers that can be abused as path markers, especially in the main city zones.
Just to give you a heads-up about what you're in for, to get everything that you see in Icon you'll have to parse a lot more than just the map files.
Map files in COH are made up of groups, and references to those groups, forming a tree (graph). But some groups in maps refer to things that aren't in the map file. They refer to groups defined in what's called the "object library", which is a repository of prebuilt objects. Things like buildings, or even chunks of city blocks that map designers could place.
Some beacons are defined in the map itself, but others are part of library pieces. Probably the best example of this is those big green arrows on the roads. Most of those do not actually exist in the map file. Instead, there is a library piece that's a chunk of road AND the beacons. So when designers built the roads by placing those chunks, the beacons are there by default.
The spawn clusters are another example. Those are typically only 1 point in the map (and do not look like a beacon), but it's actually a reference to the entire group of all the spawn points.
That means if you want a FULL scene graph, you have to chase references all the way into the object library. At a minimum, that means loading defnames.bin, building an index, then loading the appropriate library file (fortunately these are the same geobin format as the maps) when a ref to one of the library pieces mentioned there is found in a map.
-
Hey, Arcana! Any chance you've got a function laying around that returns a list of every pre-defined map coordinate in a given map file that's within X world-units of the "current position"? No? Well, it never hurts to ask, heh.
So, thinking out loud, and basing on the "beacon" situation (where I'm using "beacon" to mean any map entry at all that can be used as a reference point for a bot, rather than meaning the things inside the map file that are explicitly labeled as beacons) in a city map (Atlas Park) and a hybrid city/outdoor map (Croatoa) here's what I'm thinking I need to do:
Dump the scene graph into a SQL database of reference points. These would be the computed absolute positions of each point.
Label each reference point as a "beacon" or an "obstacle". For now, I'm not sutzing over the collision box of the obstacle but if the label of the obstacle is helpful enough to include a measurement in its name then I might save that data as well. Probably I'll want to label each reference point with a type that indicates the originally intended purpose of the reference point, to be used to "weight" actual Omni/NAV reference points higher than others.
Is it useful to know the facing (yaw) of obstacles? Maybe. Walls, certainly; not so much other kinds of obstacles. Hmmm... I think I might need to know the size of the collision boxes after all. How hard is it to pull that information out of the .geo files?
A tick of movement conceptually would go like this:
Compute a "movement unit vector" that is the unit vector of destination - current_position.
Compute a "tick vector" that is Ux,Uy,Uz + Vx,Vy,Vz
Select a list of all NAV points that lie along the "tick vector".
For each NAV point, compute a seek steering force that moves toward that NAV point. The result is a "tick vector" that lies between all nearby NAV points.
If there are no nearby NAV points (that is, map objects named "Omni/NAV") run the check again for non-NAV "beacons" - spawn points, mainly, but possibly other kinds of ref points as well.
Select a list of all obstacles that lie along the current "tick vector". These are primarily objects like bushes, trees, rocks, walls, fences, and buildings.
For each obstacle along the "tick vector", compute an avoidance steering force. This is where collision boxes might be necessary, to modify the magnitude of the steering force in order to clear the obstacle. The result should be a "tick vector" that follows an "optimal" path between all nearby reference points.
In an ideal world, this means that NAV points (or any point being utilized as a "movement beacon") are "pulling" Rover towards themselves while obstacles are "pushing" Rover away from themselves.
This still doesn't entirely solve the problem of sloping terrain but the ref points in the map files are mostly at ground level already and Rover can include some filters that drop ref points that are clearly set at a higher delta-Y than, say, the height of a small hill or a flight of stairs. I think we can pretty much ignore the idea of Rover ever navigating a nightmare like The Hollows.
-
Hey, Arcana! Any chance you've got a function laying around that returns a list of every pre-defined map coordinate in a given map file that's within X world-units of the "current position"? No?
No.
It is fair to say that I got very far in parsing files related to powers. I got pretty far in parsing files related to animations. I got a headache trying to parse geometry, and it wasn't helpful for the parts of the game I was actively contributing my expertise to.
Geometry is not intrinsically hard. In fact, I can recognize what is going on in broad strokes, and I'm familiar enough with the principles of 3D computer geometry to not be totally lost. But it is a Matryoshka doll of things within things within things that would be extremely time consuming to untangle.
So, thinking out loud, and basing on the "beacon" situation (where I'm using "beacon" to mean any map entry at all that can be used as a reference point for a bot, rather than meaning the things inside the map file that are explicitly labeled as beacons) in a city map (Atlas Park) and a hybrid city/outdoor map (Croatoa) here's what I'm thinking I need to do:
I see two solutions to your problem, you can parse geometry, or you can cheat.
Parsing geometry involves creating a space-map from the geometry and object files. Straight forward in principle, extremely tedious in detail.
Cheating involves predetermining where you want your bot to be able to travel, and just walking the terrain in Icon or something and demorecord your pathing. Then feed that data to the bot and program the bot to interpolate your own coordinates rather than attempt to figure out where the ground and obstacles are. You would be in effect tracing the ground with your feet and recording where the ground is using a demorecord.
If you're going with option one, I would suggest ignoring all of the detail and just starting with the ground. You're a ghost bot, and will walk through walls and assorted shrubbery. But just getting the ground right is a huge step forward.
Plus, I don't think City of Heroes itself consistently got this right either. Some of the moving background objects did do crazy things like walk up nonexistent stairs or pass through mailboxes. Collision detection is expensive and I don't think the game itself did it perfectly all the time for every object. Sometimes it just did what it thought it was supposed to do, which wasn't quite right in the actual game world.
-
For city zones, the list of Omni/NAV beacons is a pre-existing "cheat sheet" where the bread crumbs track the paths around all of the sidewalks and the absolute position of each beacon is easily computed. I've hand-computed a few and double-checked the results in Icon.
I suppose for starters what I should do is just load up Atlas Park's nav beacons and instruct Rover to execute a loop where he perpetually runs from one beacon to another and see where he goes. Connect the dots, in essence. That would be pretty straightforward.
If I want to layout a pre-determined path it's even easier than using Icon - I tell Rover to heel, then I walk the path in Paragon Chat and at every "checkpoint" give Rover a command to record his current position. Voila!
That sounds like a plan for "next steps".
- Create a database of sidewalk beacons for Atlas Park
- Teach Rover to "move forward to the next nearest beacon"
- Define a set of "path beacons" that define a complete hard-coded path.
- Teach Rover to create, edit, and delete paths.
That will be good enough movement functionality to move on to creating a simple scripting engine for Rover.
-
If you want just a very rough idea of the terrain and are willing to build the whole scene graph (including the object library), one thing you could do is load the .bounds files instead of trying to parse geometry.
Those are in Parse6 (bin) format as well and contain the bounding boxes for library objects.
The min and max corners of the bounding box are u001 and u004 in bindump, respectively. Other interesting fields are u011: a (calculated) spherical radius, which might be useful if you're using an avoidance algorithm rather than collision, and u015. u015 is the flags bitfield, the interesting ones there are bit 25 (0x2000000), which indicates an object that is invisible except in developer mode and should be ignored, and bit 23 (0x800000) which indicates an entire object set to not collide.
As every map is made out of pieces from the library, you could build a rough collision model from the bounding boxes. For slanted surfaces it would work if the object was rotated by the map designer (you'd just rotate the box), but it wouldn't be able to handle it if the slant is built into the geometry itself. i.e. things like the tech ramps, or the block around city hall, where the slant is part of a larger object.
For that you'd have to open the can of worms of loading the geos (iirc there are 6 different 'versions' of the geo format that need to be supported), and then loading the tricks to get texture flags and figure out which surfaces are collidable, and so on...
-
Oh, bounds boxes in a readable bin format could makes things doable!
Going back to the whole "there'll never be an Issue 25" thing, the "rendering" of the scene graph only has to be done once and stored in a database. It's not something that Rover has to know how to do. All he needs to know is how to analyze the portions of the current map's scene graph that are nearby at any given moment.
-
Wellllll... given the Pocket D AE wing expansion and the (seasonal) mysterious floating island in Croatoa, I'm not sure that's the best assumption to make.
-
Wellllll... given the Pocket D AE wing expansion and the (seasonal) mysterious floating island in Croatoa, I'm not sure that's the best assumption to make.
Heh. Well, given that pre-rendering will work (assuming it works) for 99.9% of the maps in the game, I'm willing for now to deal with the risk that a map might have to be re-rendered once in a while.
I want to keep Rover's overhead as light-weight as possible, given that he's running in an interpreted environment and he already has a slow clock. (Though, I'm not certain he SHOULD have a slow clock - I might have to investigate whether that's correctable. Arcanabot was running in Python and happily doing 30fps.)
Then again, it's possible that I'm exaggerating the extent of the memory footprint of Rover carrying all of the path nodes around in his head. Hmm... I suppose I should at least experiment with that approach rather than dismiss it outright.
-
Perhaps it should be based on how intensive (long) the process is for processing the maps.
If it's only a few seconds, that could plausibly be done once on bot load (and cached for some short duration).
If it's a few minutes, that probably should be done once and exported to a file for the bot to use.
Maybe make a util that updates the file after execution.
But either way, the code for processing all that has to be made first.
The best use might be a bit hard to discover until after the fact.
Just the thoughts off the top of my head. All this stuff really interests me.
-
Perhaps it should be based on how intensive (long) the process is for processing the maps.
It will take some experimenting to determine the best approach. I may have to table a lot of the testing until after the holidays.
-
I prefer to avoid re-inventing the wheel when a perfectly good wheel already exists, so I'm just going to ask - Is there an algorithm that illustrates how to apply the schemas to the game data files?
For instance - the geobin.xml schema shows the data structures, without acknowledging the file header and its various "magic" values or any mechanism for determining the length of the dynamic arrays in the structures. It sort of looks like you process a def by loading a four-byte value for an <entry/> and if the value is non-zero then you treat it as the beginning of an array element and you proceed to load array elements until you load a zero-filled element; or at least one where the first four bytes are zero-filled. It kind of looks like there are some filler bytes in there, so the length of the array may be encoded and I'm just not seeing it yet.
So, I figure I can spend a couple of days futzing around with Kaitai or I can just ask the expert.
And, if this ends up boiling down to "maybe you should just parse the output of bindump because it understands ALL of the old formats and not just parse6" then the next question will be, "Can bindump have an option added to output xml?"
-
After some time spent with the geobin files and geobin.xml, I've concluded the following:
There's a magic header.
The data is stored "little endian".
An array of structs begins with a four-byte index showing the number of records. If the index is zero, the array is empty. Each individual struct in the array begins with a four-byte size followed by the data for that struct.
Strings, aside from the magic values in the file header, are stored as a two-byte length parameter, the string value, AND a null-char terminator (even though you don't strictly need the null terminator given that you know the length already). This might mean an empty string would be three bytes: 2 for the length of zero, and another for the requisite terminator.
INT is a four-byte integer. U8 and F32 indicate the number of bits respectively. Enum values are U16, though there's no reason they couldn't be signed.
Unless I made an error in the above, that looks like enough info to read a geobin file (and probably any other file that Codewalker has provided a schema for). I'm going to feed the schema plus the various extra bits noted above into a Kaitai schema and see if it spits out a parser that produces the same info as bindump. If it does, then I've got my data extractor and the potential for the bot itself to read a map file if that's a useful thing to do.
***EDIT***
There might be some question as to how DEGREES is represented. I'd assumed it was F32 but if it's a series of bit-coded quaternions or something arcane like that then it might actually be something that needs to be examined closer. (That almost sounds like I actually understand what quaternions are, heh.)
***EDIT MORE***
I'm not sure what's going on with the Omni's. Sample dump:
A0 00 00 00 0D 00 67 72 70 6C 69 74 65 31 34 32
31 32 39 00 01 00 00 00 2C 00 00 00 0E 00 4F 6D
6E 69 2F 5F 4C 69 67 68 74 44 69 72 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 01 00 00 00 14 00 00 00 3A 00 00 00
3A 00 00 00 3A 00 00 00 6C E0 EE 41 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The omni in question is from grplite142129 in City_01_01.bin.
I read this as:
"01 00 00 00" is one omni record.
"14 00 00 00" is a record size of 20 bytes.
"3A 00 00 00" is fixed array element 0 aka u060
"3A 00 00 00" is fixed array element 1 aka u061 (I can see this might not have been the best example to use)
"3A 00 00 00" is fixed array element 2 aka u062
"6C E0 EE 41" is float value aka u063
"00 00 00 00" following all that is presumably the flag enum, aka Flags
I've got some issues here.
First, is that geobin.xml specs those array elements as U8 when they appear to actually be stored as U32.
Second, that "fixed" arrays don't seem to have a size element. Since they're "fixed", it seems to be presumed that anyone reading it already knows its size.
Finally, even though Flags enumerations appear to be specced as U16, it must also be stored as U32 because otherwise the record size doesn't pan out.
I guess I need to take a close look at the raw file to see if all of the U8-specced values in the schema are actually stored as full-size U32 values in the data file.
-
I prefer to avoid re-inventing the wheel when a perfectly good wheel already exists, so I'm just going to ask - Is there an algorithm that illustrates how to apply the schemas to the game data files?
I don't think there's one. From time to time I've considered tossing bindump up on github or something, but tbh it's a huge pain in the rear to build. Have to find the right boost version for starters, and the python-based code generator for the schema readers needs some extra dependencies to run. It needs a little work just to compile on something like Linux and a LOT of work to compile on something like Windows.
There's a magic header.
The data is stored "little endian".
An array of structs begins with a four-byte index showing the number of records. If the index is zero, the array is empty. Each individual struct in the array begins with a four-byte size followed by the data for that struct.
Strings, aside from the magic values in the file header, are stored as a two-byte length parameter, the string value, AND a null-char terminator (even though you don't strictly need the null terminator given that you know the length already). This might mean an empty string would be three bytes: 2 for the length of zero, and another for the requisite terminator.
Sounds about right.
There might be some question as to how DEGREES is represented. I'd assumed it was F32 but if it's a series of bit-coded quaternions or something arcane like that then it might actually be something that needs to be examined closer. (That almost sounds like I actually understand what quaternions are, heh.)
It's just an F32, but it's stored in radians. So multiply by 180?.
Quaternions are used in the network protocol over the wire, but not in any data files. Well, they're in .anim files, but nothing in Parse6 bin format.
First, is that geobin.xml specs those array elements as U8 when they appear to actually be stored as U32.
All integers in bin files are stored as 32-bit integers. The U8 and INT16 describe the *in-memory* structure and also give a hint as to the maximum range, but when serialized to bin files are always 32 bits.
Everything in a bin file should be 4-byte aligned. Even strings need to be padded if they don't end on a 4-byte boundary.
Second, that "fixed" arrays don't seem to have a size element. Since they're "fixed", it seems to be presumed that anyone reading it already knows its size.
Yes, fixed arrays have a set number of elements. It should be in the XML file, right?
-
Bit of context here, the bin format is not the "true" format, but is the one we have easy access to. The xml files I posted are extracted and converted from binary tables in the client exe -- tables that are used by what's called the textparser.
The game is natively designed to load 95% of its data from a bunch of plain text files (hence textparser) on a developer's machine. This is a structured format that can vary a bit depending on the particular schema, but generally has a consistent set of rules. A good example is costume files. These are stored in the text format and the textparser is used to load and save them. Take a look at the costume.xml schema file while viewing a saved costume in a text editor.
Loading thousands of text files isn't particularly conducive to a published game (and is rather slow), so whenever a text file is loaded by textparser, it can optionally be persisted in a binary format -- the latest and last version of that being Parse6. The internal layout of these binary files didn't even stay the same from version to version; they're just temporary storage generated by textparser as a byproduct and were not the "main" data format -- but ARE what got published and the form that we have as a result.
So the binary tables the schema XML was built from actually describe 3 things simultaneously, in order of descending importance:
- The format of the master text files for all of the game data.
- The in-memory layout of structures that those text files get loaded into. This is especially handy for me because it makes hot-patching stuff in memory easier if this is known.
- The format of the binary files that the text data gets persisted into.
As a result of this, the textparser is also used to load bin files directly when the text files aren't available (or are not any newer than the cached bins). So it may sound like a misnomer when I talk about the textparser tables being used for loading bin files, but it actually makes sense in the full context.
-
The "magic" header:
8 bytes: "CrypticS"
4 bytes: CRC32 of the textparser table [1]
2 byte string length + string: bin file version (i.e. "Parse6")
Then the filelist[2] header:
2 byte string length + string: "Files1"
4 bytes filelist size
4 bytes number of filelist entries
Each entry:
2 byte string length + string: filename
4 bytes: 32-bit unix timestamp - file modify time
----- Actual data starts here -----
[1] This is the CRC32 of the binary table used by textparser to describe the data structure. It's used to determine when the table has changed from one version to the next and a bin file needs to be regenerated. However, if the text file isn't present, it will still attempt to load a bin with an incorrect value here - which may result in a crash if the format isn't really what the code is expecting.
[2] The filelist is just a list of text files that this particular bin file was built from. This isn't needed at all for loading data from bins and can be skipped. If both text and bin versions of the data are available, textparser will use this to determine if the bin file is up to date, or if it needs to be regenerated from the text files because some of them have changed.
-
??? (Tahquitz is now dizzy and goes back to his PHP work...)
-
??? (Tahquitz is now dizzy and goes back to his PHP work...)
Heh. Trust me, I feel the same way. This project has frequently put me in the position of the competent-but-not-particularly-gifted carpenter who says, "I need a house to live in, but there's nobody here to build it for me. How hard can it be?" without really considering the ramifications of wiring, plumbing, and building codes. Thankfully, people like Codewalker and Arcana are around. If I see further, it's because I've stood on the shoulders of giants and all that.
Everything in a bin file should be 4-byte aligned. Even strings need to be padded if they don't end on a 4-byte boundary.
Thanks for that confirmation. I'd begun to suspect that was the case after I ran into a whole series of property entries that seemed to have unexpected null bytes in between the strings and the following data. After seeing some entries where a number immediately followed a string without any null bytes at all, I'm thinking now that the "unnecessary" null terminator I mentioned on the strings is actually a case of byte padding and not a "terminator" at all.
Likewise - Thanks for the file header format. I'm especially happy to learn that the file list is basically noise. I saw one costume bin, I think it was, that had a big file list in the header and I wasn't at all sure what to make of it in comparison to something like the Atlas Park map bin.
That all makes me curious, though - Does the textparser still look for those original text files when the client launches? If someone was able to recreate a developer working directory, could you load your own versions of those text files?
I've mentioned Kaitai a couple of times. It's a parser-builder that sounds like maybe it's similar to whatever Boost is. I'm taking the geobin schema plus whatever extra details I've learned from examining the files and the info you've provided to write a schema that Kaitai uses to spitout a parser in any of a dozen or so different programming languages/environments. Analyzing game assets seems to be one of the bigger uses of it. The website is https://kaitai.io (https://kaitai.io) if you're curious.
One thing I'm NOT doing is writing a virtual file system. I just dumped everything out of the pigg files into a corresponding directory structure on disk. Most of the data that interests me from the bin files is stuff I intend to put into a SQL database anyway, so I don't see any mileage in trying to act like a pigg viewer or the ACTUAL game client on top of everything else.
-
div {
color: #FFFFDD;
background-color: white;
font-weight: 900;
}
You're welcome.
-
Why is it that when you grab a tool on the internet, even a tool that is doing conceptually simple things and that has a track record of working with much more esoteric sorts of jobs, that inevitably a bug or an unexpected issue arises to throw a monkey-wrench into the works?
I may end up having to roll my own geobin parser. Tedious, but not particularly difficult thanks to knowing the spec ahead of time. I was kind of hoping to avoid the tedium and have a tool that I could use to likewise quickly prototype costume and entity parsers. If the Kaitai people confirm that I found a bug and that they aren't in a hurry to fix it, then my little project here will once again put me in the position of re-inventing the wheel - in this case, a version of bindump, albeit one that only speaks Parse6 and that has an API that a bot can hook into if it needs to. I'm hoping it doesn't come to that, though.
-
Why is it that when you grab a tool on the internet, even a tool that is doing conceptually simple things and that has a track record of working with much more esoteric sorts of jobs, that inevitably a bug or an unexpected issue arises to throw a monkey-wrench into the works?
If I had a nickel for every time I said that, I would have collapsed into a black hole by now.
-
Step One: Create a Kaitai grammar for geobin format.
Step Two: ????
Step Three: Profit!!!
Step One is complete. Since they say up-front that their Python support is dodgy and I've seen one bug myself before I even really got started, I'm being cautiously optimistic until I've managed to actually do something with their generated Python code that spits out the same data that bindump spits out. The web IDE definitely is able to parse the map file successfully. Worst case is that I'd end up making a stand-alone C program and using "geodump" to dump the geobin data I care about into XML that can be transferred into a database that the bot can deal with.
(https://s22.postimg.cc/l6dbxbnbh/geo_ksy.jpg) (https://postimg.cc/image/l6dbxbnbh/)
For the curious, here's the Kaitai grammar for geobin.xml:
meta:
id: geobin
file-extension: bin
endian: le
seq:
- id: header
type: root_header
- id: bin_data
type: root_root
types:
root_header:
seq:
- id: signature
type: str
encoding: ASCII
size: 8
- id: crc
type: u4
- id: parse_version_size
type: u2
- id: parse_version
type: str
encoding: ASCII
size: parse_version_size
- id: file_string_size
type: u2
- id: file_string
type: str
encoding: UTF-8
size: file_string_size
- id: table_size
type: u4
- id: file_count
type: u4
- id: table_data
size: table_size
files_rec:
seq:
- id: filename
type: pad_string
- id: filedate
type: u4
root_root:
seq:
- id: version
type: u4
- id: scenefile
type: pad_string
- id: loadscreen
type: pad_string
- id: def_count
type: u4
- id: def_array
type: root_def
repeat: expr
repeat-expr: def_count
- id: ref_count
type: u4
- id: ref_array
type: root_ref
repeat: expr
repeat-expr: ref_count
- id: map_imports
size-eos: true
pad_string:
seq:
- id: strlen
type: u2
- id: strval
type: str
size: strlen
encoding: ASCII
- id: padding
size: (4 - _io.pos) % 4
doc: This adds padding to the next full four-byte word. Also, generated Python code may have a bug (duplicate /) that needs to be checked in the output code here.
root_ref:
seq:
- id: ref_size
type: u4
- id: ref_body
type: root_ref_body
size: ref_size
root_ref_body:
seq:
- id: ref_str1
type: pad_string
- id: ref_pos_x
type: f4
- id: ref_pos_y
type: f4
- id: ref_pos_z
type: f4
- id: ref_pyr_pitch
type: f4
- id: ref_pyr_yaw
type: f4
- id: ref_pyr_roll
type: f4
root_def:
seq:
- id: def_size
type: u4
- id: def_body
type: root_def_body
size: def_size
root_def_body:
seq:
- id: def_name
type: pad_string
- id: group_count
type: u4
- id: group_array
type: root_def_group
if: group_count > 0
repeat: expr
repeat-expr: group_count
- id: property_count
type: u4
- id: property_array
type: root_def_property
if: property_count > 0
repeat: expr
repeat-expr: property_count
- id: tintcolor_count
type: u4
- id: tintcolor_array
type: root_def_tintcolor
if: tintcolor_count > 0
repeat: expr
repeat-expr: tintcolor_count
- id: ambient_count
type: u4
- id: ambient_array
type: root_def_ambient
if: ambient_count > 0
repeat: expr
repeat-expr: ambient_count
- id: omni_count
type: u4
- id: omni_array
type: root_def_omni
if: omni_count > 0
repeat: expr
repeat-expr: omni_count
- id: cubemap_count
type: u4
- id: cubemap_array
type: root_def_cubemap
if: cubemap_count > 0
repeat: expr
repeat-expr: cubemap_count
- id: volume_count
type: u4
- id: volume_array
type: root_def_volume
if: volume_count > 0
repeat: expr
repeat-expr: volume_count
- id: sound_count
type: u4
- id: sound_array
type: root_def_sound
if: sound_count > 0
repeat: expr
repeat-expr: sound_count
- id: replacetex_count
type: u4
- id: replacetex_array
type: root_def_replacetex
if: replacetex_count > 0
repeat: expr
repeat-expr: replacetex_count
- id: beacon_count
type: u4
- id: beacon_array
type: root_def_beacon
if: beacon_count > 0
repeat: expr
repeat-expr: beacon_count
- id: fog_count
type: u4
- id: fog_array
type: root_def_fog
if: fog_count > 0
repeat: expr
repeat-expr: fog_count
- id: lod_count
type: u4
- id: lod_array
type: root_def_lod
if: lod_count > 0
repeat: expr
repeat-expr: lod_count
- id: def_type
type: pad_string
- id: def_flags
type: u4
enum: def_enum1
- id: def_alpha
type: f4
- id: def_obj
type: pad_string
- id: texswap_count
type: u4
- id: texswap_array
type: root_def_texswap
if: texswap_count > 0
repeat: expr
repeat-expr: texswap_count
- id: soundscript
type: pad_string
- id: leftovers
size-eos: true
enums:
def_enum1:
0x00000001: ungroupable
0x00000002: fadenode
0x00000004: fadenode2
0x00000008: nofogclip
0x00000010: nocoll
0x00000020: noocclusion
0: none
root_def_group:
seq:
- id: group_size
type: u4
- id: group_name
type: pad_string
- id: pos_x
type: f4
- id: pos_y
type: f4
- id: pos_z
type: f4
- id: pyr_pitch
type: f4
- id: pyr_yaw
type: f4
- id: pyr_roll
type: f4
- id: group_flags
type: u4
enum: group_enum1
enums:
group_enum1:
0x00000001: disableplanarreflections
0: enableplanarreflections
root_def_property:
seq:
- id: prop_size
type: u4
- id: prop_str1
type: pad_string
- id: prop_str2
type: pad_string
- id: prop_int1
type: u4
root_def_tintcolor:
seq:
- id: tint_size
type: u4
- id: tint_uarray1_0
type: u4
- id: tint_uarray1_1
type: u4
- id: tint_uarray1_2
type: u4
- id: tint_uarray2_0
type: u4
- id: tint_uarray2_1
type: u4
- id: tint_uarray2_2
type: u4
root_def_ambient:
seq:
- id: ambient_size
type: u4
- id: ambient_uarray1_0
type: u4
- id: ambient_uarray1_1
type: u4
- id: ambient_uarray1_2
type: u4
root_def_omni:
seq:
- id: omni_size
type: u4
- id: omni_uarray1_0
type: u4
- id: omni_uarray1_1
type: u4
- id: omni_uarray1_2
type: u4
- id: omni_f1
type: f4
- id: omni_flags
type: u4
enum: omni_enum1
enums:
omni_enum1:
0x00000001: negative
0: positive
root_def_cubemap:
seq:
- id: cm_size
type: u4
- id: cm_int1
type: s4
- id: cm_int2
type: s4
- id: cm_f1
type: f4
- id: cm_f2
type: f4
root_def_volume:
seq:
- id: vol_size
type: u4
- id: vol_f1
type: f4
- id: vol_f2
type: f4
- id: vol_f3
type: f4
root_def_sound:
seq:
- id: sound_size
type: u4
- id: sound_str1
type: pad_string
- id: sound_f1
type: f4
- id: sound_f2
type: f4
- id: sound_f3
type: f4
- id: sound_flags
type: u4
enum: defsoundenum1
enums:
defsoundenum1:
0x00000001: exclude
0: include
root_def_replacetex:
seq:
- id: rt_size
type: u4
- id: rt_int1
type: s4
- id: rt_str1
type: pad_string
root_def_beacon:
seq:
- id: beacon_size
type: u4
- id: beacon_str1
type: pad_string
- id: beacon_f1
type: f4
root_def_fog:
seq:
- id: fog_size
type: u4
- id: fog_f1
type: f4
- id: fog_f2
type: f4
- id: fog_f3
type: f4
- id: fog_uarray1_0
type: u4
- id: fog_uarray1_1
type: u4
- id: fog_uarray1_2
type: u4
- id: fog_uarray2_0
type: u4
- id: fog_uarray2_1
type: u4
- id: fog_uarray2_2
type: u4
- id: fog_f4
type: f4
root_def_lod:
seq:
- id: lod_size
type: u4
- id: far
type: f4
- id: far_fade
type: f4
- id: scale
type: f4
root_def_texswap:
seq:
- id: ts_size
type: u4
- id: ts_str1
type: pad_string
- id: ts_str2
type: pad_string
- id: ts_int1
type: s4
-
If you want just a very rough idea of the terrain and are willing to build the whole scene graph (including the object library), one thing you could do is load the .bounds files instead of trying to parse geometry.
Sorry for "thinking out loud" but it helps me think. If someone had told me from the beginning that something as conceptually simple as "run away from me until I catch you or you reach a target destination" was going to involve learning how to solve real game development problems, I would probably have just said, "I guess I'll wait until Codewalker solves it."
In a lot of ways, I should just "settle" for something that I can develop and run with right now. Right now, today, I could make a system where I lay out a bunch of waypoints on a map and script a bot to travel between those waypoints. Doing that would give me some concrete results.
It would also be completely mechanical. What I desire is to instruct a bot to "go to City Hall" and have it find its own path to that waypoint, or to instruct it "run away from me" and have it find its own flee path without a specific destination in mind, and without following a predictable path on rails. In short, I want a solution that works for any map in the game, and not something that is specific to a particular map, and I want to avoid trying to "render" geometry where it's at all possible.
Currently, and the reason I'm working to be able to read a city zone geobin file, I've been planning to use enemy spawn points and existing navigation beacons as a network of pre-existing waypoints where the ground level is known and the waypoints are guaranteed to be "outside". (For the moment, I'm ignoring the fact that areas like "indoors" City Hall or Freedom Corp exist. I can eventually remove those nodes from the network.) If the bot occasionally falls through the ground on a hillside, or gets momentarily hung up on a fence, I'm okay with that for the time being because the distance between most waypoints is short enough that Rover's "ground level" will be corrected quickly.
Outside of waypoints, I've been reading about navigation meshes and that got me thinking about the bounding boxes again. If a bot can compute the bounding boxes in the local area, then invert that model, it would essentially have a sort of dynamically generated navigation mesh where any point within that "inverted" area is a point it can travel through. The waypoint network would still be useful as a sanity check for "where's the ground" but the bot would be free to plot a path anywhere in the general area of the waypoints instead of strictly on rails between them.
However, first things first. If a navigation mesh, of sorts, is to work as I've outlined above then I still need to first have a working network of waypoints, so that remains the current goal. Create a network of waypoints. Create a pathing algorithm to traverse the network given a start and a destination. Generalize it to work with any existing zone or mission map.
-
Okay, time to admit that I've reached the limits of my current skillset and that I am just not that interested in learning how to parse the geometry of a city zone and then ray cast around in it to find collisions.
My original idea/theory was that I could use the pre-existing spawn points and beacons as a kind of neural network. If I plotted all of the edges between all of the points and then did an A* search or something similar, the final path would, in most cases, go around large obstacles and get hung up only briefly on small obstacles that it didn't also go around. I've lost confidence in this idea recently as I've been learning about just what a pain path-finding is in real gaming situations, but I might still try pursuing it just to see where it leads. This might hold some promise if I ever got a good enough understanding of the geometry/bounds situation to be able to test each possible edge and eliminate any that intersected the bounds box of an object, then I hand-added any "missing" paths for special cases like, say, walking around underneath Atlas. Since this is basically ray-casting, it's a question of whether I want to learn yet another thing (or more likely a half-dozen other things related to OpenGL) that I never intended to learn in order to accomplish it.
My alternate idea/theory was that I could do a movement-based method of path-finding instead of graph-based, where the bot would "look" in the direction it was facing towards the nearest known waypoint, and be "pulled" toward it if it was a beacon/spawn or "pushed" away from it if it was an object. The problem with this idea is that a whole bunch of the "objects" in the Atlas Park map, for instance, are buildings with walkable plazas attached. Doing things this way, the bot would completely avoid walking near the train station ramp, let alone up it OR under it, and likewise it would never even attempt to walk under the statue of Atlas. I have no clue at all how it would handle something like the Croatoa forest paths or Perez Park.
Yeah, and the idea of somehow rendering all of the bounds boxes into some kind of graph and turning that into a navigation mesh is making my head spin.
So, I guess I'm just going to have to deal for now with the fact that my bot is going to have to have its paths on any given map graphed out ahead of time. For the case where I want the bot to "run away" or "run to $DESTINATION" I'll have to just brute force trace all of the paths specifically for that particular map and try to make enough vertices that a bot starting from point A and running to point B will at least feel different from a bot starting from point A and running to point C or point B'.
Besides - it's going to have to do it that way anyway, if it's inside a base instead of on a zone or mission map. Bases don't have a map file with all of the geometry laid out ahead of time.
And people wonder why nobody just "makes an emulator". It's really a shame that nobody at Paragon Studios smuggled out a dvd of the server source code.
On the plus side, if you'd tried talking to me about "A* searching" a couple of weeks ago, I probably would have given you a blank look.
-
At the risk of appearing to be using this thread as my personal blog, I'm going to toss out a few more ideas I'm thinking about; both to codify them in my own mind and to put them out there where smarter people who already know about these things can say, "No; that's a terrible idea." (Though, I'm keeping in mind Arcana's admonition that paraphrases to, "If people only ever waited for perfect solutions, most of the time they end up doing nothing.") OTOH, if I'm turning into Joshex, then someone needs to slap me upside the head and say, "Hey! Wake up!"
The biggest problem I have with all of this is that I'm having to embrace the idea of Rover as a pseudo-game-client and I have zero experience with 3D environmental programming. Despite that, and despite the fact that anything I do currently is likely to be rendered obsolete the first day that Codewalker posts, "Hey, I've added NPC's and LUA scripting to Paragon Chat!", I seem to still be stuck with a desire to see Rover "run away" in a "life-like" fashion instead of in a "train on rails" fashion.
So, I have two semi-related problems to solve: One is pathfinding, and the other is terrain navigation.
"Semi-related" because you can't begin to understand where the paths are without first understanding where the terrain is. This is the bit that I've been having trouble wrapping my head around. Scene graphs, I get, mostly. Collision math, I get, mostly. Determining how you decide what things to test for collision without having to test EVERYTHING, every clock tick, is the bit that's been eluding me.
The first step was to read a geobin file. I've got a parser that can do that now, even though I haven't put it to use yet. As an exercise of sorts, I DID make a clientmessages parser and generate a strings.cfg file from it. I can now modify all the strings in my personal Paragon Chat instance. Whee! I also learned, for instance, that the game client pre-generated instance names for every city zone up to around fifty instances, which is interesting, if not immediately useful. (Though it did make me wonder if I could ADD strings to strings.cfg that don't exist in clientmessages. I'm not sure what Paragon Chat would do with that. Probably just ignore it as a presumed typo.)
The point being that if I could use my Kaitai-generated parser to load up the whole clientmessages file without running out of memory, access the "P-strings" array and use that info to access the associated strings in the strings array, and write the whole thing out as a strings.cfg file then I should, in principle, be able to do the same process with the geobin parser. That is, I assume at this point that the parsers I'm generating out of Kaiti (I made a .bounds parser as well) will work as expected with minor touchups to insure that text decoding happens correctly. (Python gets touchy about insisting that things conform to your current code page if you're working in a Windows environment.)
Scene Graph
So, now I can theoretically read a zone map geobin, which is basically a serialized scene graph. So, the question is, "How do you deserialize it?"
Based on the output of bindump, it looks a lot like the original scene graph is being traversed and written out "leaves-first". The Refs array represents the first-level children of the scene Root. I'm not really sure why TextParser felt the need to write the "eldest" children seperately from the others, but there you go.
In any case - the first Defs encountered in the geobin file are the outer "leaves" of the grp_Atmosphere_NPC "branch", which is the first entry in the Refs array. Each group represents its own "branch" that can contain many child groups. The children refer back to their parent, and the parents eventually lead back to one of the root Refs. A parent group can contain multiple instances of a single child group. Defnames that contain file paths lead into the object library, where the geometry lives along with bin files describing each individual model in the .geo file as well as groups of models, and bounds data for the models, represented as both rectangular and spherical bounds.
Once all of the grp_Atmosphere_NPC nodes are read, the next node is a grand-child of Ref grp_audio, which is the next Ref in the Refs array. It appears safe to expect each of the following Refs to be "untraversed" in turn in order to completely deserialize the data in the bin file and arrive at something that resembles the original scene graph.
The issue of geometry rears its head at this point, since the ultimate termination of any particular path through the scene graph should be a piece of geometry that can be drawn or manipulated. Here is where I intend to cheat, using the advice Codewalker recommended last year - My bot can't actually "see" the world and it never intends to render anything visually. The contents of the actual .geo file is irrelevant. All that the bot cares about is the collision bounds. Which means that as far as the bot is concerned, it is living in either Boxville or Spheretown. Either way, the "geometry" of the zone is faked at this point using the data from the .bounds file for the individual objects in the zone. As far as I can tell, EVERYTHING is an object of some sort. Even the streets and sidewalks and plazas are all objects with a bounding box defined as well as the radius of a sphere that presumably likewise encompasses the object.
In short - there is no "ground", as such. Rather, the "ground" is whatever "object" an entities feet happen to be colliding with due to gravity. (I haven't actually followed this line of logic through for a forested/outdoor zone like Croatoa, though I EXPECT it to apply the same way.)
I should emphasize at this point that my tossing around terms like "scene graph" is not some indication that I'm at some high level understanding of the term. When it comes to most of this 3D visualization and rendering stuff, I can say that I'm past the level where the terms represent some sort of mystical incantation that causes pretty pictures to appear on a computer screen. Beyond that, I'm still pretty much Harry Potter in his first year at Hogwarts, and I'm definitely no Hermione.
So, now the question is: What do I do with this scene graph?
For starters, if Rover is going to be acting like a pseudo-client anyway then I figure I should use software suited to the task. That means loading up Openscenegraph and using it to manage the de-serialized scene graph for me. The minimal benefits are that I'll be using software designed for the task I'm trying to accomplish and if I want to actually visualize Boxtown or Sphereville for some reason, it should be possible to come up with some coding examples that illustrate how to do that.
It isn't entirely clear to me yet just how much collision detection is available in Openscenegraph (OSG, hereafter). There are different versions and the documentation ranges from recent to over a decade old. I THINK that I might be able to insert Rover's position and bounding box into the scene graph, and have it automagically incorporated into the scene, such that casting a ray in any given direction will return a list of objects that are potentially things Rover would collide with.
If that doesn't work, I've also found a fallback - Open Dynamic Engine (hereafter, ODE). This is more or less a physics engine sort of like Phys X but intended for doing rigid body simulations, with the "dynamic motion" component separated from the "collision detection" component, so you can use either of them standalone. Plus, ODE has a reasonably simple and easy-to-understand idea of the World and of Spaces. I THINK it would be relatively simple to load Boxville into a ODE World instance and have it treat all of the boxes as motionless/massless bits of terrain and then ODE can, given the position of Rover, deliver a list of collision points that he's likely to hit at his current position and velocity.
Either of these solutions results in Rover sort of "feeling" his way around the geometry of the world, without actually knowing the precise geometry of the world. Practically speaking, I think it would work like I mentioned back when I first brought this up last year, where I'd use the collision points to create an avoidance force that "repelled" Rover away from the obstacle but hopefully still in the general direction he "wants" to go in.
Pathfinding
The other benefit of the fully realized scene graph is that I've got the list of beacons and spawn points that I've mentioned several times previously. These are suitable for use as path waypoints, since each of them is guaranteed to be at "ground level" (with a few exceptions, like the blimp's path around Atlas Park). However - Waypoints without any paths between them aren't really useful.
The solution, as I see it, is to build a "neural net" graph that connects every waypoint with every other waypoint. I then feed that graph into a scanner whose job is to brute-force test each of the resulting line segments for collisions using the scene graph/terrain visualization. Segments that come up with a positive hit are dropped from the database. POSSIBLY, the scanner could check the height of the bounding box for the offending object and if it was shorter than some arbitrary jump height, that segment could be broken into two new segments that both terminate at the object and that both are marked as "jump over" at the endpoints near the obstacle. I think that would fall into "stretch goal" territory.
The end result would be a database of waypoints that each connect to at least one other waypoint, forming a graph of the "walkable" space in the zone. Such a graph could then be searched with a A* search or some similar pathfinding algorithm to dynamically find a path from any "point A" to any "point B" that lies within the graph. Such a database could be generated once and then used by any bot, even if that bot knows nothing at all about the geometry of the zone. As long as it was located somewhere on the "pathnet", it could find a way to every other part of it.
I'm only just at the barely beginning stages of realizing any of the above vision, but I believe that the above actually represents an achievable "spec" for a bot that can wander freely around any zone map or mission map in the game, given appropriate preparation.
-
Some really good info in this thread, I'd wondered what the files section of the bin files was for!
I've been working on (hopefully eventually) a map viewer for a bit of fun, but I'm also struggling with geometry. I've found enough code out there for version 0 .geo files, but I'm having trouble with later versions. GeoDraw can open all .geo files that I throw at it, but I can't find the source anywhere - I think it was made by CW and GuyPerfect so hopefully, they'll see this and possibly shed some light on those.
It would be really great to get some comprehensive documentation somewhere, people with reverse-engineering skills are few and far between, so unless we write this stuff down I worry it might die one day. I know there are certain legal issues around reverse-engineering, but I don't believe there are any legal issues with sharing RE'd info that you've learned without actually doing the RE. I think, anyway.
Anywho if anyone's interested in what I've got so far (just doing file parsing tests so far) GitHub (http://"https://github.com/spectre1989/thor")
-
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.
Let me see what I can dig up.
EDIT: I do remember seeing something about the SEGS folks having a mostly working map viewer, so they might have some useful info there. Though since they're using a really old client it may only have to deal with version 0 geos.
-
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.
Let me see what I can dig up.
EDIT: I do remember seeing something about the SEGS folks having a mostly working map viewer, so they might have some useful info there. Though since they're using a really old client it may only have to deal with version 0 geos.
That'd be awesome if you could - I've been using segs as my main source of information so far, but I found for pigg/bin loading that having multiple sources to draw on helped fill in the gaps missed by one or the other. Oh, and as you say, segs expects version 0 I think, it certainly doesn't handle the version 7/8 I threw at it.
There are other files I have yet to look at:
.bounds - well, I had a brief look and figured out the min/max/radius fields, not sure what any of the others mean.
.texture - segs should help me here, but I'm not sure if there's any versiony business to watch out for
.anim - has anyone reversed these yet?
I plan to write up some file format docs in the github repo at some point
-
Hey, Codewalker - I've got a question about the schema.xml files:
When there's something like this:
<entry name="ObjFlags" type="FLAGS" enum="static">
<enumval value="0x00000001" name="ForceAlphaSort" />
<enumval value="0x00000400" name="ForceOpaque" />
<enumval value="0x00000100" name="TreeDraw" />
<enumval value="0x00000008" name="SunFlare" />
<enumval value="0x00001000" name="WorldFx" />
<enumval value="0x00008000" name="StaticFx" />
<enumval value="0x00000004" name="FullBright" />
<enumval value="0x00000010" name="NoLightAngle" />
<enumval value="0x00000002" name="FancyWater" />
<enumval value="0x00080000" name="DontCastShadowMap" />
<enumval value="0x00100000" name="DontReceiveShadowMap" />
</entry>
An enumeration, to me, says that the field in question holds exactly one of those values. The enumvals are just names for various potential values. So, ObjFlags, for instance, can be set to StaticFX or FullBright, but not both.
However, I can't help noticing that all of these enumerations are bit-wise values and the type of ObjFlags is "FLAGS", plural. Is the above paragraph correct, or is the schema actually using "enum" to describe all of the possible flags that could be OR'ed together to make the composite value of ObjFlags (or whatever enum variable is being described)?
-
If the type of the field is FLAGS is generally indicates a bitmapped flags field, so you can probably OR them together.
The same type of internal structure is used for the parser to recognize text names of both enums and flags, that's why it gets called <enumval/> in the schema dump.
-
Groovy. I know it was kinda obvious but I hadn't really paid attention in the past to how some enums had type "INT" and some had type "FLAGS" so I figured better to clarify than assume the "obvious" thing.
-
Some of the fields aren't named in the schema, but I've had some success in filling in the blanks by dumping symbols from a PDB and finding the struct/class definitions. Some fields are missing due to the symbols being outdated but gets a bit closer.
-
.anim - has anyone reversed these yet?
I'm not sure how much it helps, given that it was released back around I-9 or I-10ish timeframe, but the Cryptic Animation Rig is still available with a bit of Wayback Machine-fu. A bit more searching turned up some efforts by a guy in the past to reproduce it in Blender, though I didn't manage to find any downloadable data files at the time I was looking for them.
If nothing else, just being able to see the rig makes it easier to conceptualize how the costume bits fit together. I don't think it necessarily helps with the .geo or .anim files directly, in that they must have had some custom import plugins to 3DS Max to handle their custom data files, but maybe the data points in the rig can help identify data points in the .geo/.anim files.
In fact, here, I'll save you the searching of the Wayback - Cryptic was giving this stuff away before they let the domain lapse so I don't see any harm from sharing what they already shared: https://www.dropbox.com/sh/h3h3fjxhaces2n2/AACfmQbJHK-h5G2al-hXwzD_a?dl=0 (https://www.dropbox.com/sh/h3h3fjxhaces2n2/AACfmQbJHK-h5G2al-hXwzD_a?dl=0)
-
Nice one, thanks :)
-
anim files are simultaneously really simple and deceptively complex.
The format itself is very barebones (no pun intended), just some basic headers. Each bone in the rig has a flat list of rotations and positions for each frame (30 frames in 1 second).
Where it gets complicated is that the animation data is compressed using one of several different methods. *Most* of the rotation data is compressed down to 5-byte packed quaternions, where 3 of the components are stored as 12-bit fractions that map to the range [-1/sqrt(2), 1/sqrt(2)]. The largest component is omitted and reconstructed from the other 3 when loading, which gets a little mathy.
If that weren't enough, rotations can also be stored as 8-byte quaternions where each component is a 16-bit fraction, or simply uncompressed as 32-bit floats. There's also an alternate 5-byte format that doesn't use a linear mapping but instead represents the components as the output of a sine function, but while the code can read it, I haven't encountered any anim files that actually use that one.
Similarly, position data can either be stored as floats, or with a 16-bit half precision variant that is used if the data falls into the (-1,1) range.
Since anims aren't really needed serverside except for the skeleton hierarchy and to get animation lengths in some cases, I haven't done much code work on them. If you're looking at client modding / expansion, I'd recommend getting a solid handle on geos first and some usable tools for them written before going down the anim rabbit hole.
-
anim files are simultaneously really simple and deceptively complex.
The format itself is very barebones (no pun intended), just some basic headers. Each bone in the rig has a flat list of rotations and positions for each frame (30 frames in 1 second).
Where it gets complicated is that the animation data is compressed using one of several different methods. *Most* of the rotation data is compressed down to 5-byte packed quaternions, where 3 of the components are stored as 12-bit fractions that map to the range [-1/sqrt(2), 1/sqrt(2)]. The largest component is omitted and reconstructed from the other 3 when loading, which gets a little mathy.
Oh cool, smallest 3 compression - we use that in netcode often for sending rotations over the network.
If that weren't enough, rotations can also be stored as 8-byte quaternions where each component is a 16-bit fraction, or simply uncompressed as 32-bit floats. There's also an alternate 5-byte format that doesn't use a linear mapping but instead represents the components as the output of a sine function, but while the code can read it, I haven't encountered any anim files that actually use that one.
Similarly, position data can either be stored as floats, or with a 16-bit half precision variant that is used if the data falls into the (-1,1) range.
Since anims aren't really needed serverside except for the skeleton hierarchy and to get animation lengths in some cases, I haven't done much code work on them. If you're looking at client modding / expansion, I'd recommend getting a solid handle on geos first and some usable tools for them written before going down the anim rabbit hole.
Yeah I'm doing clienty stuff, as I said before, starting with a map viewer, and if I ever finish that it'd be cool to spawn some characters in there. As you say though, start with static stuff first and worry about anims if I actually make that far!
Thanks for the info :)
-
I don't think there's any formal documentation for the format. I've given the geodraw source to a few people who were working on various tools, but don't remember if I've posted it anywhere publicly or not. It's pretty terrible code though -- we hastily threw it together after a long night of reverse engineering the game's geo loader with a debugger.
Let me see what I can dig up.
Just wanted to add my vote here as interested in seeing the geodraw source code if it's available. Trust me, I'm not judging anyone on how "bad" their code is. Once I made the decision that Rover has to load a scene graph and act like a client, I've come more and more around to the idea that I originally had when I started this whole thing, which is that what I should be writing is a bot controller that can manage several bots rather than launching one process per bot.
In short; if I have to get intimately involved with the scene graph and the geometry, then what makes the most sense is a central "Director" that knows the geometry of a zone and tracks all of the moving parts, and that spawns individual "Actors" that are capable of following their individual scripts but can query the "Director" for world-related information when they need it, so that only the "Director" has to carry the whole zone around in its head. That ends up sounding a lot like a map server instead of like a "bot". If I'm going that far, then I might as well get down to the actual geometry of the world or at least have the option to do that.
I've added Panda3D to the list of 3D engines to get educated about. I'm starting to think it's the tool I need for this: It has the scenegraph support, it has native collision detection and it also supports Bullet and ODE, it's a professional-level tool (though mostly supplanted by Unity in recent years), it actually has decent documentation, and it's written in C++ but primarily aimed at python so the python bindings are up-to-date, maintained and documented.
-
Yeah, we've all written messy code - you can find some of my embarrassments on GitHub :D
-
Btw slickriptide, what is it from the geos that you actually need? If you just need meshes I think I'm close to having that much figured out, it's just a lot of the other data in the files that I'm not sure about.
-
Btw slickriptide, what is it from the geos that you actually need? If you just need meshes I think I'm close to having that much figured out, it's just a lot of the other data in the files that I'm not sure about.
For the moment, I'm still figuring out the scene graph deserialization, so I don't need geo data yet, and I'll probably fake it at first by just using boxes based on the bounds in the .bounds files. Assuming that all that works and Boxtown becomes a reality, then I might see about incorporating actual geometry for collision purposes. I THINK I'd just need mesh data for that, since a bot wouldn't care what the texture looked like. OTOH, I seem to be heading down the road of using Panda3d to handle my scenegraph, and that means that once I had the scenegraph fully loaded and ready to use that I'd be halfway to having a map viewer, so THAT might be a good reason to be able to load textures and geometry together.
-
Speaking of scenegraphs, I think I'm almost ready to have a go at building a scenegraph in Panda3D. The initial levels, anyway.
The Kaitai parser is working fine. It's pretty cool, really, to write a few lines of code and have an entire geobin file load up, ready to be used.
My first little geobin parser looks like this:
from geobin import *
import psutil
import gc
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
print("Loading geobin...")
gb = Geobin.from_file("C:/Users/Scott/Downloads/tequila/Extracted_Full/geobin/maps/City_Zones/City_01_01/City_01_01.bin")
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
defs = gb.bin_data.def_array
num_defs = gb.bin_data.def_count
refs = gb.bin_data.ref_array
num_refs = gb.bin_data.ref_count
print("Num Defs: %d / Num Refs: %d" % (num_defs, num_refs))
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
print("Listing Refs")
for ref_index in range(0, num_refs, 1):
rb = refs[ref_index].ref_body
ref_name = rb.ref_str1.strval
ref_pos = (rb.ref_pos_x,rb.ref_pos_y,rb.ref_pos_z)
ref_o = (rb.ref_pyr_yaw, rb.ref_pyr_pitch, rb.ref_pyr_roll)
print("Refname:%s -- Pos%s Hpr%s" % (ref_name, ref_pos, ref_o))
print("Freeing geobin...")
del defs
del num_defs
del refs
del num_refs
del gb
print("Mem footprint: %fmb" % (psutil.Process().memory_info().rss/2**20))
and produced this:
PS C:\errbot.bogart\kaitai> python gb_test.py
Mem footprint: 16.636719mb
Loading geobin...
Mem footprint: 52.238281mb
Num Defs: 7239 / Num Refs: 33
Mem footprint: 52.238281mb
Listing Refs
Refname:grp_GC_Shutdown_Atmo -- Pos(-71.0, 15.931458473205566, -474.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_audiogroup -- Pos(186.380859375, 67.989501953125, -938.831298828125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Badge_tourism -- Pos(721.3763427734375, 164.83200073242188, -699.31494140625) Hpr(0.07853981852531433, 0.0, -0.0)
Refname:grp_beacons -- Pos(522.87060546875, 262.0234375, -574.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_blackmap -- Pos(512.0, -128.0000457763672, -763.474609375) Hpr(0.0, 0.0, 0.0)
Refname:grp_blimp -- Pos(176.0, 707.0, -1040.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Day_Job_Volumes -- Pos(-59.5, -867.0, -914.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Geometry -- Pos(511.38037109375, 569.0, -703.999755859375) Hpr(0.0, 0.0, 0.0)
Refname:grp_Halloween_Event_06 -- Pos(-68.01048278808594, 4.472891807556152, -10.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_Holiday_Event_06 -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HollowsBeacons -- Pos(189.818359375, 245.71826171875, -2069.03076171875) Hpr(0.0, 0.0, 0.0)
Refname:grp_Makeover_Spawndefs -- Pos(642.7833251953125, 7.882775783538818, -797.4718017578125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Manholes -- Pos(1235.75, 156.25277709960938, -242.75) Hpr(0.0, 0.0, 0.0)
Refname:grp_mission_doors -- Pos(120.030517578125, 465.505859375, -748.54931640625) Hpr(0.0, 0.0, 0.0)
Refname:grp_neighborhoods -- Pos(-32.0, 0.0, -351.9999694824219) Hpr(0.0, 0.0, 0.0)
Refname:grp_patrol_box -- Pos(-5.161986827850342, 18.90635108947754, -706.25) Hpr(0.0, 0.0, 0.0)
Refname:grp_PersistentNPC -- Pos(-25.9998779296875, -159.36572265625, -899.901611328125) Hpr(0.0, 0.0, 0.0)
Refname:grp_Plaques -- Pos(206.2913818359375, 24.575931549072266, -414.63629150390625) Hpr(0.0, 0.0, 0.0)
Refname:grp_PVP_Scripts -- Pos(140.43020629882812, 20.40468406677246, -241.53944396972656) Hpr(0.0, 0.0, 0.0)
Refname:grp_Rikti_Invasion_Event -- Pos(2260.0, 0.0, 3388.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_RiktiBetaEvent -- Pos(-83.82254028320312, 841.516357421875, -643.6749877929688) Hpr(0.0, 0.0, 0.0)
Refname:grp_ShadowShardInvasion -- Pos(511.4013671875, 409.1253356933594, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Truck_Door -- Pos(340.6702880859375, 135.29078674316406, -1199.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_warpvolumes -- Pos(-1664.0, -896.0, 1408.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Winter_Event -- Pos(511.4013671875, 409.2863464355469, -812.486572265625) Hpr(0.0, 0.0, 0.0)
Refname:grp_HalloweenEvent_09 -- Pos(1199.0, 41.59211349487305, -1189.7000732421875) Hpr(0.0, 0.0, 0.0)
Refname:grp_ParagonCityTeleport -- Pos(-33.21428680419922, 0.0, -171.35714721679688) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Phase -- Pos(637.5, -0.4998469948768616, 67.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Mission_Trays -- Pos(2450.0, 56.017494201660156, -1372.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_spawndefs -- Pos(772.814208984375, -16.52880859375, -699.10009765625) Hpr(0.0, 0.0, 0.0)
Refname:grp_Zowies -- Pos(2442.5, 57.2479248046875, -1374.5) Hpr(0.0, 0.0, 0.0)
Refname:grp_GalCty_PerstntNPC -- Pos(391.30340576171875, 68.48408508300781, -864.0) Hpr(0.0, 0.0, 0.0)
Refname:grp_Kraken_Event -- Pos(1354.25, -32.00006103515625, -736.5) Hpr(0.0, 0.0, 0.0)
Freeing geobin...
Mem footprint: 52.238281mb
PS C:\errbot.bogart\kaitai>
My concern at the moment is this bit: "Freeing geobin...Mem footprint: 52.238281mb". Freeing the geobin parser didn't actually free the memory it used, probably because the underlying Kaitai structures are still referencing the data. There's not a lot I can do about that, unless I want to roll my own parser and handle the memory management myself, which I don't. By itself, 52mb isn't any big deal. I'm more concerned with what happens when I start going into the object library and opening up dozens of geobins associated with object library pieces. I guess I'll cross that bridge when I come to it. Most of the low level Kaitai stuff is undocumented, and there may be some method it has to free up its memory that I just don't currently know about.
That aside, it's handy that the Kaitai parser loads up the entire geobin file at once. I've pretty much verified that the "right" way to deserialize the scenegraph is to do it in reverse, with respect to the way the data is stored in the geobin file. I could build it up leaf-to-branch-to-root but it should be easier and less error-prone to just start at the end of the refs array and the defs array and move backwards through them, building the tree naturally from the root to the leaves.
The refs are the first-level children of the root. Everything else depends from there. Bindump confirms that moving backwards from array-end to array-start goes like this:
-------- Defs 6055 --------
Name = grp_worldlink_beacons
Group =
-------- Group 0 --------
Name = grp6676
Pos =
X = -1283.57
Y = -31.8641
Z = 311.098
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
-------- Group 1 --------
Name = grp1745
Pos =
X = -309.435
Y = -33.1197
Z = 1652.77
PYR =
Pitch = 0
Yaw = 0
Roll = 0
Flags = 0
Property =
-------- Property 0 --------
u034 = Layer
u035 = worldlink_beacons
u036 = 0
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = 0
Alpha = 0
Obj =
TexSwap = []
SoundScript =
"Name" is the name of the parent node. The Group array represents the children to add to that parent. The other arrays likewise are attributes that either apply directly to the parent or that are attached as children to the parent.
Here's our bindump excerpt for the next two defs, moving backwards:
-------- Defs 6053 --------
Name = grp1744
Group =
-------- Group 0 --------
Name = Omni/MissionBeacons/_PlayerSpawn
Pos =
X = 0
Y = 0
Z = 0
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
Property =
-------- Property 0 --------
u034 = SpawnLocation
u035 = LinkFrom_City_01_02
u036 = 0
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = Ungroupable (1)
Alpha = 0
Obj =
TexSwap = []
SoundScript =
-------- Defs 6054 --------
Name = grp1745
Group =
-------- Group 0 --------
Name = grp1384
Pos =
X = 43.5209
Y = -421.523
Z = -24.6548
PYR =
Pitch = 0
Yaw = -2.79253
Roll = 0
Flags = 0
-------- Group 1 --------
Name = grp1388
Pos =
X = 43.0209
Y = -421.523
Z = -9.65479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 2 --------
Name = grp1403
Pos =
X = -52.4791
Y = -421.523
Z = -9.15479
PYR =
Pitch = 0
Yaw = 2.96706
Roll = 0
Flags = 0
-------- Group 3 --------
Name = grp1617
Pos =
X = -38.9791
Y = -421.523
Z = -25.1548
PYR =
Pitch = 0
Yaw = 2.61799
Roll = 0
Flags = 0
-------- Group 4 --------
Name = grp1618
Pos =
X = -39.9791
Y = -421.523
Z = -9.65479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 5 --------
Name = grp1619
Pos =
X = -4.47913
Y = -422.023
Z = -1.15479
PYR =
Pitch = 0
Yaw = -3.14159
Roll = 0
Flags = 0
-------- Group 6 --------
Name = grp1744
Pos =
X = 43.0209
Y = -421.523
Z = -33.1548
PYR =
Pitch = 0
Yaw = -2.61799
Roll = 0
Flags = 0
Property = []
TintColor = []
Ambient = []
Omni = []
Cubemap = []
Volume = []
Sound = []
ReplaceTex = []
Beacon = []
Fog = []
Lod = []
Type =
Flags = 0
Alpha = 0
Obj =
TexSwap = []
SoundScript =
Sure enough - Def 6054 is the last child of Def 6055, and Def 6053 is the last child of Def 6054. The pattern repeats until we hit a leaf; typically an object library piece. Then the "de-traversal" moves back up the tree and follows the next branch to its end and so on.
So, as long as I've got a handle on creating appropriate Panda3D NodePaths and attaching them together correctly, the scenegraph pretty much writes itself, at least up to the point where the object library and the actual geometry come into play.
The object library geobins are technically identical in format to the map geobins, except that they don't have refs, and they have a corresponding .bounds file that needs to be attached to the scene as part of the leaf data. The one unusual thing is that "leaf" nodes in an object library have TWO terminating records. One is a record with the object name and an Object attribute as a child. The last is an empty node with the object's name as the parent. Presumably, the former is meant to indicate that it contains geometry and the latter is intended to have the geometry attached to it. At least, that's my best guess at the moment.
Despite what I said about the scenegraph "writing itself", there are a lot of attributes that I'm not sure what to do with or that I'm considering disposing of because I DON'T care about them. Sounds, or Level of Detail (LOD) nodes, for instance. The real game client cares, of course, but a bot doesn't care; at least not in any way I'm currently envisioning. If I end up having to define a bunch of custom nodes to handle that stuff, or the memory footprint becomes unwieldy, then it might be easier to dispose of it. I guess, like other things, that I cross that bridge when I come to it. I'm trying to keep in mind that if I do this thing "right", that some other person may find a use for these details even if they aren't particularly useful to me at the present time.
-
While "writing itself" was a bit optimistic, I'm getting the hang of the Panda3d scenegraph.
(https://i.imgur.com/66tmAsw.jpg)
I need to add instancing, so that multiple copies of a node are indirect references to a single node rather than a hundred individual copies of that node. Then I need to work on getting down into the object library.
That's the bare minimum. Then I have to decide what to do with attributes like Properties and Omni and whatever. It turns out it's not as simple as just sub-classing the basic node because of the whole "Python on top of C++" thing). The simplest solution might simply be to program in C++ instead of Python but we'll see how it goes as I go along here.
-
I've written up some of what I've found out or figured about the geo file format, vertex/triangle/normal/texcoord data for all geo versions present in I24. I haven't figured out how the textures are applied yet, but if/when you get to the point of needing to read geos it might save you some time.
https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md (https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md)
-
If you're serious about doing correct collision, you'll probably need to load texture info at some point. Texture names are effectively material names once you tie them to tricks. Aside from editor-only stuff that can probably be excluded at the object level, there are some objects that have faces which are visible but do not block collision. Spider webs and other environmental details come to mind.
Here's the geodraw code. It's ugly, but the logic is almost exactly what the game itself does to load geos (we disassembled the geo loader and single stepped through it).
https://github.com/cwtitan/geodraw
-
https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md (https://github.com/spectre1989/thor/blob/master/documentation/Geo%20File%20Format.md)
Well done, Spectre! You must have been a tech writer in a previous life. That's some of the most complete and readable docs I've read in a while.
https://github.com/cwtitan/geodraw
Thanks, Codewalker! This will help a lot as far as understanding what's happening under the hood.
We'll see about correct collision, but it's certainly something I'd like to succeed at implementing. Whether it works or whether the bots ultimately live in Boxtown will depend a lot on whether I can get a grasp on how to convert a geo model into something that a non-Cryptic engine can use. Panda, in this case, but it could be Ogre or Godot or Openscenegraph or one of the dozens of others out there. I keep hoping that one of the ex-Paragon modelers would "accidentally" let their 3DS import/export plugins escape into the wild.
-
spectre1989: Your "unknown section" that exists from version 2-6 is LOD (level of detail) info. Version 2-5 include one of these for each model in the file, in the same order as the models themselves:
int32 num; // (can be 6 at most)
{
f32 allowedError;
f32 near;
f32 far;
f32 nearfade;
f32 farfade;
uint32 flags;
} // repeated 6 times
Version 6 is similar, but instead of having a static array of 6 structures, it only repeats "num" times, and after each flags field there are two null-terminated strings: model name and file name to substitute at that distance.
Version 7 and newer geos omit this information. Some of the info was moved to lods.bin instead, indexed by model name. This version and higher can do automatic mesh reduction instead of substituting other models -- the information for this is a little complex to describe here but is one of the later sections in the model header.
-
Here's the geodraw code. It's ugly, but the logic is almost exactly what the game itself does to load geos (we disassembled the geo loader and single stepped through it).
https://github.com/cwtitan/geodraw
Thanks very much for sharing that :D
Well done, Spectre! You must have been a tech writer in a previous life. That's some of the most complete and readable docs I've read in a while.
Thanks mate :)
spectre1989: Your "unknown section" that exists from version 2-6 is LOD (level of detail) info. Version 2-5 include one of these for each model in the file, in the same order as the models themselves:
int32 num; // (can be 6 at most)
{
f32 allowedError;
f32 near;
f32 far;
f32 nearfade;
f32 farfade;
uint32 flags;
} // repeated 6 times
Version 6 is similar, but instead of having a static array of 6 structures, it only repeats "num" times, and after each flags field there are two null-terminated strings: model name and file name to substitute at that distance.
Version 7 and newer geos omit this information. Some of the info was moved to lods.bin instead, indexed by model name. This version and higher can do automatic mesh reduction instead of substituting other models -- the information for this is a little complex to describe here but is one of the later sections in the model header.
Oh awesome - some questions:
* do you happen to know how nearfade/farfade work off hand? Saves some brain ache if you do, no worries if not
* do you know what the various flag values mean?
* are there any version 6 geos in I24? I couldn't find any myself, unless there are files which use this same file structure which don't use .geo as the extension
* while we're here, there's something I've always wanted to know about piggs - what's the table of compressed blobs of data which comes after the strings table? I've heard it referred to as "meta"/"headers", but no idea what it's actually for?
-
Fun factoid about geobin files - you have to deserialize them in the order they're written, leaves-first. Otherwise, you decode a lot of instances before you decode the thing they're instancing from.
I discovered this by doing the exact opposite, of course. Back to the drawing board.
-
Fun factoid about geobin files - you have to deserialize them in the order they're written, leaves-first. Otherwise, you decode a lot of instances before you decode the thing they're instancing from.
That's correct. The proper way to read a geobin is to build up a library of Defs by reading them and creating an in-memory Def structure for each one as you go. By doing them in order, it's guaranteed that any child defs of the group are already in memory -- child defs should just be a pointer to one that's already been loaded.
Then Refs, which are the roots of the scene graph, are simply pointers to Defs that have already been loaded.
The object library is a bunch of Defs that are assumed to always be available (preloaded) regardless of which geobin you're reading. Technically they're demand loaded when first referenced, but there's an index so the names are effectively already defined, and they can hang around in the cache when changing maps. There's nothing preventing someone from making a map with a Ref directly to something out of the object library that isn't even defined in the geobin.
Refs are also special in that top-level refs are part of the scene packet and explicitly sent by the server to the client. There is a protocol for tracking the visibility of top-level Refs -- this is used for things like making the Ski Chalet visible, or for making the War Walls invisible during an invasion.
-
Oh awesome - some questions:
* do you happen to know how nearfade/farfade work off hand? Saves some brain ache if you do, no worries if not
Those are easy, it's just the distance ("inside", relative to near/far) that the object starts to fade out.
So if near=50, nearfade=25, far=200, farfade=10,
the object would be invisible if you're closer than 50', would gradually fade to full opacity between 50' and 75', then at 190' would start fading out, and invisible again beyond 200'.
Most full-quality 'master' models don't use near or nearfade, but lower-quality models use those so that they are hidden until you get to a certain distance, which is the old way of doing things.
Objects with AutoLOD (most newer models) just specify far/farfade as the mesh reduction info handles the distances for deleting triangles.
* do you know what the various flag values mean?
According to the textparser table there's only two:
ErrorTriCount: 0x02
UseFallbackMaterial: 0x80
I have no idea what ErrorTriCount does. UseFallbackMaterial I would *guess* from the name might force the use of the fallback textures for the non-multi9 shader (used for graphics cards that can't handle ultra mode features), but I haven't tested that.
* are there any version 6 geos in I24? I couldn't find any myself, unless there are files which use this same file structure which don't use .geo as the extension
Honestly I'm not sure. That info is based on how a version 6 geo would be loaded by the game, but I haven't run a full sweep to see if any of those actually exist. All of the assets created at that point in time might have been re-exported later for some reason, wiping them out.
* while we're here, there's something I've always wanted to know about piggs - what's the table of compressed blobs of data which comes after the strings table? I've heard it referred to as "meta"/"headers", but no idea what it's actually for?
My old C++ libpigg called it "extra", but "headers" is probably the best term.
It's an optimization trick. Certain types of files, specifically geo, texture, and anim, all start with a 4-byte header size field. When pigg files are generated, those types of files have the header extracted and saved uncompressed into a special section of the pigg.
That section is loaded as a single chunk with the pigg file and always kept resident in memory. When the pigg library handles a read request for one of those files, if the offset+size is entirely within the cached header, it can be served without ever having to perform disk I/O.
This mostly affects game startup, when all of the in-memory indexes are built and every texture and geo header is scanned.
-
***edit***
See below
*********
-
That's correct. The proper way to read a geobin is to build up a library of Defs by reading them and creating an in-memory Def structure for each one as you go. By doing them in order, it's guaranteed that any child defs of the group are already in memory -- child defs should just be a pointer to one that's already been loaded.
Well, that worked for Galaxy City. Generalizing from a single example, I'm sure it will work for every other geobin. ;-)
While I need to work on loading the object library before I make any pronouncements, it was also pretty quick to load and not nearly as memory intensive as I was afraid it would be. Of course, it's not holding any actual geometry at the moment, so it's too early to decide that "hey, this ain't so bad after all".
-
It's an optimization trick. Certain types of files, specifically geo, texture, and anim, all start with a 4-byte header size field. When pigg files are generated, those types of files have the header extracted and saved uncompressed into a special section of the pigg.
That section is loaded as a single chunk with the pigg file and always kept resident in memory. When the pigg library handles a read request for one of those files, if the offset+size is entirely within the cached header, it can be served without ever having to perform disk I/O.
This mostly affects game startup, when all of the in-memory indexes are built and every texture and geo header is scanned.
Ah right! Yeah, just looked at the meta for a file along with the actual file in a hex editor, makes sense now. Thanks! :)
-
Hmm whats allowedError for then, if the fading takes care of pop-in?
-
***edit***
I'm going to make this simple.
Codewalker, Back in the thread Technical Discussion of Bases (https://www.cohtitan.com/forum/index.php?topic=12087.msg211673#msg211673) from 2016, you said this:
When I say things like "the server just sends a map filename for the client to load", that's actually a gross simplification to keep conversations manageable. What actually happens is that the server first sends a list of files to load defs from - first all of the various object library files it depends on*, then the map itself. Then it sends the list of top-level refs that point to some of the defs just loaded.
What I need to know - How is the Paragon Chat "server" building the list of object library bin files that it sends to the client? I'm asking because processing the map geobin is leading me to all kinds of loose ends that APPEAR to be irreconcilable with either the virtual filesystem in the .piggs or the index in defnames.bin. It looks like an object path in the map geobin is a reference to a scene graph branch rather than some kind of absolute file path. How are you finding the absolute file paths to load from?
*********
-
Well, while we wait for Codewalker to return from whatever dimension he's currently exploring...
A small victory - Here's the shuttle model in a Panda3d window:
(https://i.imgur.com/TYpeZcM.png)
I don't want to oversell it as an "accomplishment". I pretty much just turned geodraw into an exporter that creates Panda .EGG files from CoH .GEO models. What I really need to do is create an API that can skip the intermediate step and load into an in-memory Panda model node straight from the .GEO file. I figure that first I'll see if I can get tricks.bin and the model textures figured out, then go for the "generating geometry on the fly part".
Though, as I've observed in the past, there's never going to be an Issue 24+ so there may be something to be said for just exporting ALL of the models and ditching the .PIGG and .GEO files.
***edit***
So, now it's textured, but I'm doing this all by hand right now. What I still don't see is how "x_H_Shuttle" in the .GEO file turns into "/texture_library/World/Space/Shuttle/H_Shuttle.dds". I expect this goes along with all of the defnames.bin entries that are numbers that don't have a corresponding path they index to. There's some part of the index that is either "missing" or that needs to be built up by the client.
-
Great work dude!
-
Great work dude!
Thanks. It's a way's away still, but I'm starting to think that approximating the city geometry for a collision/pathfinding model is a doable thing.
Back to questions for Codewalker - I've been using TonyV's Pigg Viewer Pro as my pigg viewer/extractor. It's pretty much the best of the bunch and the source code is still available in the Google Code archive, which means I can crib from his code if I ever decide to the "right" thing and implement a pigg-based virtual file system. (Panda3D has its own concept of virtual file system, so there may be some mileage in writing a glue lib that lets Panda "mount" the piggs onto its own VFS.)
PVPro tells me that all of those textures in the texture library are stored in the piggs as .texture files. However, all of the references in the databases refer to Targa .tga files and that's the one option that PVPro does NOT export them as. PVPro lists "original" as .dds. It's easy enough to work around that, but I want to make sure that I'm not missing something or losing some functionality by dealing with the textures as .dds or some other format compared to whatever raw format a .texture file is. Of course, it would make some sense if the Textparser turned the original .tga files into .dds files in the process of compiling them into the .pigg files.
-
@Spectre1989 - I learned something today from @Leandro that you'll want to know: When a model name has a suffix that begins with a double-underscore, that suffix is a trick from tricks.bin. That info helps me reconcile some of the issues I was having with model names not matching up with def names in the scene graph. For instance, a def named AP_crack_decal_01_sml where the corresponding model is named AP_crack_decal_01_sml__Graflg (and every other model in that .GEO likewise suffixed "__Graflg"). "Graflg" is a trick that sets level of detail and collision flags. In this case, it's a decal of a crack in a sidewalk, so it's flagged as "no collide" via the suffix.
So, if you're checking a def name or some other string against the model names in a .GEO, you need to do a substring match or trim the suffix or something similar. The actual model name might not be what you're expecting.
-
@Spectre1989 - I learned something today from @Leandro that you'll want to know: When a model name has a suffix that begins with a double-underscore, that suffix is a trick from tricks.bin. That info helps me reconcile some of the issues I was having with model names not matching up with def names in the scene graph. For instance, a def named AP_crack_decal_01_sml where the corresponding model is named AP_crack_decal_01_sml__Graflg (and every other model in that .GEO likewise suffixed "__Graflg"). "Graflg" is a trick that sets level of detail and collision flags. In this case, it's a decal of a crack in a sidewalk, so it's flagged as "no collide" via the suffix.
So, if you're checking a def name or some other string against the model names in a .GEO, you need to do a substring match or trim the suffix or something similar. The actual model name might not be what you're expecting.
Very useful! Thanks for sharing
-
I'll apologize again for appearing to use this thread as my running development blog but I'm treading in waters that nobody anywhere has documented, as far as I can tell, and when Codewalker eventually drops by again, I want him to be able to say right then, "Yeah, you're on the right track" or "No, you're completely off there."
Anyway - after spending some quality time with defnames.bin today and making a kaitai parser for it (the kaitai parsers aren't necessarily the most efficient but the web IDE is handy for visually analyzing data and structure of a file) I realized that defnames.bin is, in fact, organized just like one would expect it to be organized. The weird thing that's going on with the indexes is that some of them are having their high-bit set for some reason that is not intuitively obvious (to me, anyway). If I mask off those high bits then the indices work out just like they should. This is probably yet another shortcut by the Cryptic/Paragon Studio devs to incorporate flags into records without explicitly making space to store them, like the instance I mentioned earlier of trick names being tacked onto model names.
Anyway, I think when I pick this back up tomorrow that I may stand a reasonable chance of realizing an entire scene graph now that I have some clues to these mysteries. Famous last words, yeah.
-
What I need to know - How is the Paragon Chat "server" building the list of object library bin files that it sends to the client? I'm asking because processing the map geobin is leading me to all kinds of loose ends that APPEAR to be irreconcilable with either the virtual filesystem in the .piggs or the index in defnames.bin. It looks like an object path in the map geobin is a reference to a scene graph branch rather than some kind of absolute file path. How are you finding the absolute file paths to load from?
Loading defnames, loading geobins for library objects, and parsing geos. I'll try to drive in deeper where I can but there's a lot going in in the thread to try and keep up with.
-
PVPro tells me that all of those textures in the texture library are stored in the piggs as .texture files. However, all of the references in the databases refer to Targa .tga files and that's the one option that PVPro does NOT export them as. PVPro lists "original" as .dds. It's easy enough to work around that, but I want to make sure that I'm not missing something or losing some functionality by dealing with the textures as .dds or some other format compared to whatever raw format a .texture file is. Of course, it would make some sense if the Textparser turned the original .tga files into .dds files in the process of compiling them into the .pigg files.
Textparser doesn't have anything to do with textures. That's all handled completely separately.
Ignore the file extensions you see for textures. They're meaningless. Completely meaningless - the client strips them out and ignores them. You can put any junk you want there and it'll work fine. Assume you're looking for a '.texture' file with that base name.
Many of the files reference tga, but I've seen no evidence that tga was used as a file format during development for many years - it's just something devs threw in there because they copied it from elsewhere.
A texture file is an extra header that wraps around one of two formats. It can either be:
* A dds file. The whole thing, header and all. I've only seen uncompressed RGBA, DXT3 and DXT5 used; I don't think other compression algorithms are supported.
* A jpeg file. (this only works for loading screens, you can't use jpeg textures ingame)
Since the generation of dds used by COH only supports power of 2 sizes, the texture header includes the "real" size of the texture. Badge images are a great example of this in use.
I don't think targa is natively supported for textures. It might have been in older builds, but not anymore. Those would probably need to be converted to a lossless (RGBA) dds first and wrapped in a texture header.
-
So, now it's textured, but I'm doing this all by hand right now. What I still don't see is how "x_H_Shuttle" in the .GEO file turns into "/texture_library/World/Space/Shuttle/H_Shuttle.dds". I expect this goes along with all of the defnames.bin entries that are numbers that don't have a corresponding path they index to. There's some part of the index that is either "missing" or that needs to be built up by the client.
tricks.bin
-------- Texture 10421 --------
Name = x_H_Shuttle
FileName = TRICKS/SPACE/SHUTTLE.TXT
FileAge = 1152634253
u002 = x_H_Shuttle
Gloss = 1
Surface =
Fade1 = 0
Fade2 = 0
ScaleST0 =
X = 1
Y = 1
ScaleST1 =
X = 1
Y = 1
Base =
BumpMap = H_Shuttle_bump.tga
ObjFlags1 = 0
ObjFlags2 = 0
ObjFlags3 = 0
DF_ObjName =
Base1 = H_Shuttle.tga
Base1Scale =
X = 1
Y = 1
Base1Scroll =
X = 0
Y = 0
Base1ScrollType = 0
Base1Swappable = 0
Multiply1 = None
Multiply1Scale =
X = 1
Y = 1
Multiply1Scroll =
X = 0
Y = 0
Multiply1ScrollType = 0
Multiply1Swappable = 0
DualColor1 = none
DualColor1Scale =
X = 1
Y = 1
DualColor1Scroll =
X = 0
Y = 0
DualColor1ScrollType = 0
DualColor1Swappable = 0
AddGlow1 = none
AddGlow1Scale =
X = 1
Y = 1
AddGlow1Scroll =
X = 0
Y = 0
AddGlow1ScrollType = 0
AddGlow1Swappable = 0
BumpMap1 = H_Shuttle_bump.tga
BumpMap1Scale =
X = 1
Y = 1
BumpMap1Scroll =
X = 0
Y = 0
BumpMap1ScrollType = 0
BumpMap1Swappable = 0
Mask = H_Shuttle.tga
MaskScale =
X = 1
Y = 1
MaskScroll =
X = 0
Y = 0
MaskScrollType = 0
MaskSwappable = 0
Base2 = H_Shuttle.tga
Base2Scale =
X = 1
Y = 1
Base2Scroll =
X = 0
Y = 0
Base2ScrollType = 0
Base2Swappable = 0
Multiply2 = Shader_Reflect_chrome_09.tga
Multiply2Scale =
X = 1
Y = 1
Multiply2Scroll =
X = 0
Y = 0
Multiply2ScrollType = 0
Multiply2Swappable = 0
DualColor2 = none
DualColor2Scale =
X = 1
Y = 1
DualColor2Scroll =
X = 0
Y = 0
DualColor2ScrollType = 0
DualColor2Swappable = 0
BumpMap2 = H_Shuttle_bump.tga
BumpMap2Scale =
X = 1
Y = 1
BumpMap2Scroll =
X = 0
Y = 0
BumpMap2ScrollType = 0
BumpMap2Swappable = 0
CubeMap =
CubeMapScale =
X = 1
Y = 1
CubeMapScroll =
X = 0
Y = 0
CubeMapScrollType = 0
CubeMapSwappable = 0
Color3 =
Red = 0
Green = 0
Blue = 0
u300 = 1
Color4 =
Red = 0
Green = 0
Blue = 0
SpecularColor1 =
Red = 250
Green = 230
Blue = 255
SpecularColor2 =
Red = 200
Green = 200
Blue = 230
SpecularExponent1 = 20
SpecularExponent2 = 1
Reflectivity = 0
ReflectivityBase = -1
ReflectivityScale = -1
ReflectivityPower = -1
AlphaMask = 1
MaskWeight = 1
Multiply1Reflect = 0
Multiply2Reflect = 1
BaseAddGlow = 0
MinAddGlow = 1
MaxAddGlow = 120
AddGlowMat2 = 0
AddGlowTint = 0
ReflectionTint = 0
ReflectionDesaturate = 0
AlphaWater = 0
DiffuseScale =
Scale = 0.35
u901 = 1
u902 = 1
AmbientScale =
Scale = 2
u901 = 1
u902 = 1
u425 = 0
u426 = 0
u427 = 0
Fallback =
ScaleST0 =
X = 1
Y = 1
ScaleST1 =
X = 1
Y = 1
Base = H_Shuttle.tga
Blend =
BumpMap = H_Shuttle_bump.tga
BlendType = 0
UseFallback = 1
DiffuseScale =
Scale = 0.75
u901 = 1
u902 = 1
AmbientScale =
Scale = 5
u901 = 1
u902 = 1
u507 = 0
u508 = 0
u509 = 0
EDIT: To clarify, the texture namespace is flat -- each texture is identified by the unique file name and the subdirectory path doesn't matter.
When the trick references H_Shuttle.tga and H_Shuttle_bump.tga, those are looked up in an internal index built at client startup time. You have to scan all of the texture files - the name is included in the header rather than using the actual file name - and index the names to the file locations. That's why the pigg header cache is so critical for startup time.
As mentioned earlier, the extension is stripped off before building the index. So you could reference H_Shuttle.texture, H_Shuttle.dds, H_Shuttle.exe, or just plain H_Shuttle and you'd get the texture.
-
@Spectre1989 - I learned something today from @Leandro that you'll want to know: When a model name has a suffix that begins with a double-underscore, that suffix is a trick from tricks.bin. That info helps me reconcile some of the issues I was having with model names not matching up with def names in the scene graph. For instance, a def named AP_crack_decal_01_sml where the corresponding model is named AP_crack_decal_01_sml__Graflg (and every other model in that .GEO likewise suffixed "__Graflg"). "Graflg" is a trick that sets level of detail and collision flags. In this case, it's a decal of a crack in a sidewalk, so it's flagged as "no collide" via the suffix.
So, if you're checking a def name or some other string against the model names in a .GEO, you need to do a substring match or trim the suffix or something similar. The actual model name might not be what you're expecting.
Ok, so you already got object trick overrides (bindump erroneously calls the field LOD but it's more than that). Good.
-
The weird thing that's going on with the indexes is that some of them are having their high-bit set for some reason that is not intuitively obvious (to me, anyway). If I mask off those high bits then the indices work out just like they should. This is probably yet another shortcut by the Cryptic/Paragon Studio devs to incorporate flags into records without explicitly making space to store them, like the instance I mentioned earlier of trick names being tacked onto model names.
It's probably a union. A number of the in-memory structures use things that are obviously C unions.
The index is a 16-bit integer. The low bit of the next field is a flag that indicates if the defnames entry is a rootname -- a model in a geo file rather than a Group from a geobin.
-
That all makes me curious, though - Does the textparser still look for those original text files when the client launches? If someone was able to recreate a developer working directory, could you load your own versions of those text files?
It's complicated. I'll try to answer this as well as the follow-up question about Paragon Chat that I remember seeing, but I think got lost in your edits.
The client has a fully functional textparser. It can load any file from either text or binary form, *if* the functions are called with the right parameters. It can even *save* any file back into text form, though for things like powers.bin, a lot of the enums get screwed up because there's a giant shared table for them. You end up with nonsensical names for certain values in the Scale field, for instance.
You can't just go overriding files from the piggs with text files willy-nilly. The way it's supposed to work in development mode is this:
- No pigg files at all
- All of the data in text form
- If a bin file is missing or the text version is newer (per the filelist), data is loaded from the text version and persisted in the bin file
- Otherwise, load the bin
The release client doesn't have all of the code paths for this. Some of it is there, but is 'dead code' and unreachable in normal execution. The client is locked in 'production mode', likely by a compile-time flag or preprocessor definition, and there's no simple way to modify that.
Production mode looks like this (I'm simplifying a bit since the first two are handled by the VFS system and the last one is handled by textparser):
- If the bin file exists in data/, and isn't in the 'overrides prohibited' list, load it from there.
- If the bin file exists in a pigg, load it from there.
- Nothing else worked, if there's a text version in data/, might as well try and load it.
So the only way to get the production client to load text files -- aside from calling the functions directly with the aid of a debugger or the dynamic code rewriting engine that I use for ParagonChatClient -- is if no corresponding .bin file exists. That's why people are able to load custom maps in text format, because of course new maps have no .bin in data/geobin/.
That means if you want to load text files for bins that already exist, you have to:
- Extract *everything* from the pigg files into data/.
- DELETE all pigg files.
- Patch the client to remove the override list (see the forum thread about map editing)
- DELETE the bin file corresponding to what you want load from a text version.
I honestly don't remember if the automatic bin-generation code will work in that circumstance. It might be completely disabled in production mode, or it might recompile a bin which you have to delete if you want to change the text file(s) again. The code *exists* and I can call it manually (and have done that to generate some of the bin data for PChat), I just don't remember if it's in the normal execution path or not.
As for Paragon Chat's imitation textparser -- unfortunately none of this will work for that. The 'server' piece knows enough to read bin files, but for most of the parse formats it doesn't know the field names for the text form, or even what file to try to load it from. Costume files are the obvious exception since it needs to be able to read those in both text and binary format, but it can't even load maps in text format. I compiled the modified Pocket D to bin format in order to bundle it.
-
Also the SEGS map viewer may help you with the scene graph, though it's using I1 data.
https://github.com/Segs/Segs/tree/develop/Projects/CoX/Utilities/MapViewer
-
Thanks for all of that, Codewalker. I need to start following @Spectre1989's example and start documenting all of this stuff as I do it, for posterity.
Since you touched on the custom maps question - I ran an experiment this past weekend around that. I patched the client with your patch and installed @GuyPerfect's "Atlas Park Echo" mod into Paragon Chat's client/data folder along with a zones.cfg entry to reference it.
It actually worked, as far as displaying the map. Something was wrong with the collision detection, though. If my char wasn't flying, it fell through the geometry after a bit of bouncing around.
Maybe you can try it one of these days and see if there's a way to correct that in such a mod.
On a separate note, seeing that map got me right in the feels. I was a closed beta tester for CoH. Seeing old City Hall and Atlas and even, weirdly, the skylights on the corner a few blocks away, reminded me what it felt like to experience CoH for the first time. I hadn't felt that in a long time.
-
It actually worked, as far as displaying the map. Something was wrong with the collision detection, though. If my char wasn't flying, it fell through the geometry after a bit of bouncing around.
That's what I would expect. Classic desync.
The client is loading the map, so you're able to see it (top-level refs get instanced in the client's scene graph even if the server fails to send them; this is necessary for demo playback, etc). The server sends the map name from the cfg, but is unable to load the map itself, resulting in an empty scene.
As a result, the client tries to do motion prediction based on the geometry that it has, but on the server you're just falling through empty space. That causes rubber banding when the client gets entity updates and corrects its 'wrong' prediction.
In order to make it work, you'll need to put a copy of the geobin for the map into the appropriate path under ParagonChat\data as well as the client's data folder.
PChat itself gets around having to duplicate it (and having to bother to patch the client's foldercache) by creating a pigg file with what it wants to load and stacking that on top of the VFS on both the client and the server.
-
PChat itself gets around having to duplicate it (and having to bother to patch the client's foldercache) by creating a pigg file with what it wants to load and stacking that on top of the VFS on both the client and the server.
I'm not sure why, but that explanation of the desync strikes me as being a cool look inside of PChat.
So, let's say that I wrote a map editor. Does the above mean that if I exported the mod data (bin files, custom geos/textures, modified defnames.bin, etc...) as a .pigg file that was installed under the Paragon Chat /data folder, that the included overrides would work for both the "client" and the "server" side of things?
-
Currently, no. Only the ParagonChat.pigg file gets special treatment that way, and you can't really modify that because it'll get overwritten by PChat itself after updates.
You could fool it by creating a pigg and putting it under the *I24* pigg directory, which both PChat and the client load *.pigg from. You'd probably want to call it Z-something.pigg, so that it's last in alphabetical order. That would result in it being loaded after all of the regular piggs, and just before ParagonChat.pigg.
My long term plans involve creating an add-on system for Paragon Chat which would remove a lot of the guesswork. The idea is to package assets up in a zip file or similar format with a manifest, then PChat would take care of the details of getting the files for all enabled add-ons to the proper place for both the server & client components to use. Whether that means repacking them into a pigg, or using some other method to inject them into the client's VFS is a implementation detail that shouldn't matter to the addon developer.
-
Do we know what all of the extra data means in the .texture files?
The first part of the header seems to be pretty well mapped out. However, there's an additional section that appears after the filepath IF the texture is not a "flat" texture; that is, it's associated with geometry and not a loading screen or city map or some such.
The extra bit is 64-bytes long. The first 16-bytes seem to be flag fields. They typically look like this:
Flag-1: 16
Flag-2: 8 or 4
Flag-2: 8 or 4
Flag-3: 33777 or 33779
The rest is a block of binary data.
Strictly speaking, I get what I want by just ignoring it and extracting the .dds file into memory and attaching that to the model in the scene graph. I'm curious, though, if it's known what that 64 bytes of additional header means, and if any of it is information that I might potentially want to know about as far as creating a visual of the object the texture is attached to.
-
Cached copy of the lowest resolution mipmap. It gets used as the placeholder texture while a background thread loads the actual textures from disk.
Header Size
Width
Height
Texture Format
(33777 = 0x83F1 = GL_COMPRESSED_RGBA_S3TC_DXT1_EXT)
(33779 = 0x83F3 = GL_COMPRESSED_RGBA_S3TC_DXT3_EXT)
(https://www.khronos.org/registry/OpenGL/api/GLSC2/glsc2ext.h)
Padding
Texture Data
UI stuff and RGBA textures don't have mipmaps, so it's omitted for them.
-
Thanks, Codewalker. You da man!
-
The index is a 16-bit integer. The low bit of the next field is a flag that indicates if the defnames entry is a rootname -- a model in a geo file rather than a Group from a geobin.
It seems to be even weirder than that. I finally figured out why the indices seemed to be going pear-shaped on me.
Bindump prints the index as a Uint32. However, making it a Uint16 also didn't work out right.
What I finally figured out is that it works like this:
idx1 = Uint16
idx2 = Uint16
idx1 = idx1 & 0X7F (Mask off low bit of idx1 where idx1 is little-endian)
IndexVal = idx1 + idx2
It looks weird but it indexes to the correct objectlibrary value every time. There must be a reason for doing it this way instead of just making it a Uint32 in the first place, but I'll be hanged if I can imagine what that reason is.
-
Hey, Codewalker - A couple of times in the past, you've mentioned the clientsave ability in Icon without explicitly describing how to use it.
Care to elucidate? Custom map-making aside, it would give me the ability to generate a reference I could use to judge how well my geobin parser is working since, theoretically, my parser should be able to spit out a similar "mymap.txt" file. In fact, I've done that and loaded it up as a demo and in Icon, with mixed results, which either indicates that I need to finish getting all of the object library bins parsed or I've got an error someplace. Either way, it would be useful for me to be able to save out a text copy of the client's scene graph to compare to my parser's version of the same data.
-
I don't think you can in Icon, at least not without making some code changes and recompiling. Icon steals the coordinate box from the map editor to use as its own coordinate entry UI, and stomps on some of the map editor code in order to do it, so it would probably crash if you tried to enter it in Icon.
When I've done it before, it's been in demo playback mode. The steps go something like this:
- Acquire client accesslevel 9 or higher (10 is the effective max). I used a forged dev_access.key file to do this that I had lying around from back when the game was live -- one that I was careful not to use while connected to live servers, or at least never got caught using. 8)
Note to game developers: Don't use symmetric encryption to "protect" clientside features that you don't want players to have access to. The keys can be easily extracted from the code. Use something more like an EC or RSA signature. Still can be bypassed but at least isn't completely trivial.
Found a copy if you want to try it: https://www.dropbox.com/s/2e3h3pbn89rwivp/dev_access.key?dl=1
Another way to get accesslevel is by modifying memory with a debugging tool, but you have to find the right offset within the player structure.
- /can_edit 1 (in demo playback, open the console with tilde and use "can_edit 1" instead)
- /edit 1
Icon has an /accesslevel command that makes this kind of cheating easy, but you'd need to compile a modified version that doesn't touch the editor code.
I have some ideas for an Icon 2.0 built on top of the Paragon Chat engine that is kind of a COH development swiss army knife. For that I want to try to go a step further and get the actual editor working, but that's purely hypothetical at this point.
-
That is some damned cool beans, Codewalker. Thanks for sharing!
Nice to see that some of my guesses about things like Properties syntax were close to the mark. This will be a big help for checking that my parser is doing what it should.
-
I have some ideas for an Icon 2.0 built on top of the Paragon Chat engine that is kind of a COH development swiss army knife. For that I want to try to go a step further and get the actual editor working, but that's purely hypothetical at this point.
Given what I've seen of the editor UI now that I've had a taste of it, I think that would be a pretty cool addition to the PChat toolset. I assume, though, that it means actually running some sort of proxy map server; moreso, that is, than PChat already pretends to be.
Well, if there's anything I can do to help make it less hypothetical, let me know. I won't pretend that I'm "skilled" in this stuff yet, but I think I'm at the level that I could do some basic grunt work for you if you need it. (The day I get my own geo parser going and get a home-brew map viewer online, I'll give myself my first gold star, heh.)
-
I assume, though, that it means actually running some sort of proxy map server; moreso, that is, than PChat already pretends to be.
PChat is a lot closer to a real mapserver than many people assume. It kind of has to be -- the mapserver protocol is so tightly bound to implementation details shared by both client and server that it's almost impossible to speak the protocol in any useful way without actually doing things the same way that the server did.
It's for that reason that if I was going to try to build a map editor, or something that does anything beyond what Icon is capable of, I'd probably use PChat as a starting point for it. The core is modular enough that it doesn't have to use XMPP as a sync protocol (see also: offline mode).
-
PChat is a lot closer to a real mapserver than many people assume.
I think it's less that people don't realize and more that we all wear self-imposed blinders because otherwise we'd all by asking "Why aren't Codewalker/Leandro/anybody-at-all taking the next step and making an emulator?" ;-)
-
Codewalker, upstream when we talked about GuyPerfect's Atlas Park Echo mod, you said:
In order to make it work, you'll need to put a copy of the geobin for the map into the appropriate path under ParagonChat\data as well as the client's data folder.
This doesn't work for me. I copy Guy's mod verbatim into ParagonChat\data, identical to how it's installed in ParagonChat\client\data. If you've got a few minutes sometime, could you verify that it works for you?
-
Apropos to nothing - today's Girl Genius comic (the last few panels; the context of the comic is an attempt to cure a lead character of mind control embedded in his brain) illustrates how I frequently feel when I imagine all of the moving parts involved with just wanting to tell my bot "Run away from me and don't get stuck on the terrain."
(https://images.weserv.nl/?url=www.girlgeniusonline.com%2Fggmain%2Fstrips%2Fggmain20181017.jpg)
-
I'm still chugging along with my map viewer, progress has been slow as I don't have a great deal of time at the moment. I got to the point of loading a map geobin, reading the defs, then attempting to traverse the scene graph via the refs. When a group name contains a slash then it's either a def in another geobin, or a model in a geo. The first problem there is which one is it? I just go looking for a geobin, and if there is one see if it has a def which matches the name. Then I ran into the problem that the group name containing the slash doesn't give the name of the actual geobin file, just a folder, and some folders contain multiple geobins. Now going back through this thread it looks like defnames.bin may be what I'm looking for, I will look at that next.
-
Where's the schema for defnames.bin? defs.xml doesn't look right for what I'm seeing in the file...
-
I think I figured it out by looking at the file, and some of the stuff you guys said before. It looks like this:
int32 file_count;
string file_names[file_count];
int32 def_count;
Def defs[def_count];
struct Def
{
string name;
uint16 is_geo;
uint16 file_index;
}
A couple of things I don't understand - even with a def which is NOT a geo, but is a geobin, the file in the file name array still has the .geo extension, but in some cases at least no such geo exists, only the bin in the geobin folder.
When the def is a model from a geo, there's also a bin in the geobin folder, which will contain a def with the same name. What's that for? I'm unsure at what point a group in a def is actually referencing a model from a geo.
-
With the holidays, my workload has increased and my free time has decreased, so I've been taking a break; long enough that I'm starting to forget some things and am going to need a refresher soon. :-/ That's one of the reasons that I so frequently detail things I learn here, so that I can find them in the correct context three months down the road.
Anyway - I think you're finding the same kind of thing that initially confused me, when I ended up writing this:
What I finally figured out is that it works like this:
idx1 = Uint16
idx2 = Uint16
idx1 = idx1 & 0X7F (Mask off low bit of idx1 where idx1 is little-endian)
IndexVal = idx1 + idx2
My experience is that what you've labeled "is_geo" isn't JUST a boolean flag. There's offset data in there also, even though in many cases it evaluates to zero and makes "is_geo" APPEAR to be just a boolean flag. If you set the flag bit to zero and then take what remains in "is_geo" and add it "file_index", then you should get the ACTUAL file_index for the thing you're trying to find. I think this is another example data being stored as a C language union structure, but I don't pretend to be the reverse engineer that Codewalker is.
Anyway - there seems to be a lot of "obsolete" data in the object library, and some of it also seems to be incomplete, which is probably not unexpected considering that we're working with a beta build of an issue that was never officially completed and released.
Also, things will get more interesting once you start to try correlating tricks to geos. :-o
-
I couldn't find any "Def"s in that array where the "is_geo" uint16 was greater than 1 though?
-
Have you dumped defnames.bin with bindump?
It has two sections. The first is a table of geo files. The second is the table of defnames that index into that table of geo files. Anything that ends in .geo is NOT a defname. It's the thing that a defname in the second table resolves to. Your code snippet looks like you're treating the whole defnames.bin file as a single homogenous table.
-
So the first "table" is just a list of file names
int32 file_count;
string file_names[file_count];
And the second table is the defnames.
int32 def_count;
Def defs[def_count];
struct Def
{
string name;
uint16 is_geo;
uint16 file_index;
}
Which row are you seeing where the first uint16 is > 1?
-
/headsmack.
Never mind me, this is what happens when you work from bindump output instead of from the actual file data and make assumptions based on that. I'm the one who's confused here, not you. (The erroneous original assumption was that the data in defnames.bin was all uint32, just like in the geobins. That's not the case, though. Then I compounded it by assuming that your intial file_names array was the array of source file names in the bin file header.)
So, given that, maybe it will help if you give some examples of the entries that don't appear to be working out correctly for you. It sounds like you're talking about objects that are made up of multiple geos but after the above headsmacking, it would be better to know exactly what you're seeing before assuming anything.
-
A couple of things I don't understand - even with a def which is NOT a geo, but is a geobin, the file in the file name array still has the .geo extension, but in some cases at least no such geo exists, only the bin in the geobin folder.
I'm assuming here that we're talking about something like this: object_library/Omni/EncounterSpawns/P_City_00_03/P_City_00_03.geo. Where, if you follow it into the actual object_library folder instead of the geobin folder, there isn't any P_City_00_03 folder and thus no geos below that non-existent folder.
This goes back to the discussion of textures upstream, about how there are all these definitions for .TGA files when there aren't many, or any, actual Targa files in the data. The textparser that compiles the raw specification files into the bin files plays fast and loose with file extensions and frequently ignores them entirely.
The important thing from the game client's standpoint is that there IS a P_City_00_03 folder in the geobins, with an associated P_City_00_03.bin and P_City_00_03.bounds underneath it. It doesn't care that P_City_00_03.geo doesn't exist because it doesn't actually use that file index result to tell it what model to use because the is_geo flag is false. Instead, it opens P_City_00_03.bin and adds it to the scene graph as an object, and then adds all of the appropriate models below that object. In this particular case, it ends up loading spawn markers of the sort you see when you've got developer mode turned on in Icon or Pchat. Essentially, they're always there in the zone but usually they are invisible and are marked as no-collision.
In short - it's marked with a .geo because everything in the defnames filename index is marked as a .geo; even things that don't need to be. The "is_geo" flag differentiates real models from objects made up of other models.
When the def is a model from a geo, there's also a bin in the geobin folder, which will contain a def with the same name. What's that for? I'm unsure at what point a group in a def is actually referencing a model from a geo.
As near as I can tell, when a leaf def has its "object" attribute set to the name of the object, that means it's referencing a model inside of the geo that corresponds to the same name and path as the bin file that holds the leaf def. That's why, for instance, there's no P_City_00_03/P_City_00_03.geo. All of the models that might conceivably appear in that geo file already exist in the base city file geos. The bin files "point" to the scene graph segments that ultimately contain them. The client doesn't give a hang about how the label reads in the defnames index as long as it can find whatever it needs to find.
-
This goes back to the discussion of textures upstream, about how there are all these definitions for .TGA files when there aren't many, or any, actual Targa files in the data. The textparser that compiles the raw specification files into the bin files plays fast and loose with file extensions and frequently ignores them entirely.
Ah I see, so the file extensions are largely useless, ok I can cope with that.
As near as I can tell, when a leaf def has its "object" attribute set to the name of the object, that means it's referencing a model inside of the geo that corresponds to the same name and path as the bin file that holds the leaf def. That's why, for instance, there's no P_City_00_03/P_City_00_03.geo. All of the models that might conceivably appear in that geo file already exist in the base city file geos. The bin files "point" to the scene graph segments that ultimately contain them. The client doesn't give a hang about how the label reads in the defnames index as long as it can find whatever it needs to find.
Interesting, I hope Codewalker can confirm/deny that this is what the "object" field is for.
If the "object" field means "model inside geo corresponding to same name/path as this geobin", I wonder if that means that all instances of models MUST come from def instances from the geo's corresponding geobin, or whether that's not necessarily the case.
E.g. if I have my map city.bin, scenery.geo and corresponding geobin scenery.bin. I wonder if city.bin can directly include models from scenery.geo, or whether it would have to have instances of defs from scenery.bin which then contain the models from scenery.geo.
-
It looks like you're right. I knocked up a bin->json converter to help me follow one def through this crazy russian doll situation all the way to a model in a geo.
I picked a random geobin line from my defnames text dump:
geobin:1578:_ES_Des_Loiter_D9_P_City_00_02
// that row is...
1578:object_library/Omni/EncounterSpawns/P_City_00_02/P_City_00_02.geo
So then looking at object_library/Omni/EncounterSpawns/P_City_00_02/P_City_00_02.bin, I look for a def called "_ES_Des_Loiter_D9_P_City_00_02", I find this:
"name": "_ES_Des_Loiter_D9_P_City_00_02",
"type": "",
"obj": "",
"groups": [
{
"name": "object_library\/Omni\/EncounterSpawns\/BaseTypes\/ES_BrawlShipRec_BaseType",
"position": {
"x": 0,
"y": 0,
"z": 0
},
"rotation": {
"x": 0,
"y": 0,
"z": 0
}
}
]
(Don't mind the screwed paths) So take the end of the path "ES_BrawlShipRec_BaseType" and search in defnames, I find this:
geobin:1486:ES_BrawlShipRec_BaseType
// that row is
1486:object_library/Omni/EncounterSpawns/BaseTypes/BaseTypes.geo
This is another geobin, so I look in object_library/Omni/EncounterSpawns/BaseTypes/BaseTypes.bin for "ES_BrawlShipRec_BaseType":
"name": "ES_BrawlShipRec_BaseType",
"type": "",
"obj": "",
"groups": [
{
"name": "Omni\/EncounterSpawns\/Encounter_V_40",
"position": {
"x": 0,
"y": 0,
"z": 0
},
"rotation": {
"x": 0,
"y": 0.872665,
"z": 0
}
},
{
"name": "Omni\/EncounterSpawns\/Encounter_E_01",
"position": {
"x": 11.5,
"y": 0,
"z": -8
},
"rotation": {
"x": 0,
"y": 1.919862,
"z": 0
}
},
// etc
I thought I'd look at "Omni\/EncounterSpawns\/Encounter_E_01", so I search defnames for Encounter_E_01:
geobin:1537:Encounter_E_01
// which is this row
1537:object_library/Omni/EncounterSpawns/EncounterSpawns.geo
So I look in object_library/Omni/EncounterSpawns/EncounterSpawns.bin for Encounter_E_01:
"name": "Encounter_E_01",
"type": "",
"obj": "",
"groups": [
{
"name": "_encounter_E_01",
"position": {
"x": 0,
"y": 0,
"z": 0
},
"rotation": {
"x": 0,
"y": 0,
"z": 0
}
}
]
This doesn't have a path, so I look in the same geobin for a def with that name:
"name": "_encounter_E_01",
"type": "",
"obj": "_encounter_E_01",
"groups": [ ]
The obj field is set, so look in defnames for "_encounter_E_01":
geo:1537:_encounter_E_01
// that row is
1537:object_library/Omni/EncounterSpawns/EncounterSpawns.geo
So finally this is one which is a geo rather than geobin, load up "object_library/Omni/EncounterSpawns/EncounterSpawns.geo" in geodraw, and I find this:
13 _encounter_E_01__ENEMY
Hooray!
Now the thing I'm wondering is - can a def have the "obj" field set (therefore having a model), as well as having child groups which also have models etc? Should be easy enough to find out.
-
E.g. if I have my map city.bin, scenery.geo and corresponding geobin scenery.bin. I wonder if city.bin can directly include models from scenery.geo, or whether it would have to have instances of defs from scenery.bin which then contain the models from scenery.geo.
Defnames is the ultimate arbiter. You should be able to reference any model "directly" as long as you have a parent group def that contains its attributes (loc, pyr, etc...) and the model def has an entry in defnames that indexes to the geo file that contains the model. The point of the object library is to avoid doing that sort of thing, of course, but it doesn't confine you to only doing things that way. It's more that the scene graph organization of the object library naturally conforms to the structure of the object library itself.
Now the thing I'm wondering is - can a def have the "obj" field set (therefore having a model), as well as having child groups which also have models etc? Should be easy enough to find out.
Probably? There's nothing preventing it, but you'd be breaking the spec, so to speak.
I've played a bit with the built-in scene editor and the concept behind it seems to be that a "group" is a container and everything else represents "things" that are being contained and defined. You don't typically find "things" with attributes added or with children depending from them. "Groups" provide the structure and the instructions for placing or otherwise modifying the "things". I think the behavior of the client if you fed it an "object" that also pretended to be a "group" with children would be considered "undefined".
-
Probably? There's nothing preventing it, but you'd be breaking the spec, so to speak.
I don't mean it's something I want to do, I wonder if any of the game geobins do it. I think I'm going to have to write a few tests to load every geobin in the game and test our theories and assumptions. I'll report back any findings :)
-
Ok so as there turns out, there's a fair few defs which have both an obj (so therefore a model?) as well as groups. There were quite a few, so if you're interested there's a pastebin here (https://pastebin.com/1BgmCCDM).
On the question of whether a geobin can directly include a model from a geo which also has a corresponding geobin (rather than including a def from the corresponding geobin) - yes, all the time.
One last interesting finding was that there's a single def in all of the geobins which has an obj field which doesn't actually exist in defnames.
-
Another thing - there are some (< 100) external geobin defs which aren't in defnames.
By "external geobin def" I mean I have a group in a def with a name like "path/to/something/defname". If I chop off the "defname" bit and search in defnames.bin, I'll usually find an entry there with is_geo flag false (makes sense, it's a geobin), and the file name will be something like "path/to/something/something.bin".
So, like I said, there are a bunch that just aren't in there. However, given "path/to/something/defname", I can just look in "path/to/something/something.bin", which seems to work. It's just that sometimes a geobin folder contains multiple bin files. I dunno, feels like these are broken references which are actual bad data, so I'm not sure whether to bother attempting to fix them like this!
Anywho, onwards..
-
Now the thing I'm wondering is - can a def have the "obj" field set (therefore having a model), as well as having child groups which also have models etc? Should be easy enough to find out.
Yes. The primary use case for doing this is 'trays' - indoor sections that are only rendered while you're inside them. Conversely, while you're inside one, the outside world is not rendered.
The top-level group will use an Obj that acts as the container geometry. For things like offices and sewers, this is often a single mesh of the ceiling, floor, walls, etc. Other styles like Arachnos bases use a solid black box with holes cut out for door connections.
The same group with the Obj also has child groups with smaller interior details that turn it from a generic room with blank walls into a finished room. Things like light fixtures, pictures on the wall, desks, machinery, etc. The child groups are only rendered while you're inside the "Obj" (or a connected tray but that gets more complicated).
-
Yes. The primary use case for doing this is 'trays' - indoor sections that are only rendered while you're inside them. Conversely, while you're inside one, the outside world is not rendered.
The top-level group will use an Obj that acts as the container geometry. For things like offices and sewers, this is often a single mesh of the ceiling, floor, walls, etc. Other styles like Arachnos bases use a solid black box with holes cut out for door connections.
The same group with the Obj also has child groups with smaller interior details that turn it from a generic room with blank walls into a finished room. Things like light fixtures, pictures on the wall, desks, machinery, etc. The child groups are only rendered while you're inside the "Obj" (or a connected tray but that gets more complicated).
Interesting, thanks!
Another thing - there are some (< 100) external geobin defs which aren't in defnames.
Here's the full list of those:
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_1&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_2&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_3&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_4&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_5&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_6&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128022:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_14&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_1&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_2&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_3&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_4&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_5&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_6&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128041:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_14&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_1&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_2&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_3&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_4&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_5&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_6&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128035:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_14&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_1&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_2&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_3&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_4&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_5&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_6&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128025:V_COV/Villian_Lairs/Arachnos/Elements/Interior/elevators/Arach_Gen_Elevator_01_14&
geobin/maps/City_Zones/V_City_05_01/V_City_05_01.bin:grp128020:V_COV/City_Zones/Grandville/Lobby_Tray/arach_rm_lobbyhospital_01_1&
geobin/maps/Missions/Unique/DoorMissions/Outcasts_Lair.bin:grp28:Omni/EncounterSpawns/Base_Missions/ES_MissionLayout_2&
geobin/maps/Missions/Unique/Flashback/FB_CoT_SAM/FB_CoT_SAM.bin:grp127:Omni/EncounterSpawns/Base_Missions/ES_MissionLayout_4&
geobin/maps/Missions/Unique/StatesmanTaskForce/IOM_CapauVictorie/IOM_CapauVictorie.bin:grp91809:Villain_Lairs/Tech/trays/halls/tek1_h_visstr_02_11&
geobin/maps/Missions/Unique/StatesmanTaskForce/IOM_CapauVictorie/IOM_CapauVictorie.bin:grp92242:Villain_Lairs/Tech/trays/halls/tek1_h_elv_1B_39&
geobin/maps/Missions/Unique/StatesmanTaskForce/IOM_CapauVictorie/IOM_CapauVictorie.bin:grp92509:Villain_Lairs/Tech/trays/halls/tek1_h_elv_2B_38&
geobin/maps/Missions/Unique/StatesmanTaskForce/IOM_CapauVictorie/IOM_CapauVictorie.bin:grp92791:Villain_Lairs/Tech/trays/halls/tek1_h_elv_3B_47&
geobin/maps/Missions/V_Mayhem/V_Mayhem_SteelCanyon_01.bin:grp76375:Streets/road_2lane/2_bend_12&
geobin/maps/Missions/V_Mayhem/V_Mayhem_SteelCanyon_01.bin:grp76375:Streets/road_2lane/2_bend_14&
geobin/maps/Missions/V_Mayhem/V_Mayhem_SteelCanyon_01.bin:grp76375:Streets/road_2lane/2_bend_15&
geobin/maps/Missions/V_Mayhem/V_Mayhem_SteelCanyon_01.bin:grp76375:Streets/road_2lane/2_bend_16&
geobin/maps/Missions/V_Office/V_Office_30/V_Office_30_Layout_08.bin:grp2:Villain_Lairs/office01/trays/rooms/dead_end/ofc1_mr_end_06_59&
geobin/object_library/Buildings/Style/deco/Deco2/Deco2.bin:decotestmaqt_C:object_library/Buildings/Style/deco/Deco1/Deco1_88&
geobin/object_library/City_Zones/Praetoria/Imperial/Imperial.bin:TempImperialCityLayout_36&:City_Zones/Praetoria/Buildings/Office/Modern_Skyscrapers/Bld_OFC_Modern_Skyscraper_B/_Bld_OFC_Modern_Skyscraper_B_07
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_L1_3_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_1&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_L1_3_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_2&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_L1_3_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_3&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_L1_3_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_4&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_Rikti_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_1&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_Rikti_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_2&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_Rikti_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_3&
geobin/object_library/Omni/EncounterSpawns/City_01_01/City_01_01.bin:_ES_Rikti_Ambush_City_01_01:object_library/Omni/EncounterSpawns/BaseTypes/ES_Ambush_BaseType_4&
geobin/object_library/temp/Test_smaller_doors/Test_smaller_doors.bin:Crappity_test3:object_library/Omni/NAVCMBT
geobin/object_library/temp/Test_smaller_doors/Test_smaller_doors.bin:Crappity_test3:object_library/Omni/NAVCMBT
geobin/object_library/temp/Test_smaller_doors/Test_smaller_doors.bin:Crappity_test3:object_library/Omni/NAVCMBT
geobin/object_library/temp/Test_smaller_doors/Test_smaller_doors.bin:Crappity_test3:object_library/Omni/NAVCMBT
-
I suppose that in a beta client, you're going to see things like "Crappity_test3", LOL.
I'd suspect that "beta test client" is a good enough explanation for most of these anomalies.
Also, thanks for the explanation of "tray", Codewalker. I'd seen to that in the past and scratched my head over what it was supposed to mean, since it clearly didn't mean the same thing as power tray or enhancement tray.
It'll be a couple of months probably before I get back to working with this stuff, but I'm keen to see where you're going, Spectre1989. I look forward to the updates.
-
OK so another question - is a geobin def obj field the only way that models in geos are instanced?
This file object_library/City_Zones/Atlas_Park_makeover/Decals/Crack/Crack.bin contains a single def with 8 groups. The names of those groups correspond to entries in defnames.bin, whereas what I had been doing was to only look in defnames if the group name was a path. Should I always be looking up group names in defnames?
-
You should always look in defnames; don't go searching for files or try to be smart about it based on the formatting of the reference.
Defnames is the master index for object library defs (available in all contexts) and should be considered authoritative.
-
Thanks CW ;)
-
Any idea why some references are just "defname", whereas some are "path/to/something/defname"? Should I be treating them the same? In the latter case I end up just chopping off the bit after the last slash and searching for that, which works, but just seems a bit odd.
-
If they're in defnames (which everything in the object library should be), just ignore everything before the last /. Doesn't matter.
The only place using the full path would make a difference is if a map referenced something that's *not* in the object library, but rather another map file. That doesn't (shouldn't?) happen in production maps. When working with the map editor, layers that can be toggled on and off are usually saved in separate files, so it could occur there. They're all consolidated back into a single file when generating bins.
As for why sometimes the full path is in maps and sometimes just the short name, it probably depends on what method the designer used to insert the group -- picked it from the library catalog on the top-left, manually entered the name, imported from a file they were working on. I'm not entirely sure what all different development tools were used, but have seen enough to know they're not particularly consistent.
-
I suppose that in a beta client, you're going to see things like "Crappity_test3", LOL.
Nah, stuff like that has been in the release client for years. I can practically guarantee that 99% of the weird stuff you find in the I24 beta was in the I23 client, and probably a good many more before it.
Having done a lot of delta comparisons, the differences between I23 release and I24 beta client are almost all related to specific I24 content.
-
You should always look in defnames; don't go searching for files or try to be smart about it based on the formatting of the reference.
Defnames is the master index for object library defs (available in all contexts) and should be considered authoritative.
I must be not getting something. I'm loading City_01_01.bin, one of the refs is grp_GC_Shutdown_Atmo (I assume for Refs I DO search the geobin rather than defnames), the first group of this def is grp_Media_Intvw, which is not present in defnames :(
-
I must be not getting something. I'm loading City_01_01.bin, one of the refs is grp_GC_Shutdown_Atmo (I assume for Refs I DO search the geobin rather than defnames), the first group of this def is grp_Media_Intvw, which is not present in defnames :(
;tldr - If a def name is a model, defnames tells you what .GEO contains the model. If the def does not have an entry in defnames, it's not geometry. It's something else.
You're not finding it in defnames because grp_Media_Intvw is not a model. It's an object in the object library that is a group of "atmosphere" NPC's. It's a group of models; though, the models in question are the dev-view-only spawn point models rather than the NPC's themselves. The NPC's are entities, which is something else entirely.
Maybe Codewalker's advice needs to be qualified to say that anytime you need to find the .GEO that holds a model, you look it up in defnames rather than futz around with parsing the path leading to the def. That's what defnames is: a mapping of def names to model file names. You go to defnames when you've reached a leaf node (or, I suppose, when you have a tray, given what we've recently learned about those from Codewalker) and that's how you find the .GEO for that model.
Typically, this would be indicated by an "object" def but as we've seen this isn't always necessarily the case. So, when you get a path, the first thing you do is look for it in the object library. If there's an object library bin for that path, you add it to the scene graph and follow that new patch to its conclusion, adding whatever other pieces are indicated. Eventually, you'll reach one or more leaves with no children; at that point, you've got a model name (assuming that it's not some other kind of def, like a sound file) and you use defnames to find the .GEO for the model.
Then the real fun starts when you try to figure out how to load the model and its texture. :)
What I've taken from Codewalker's explanations in the past is that in an ideal world, the object library would be completely loaded into memory and every piece of it would always be available permanently. In the real world, that would use an unrealistic amount of RAM and instead we demand-load the pieces that are going to be used by the current scene as they're encountered in the scene graph. Presumably, we also mark the loaded models so that when the scene changes, we can do a final pass on everything in memory and unload whatever is still in RAM that isn't currently being rendered. I've assumed that's why models get two defs - an object def and an apparently "null" def. My guess was that the "null" def is used to indicate what should be treated as cache and what should be unloaded after the scene is completely loaded but not yet rendered.
From the standpoint of loading the scene graph, you load the map bin, and when you encounter a path, you load that object from the object library, then examine the nodes in that new bin you just loaded. If there are more object library defs (say, it's a building composed of many parts that are all separate object library bits), you load them as well and process them recursively. If you encounter a leaf node and/or an "object" def, you treat it as a model (unless there's a reason to treat it as something else, because one of the other values of the def indicates that it's a sound file or LOD values or whatever).
-
(Quote edited to combine relevant comments)
;tldr - If a def name is a model, defnames tells you what .GEO contains the model. If the def does not have an entry in defnames, it's not geometry. It's something else.
...
So, when you get a path, the first thing you do is look for it in the object library. If there's an object library bin for that path, you add it to the scene graph and follow that new patch to its conclusion, adding whatever other pieces are indicated.
What do you mean? There's a ton of stuff in denames which are not geos, they're geobins. If the name is a path, you can't just use the path to find the geobin in the object library, because some folders contain multiple bins - that's why you have to go through denames.
You're not finding it in defnames because grp_Media_Intvw is not a model. It's an object in the object library that is a group of "atmosphere" NPC's.
It's not, it's a def in City_01_01.bin.
Based on what I'm seeing it looks like the flow for parsing groups is:
* if you get a path, chop of the end of the path and look in defnames for it, which also tells you if it's another geobin, or a model
* if it's not a path, first try to find it in the current geobin
* if it's not found in the current geobin, try defnames
PS: Everything I said could be wrong, this is just how I thought it worked?
-
You said, grp_Media_Intvw , right?
That's a def that has a child: grp134017. It also has some properties related to spawning parameters for the group.
The def itself is a child of grp_GC_Shutdown_Atmo, which is essentially the top of one branch of the overall scene graph tree; that is, grp_GC_Shutdown_Atmo is a direct child of the ref of the same name.
grp_Media_Intvw is just a node in the scene graph. It doesn't need a defnames entry because it doesn't connect to anything but its own child and parent. Did you mean a different def?
If you follow grp134017 through its children and grandchildren to its terminus, you end up with the spawn markers, which is where you get into the object library. I should have been more specific about what I meant in regards to that.
-
Right, I know it's just in the same scene, what I'm trying to figure out is when I should be searching in defnames. What CW said sounded to me like I should always be searching in defnames. Whereas it looks to me like I should first look in the current geobin defs, then I look in defnames if it wasn't found.
-
You use defnames when you need to reference something that isn't already loaded. Otherwise, you can just deal with whatever's already loaded into memory. Once you load something, future references to that thing (since something like a building could be referenced multiple times) just hook into the in-memory copy of it.
Codewalker was mainly emphasizing that defnames is the definitive reference for finding content, over and above any shenanigans with parsing paths and file systems.
-
I must be not getting something. I'm loading City_01_01.bin, one of the refs is grp_GC_Shutdown_Atmo (I assume for Refs I DO search the geobin rather than defnames), the first group of this def is grp_Media_Intvw, which is not present in defnames :(
You don't need to load grp_Media_Intvw, it's already defined earlier in the map; third one in the file actually. There's no reason to look it up when it's already present.
defnames is the index for loading. If something references a group that doesn't exist, that's when you fall back to the object library in order to try to make it exist -- and defnames will tell you exactly how to load it.
-
You don't need to load grp_Media_Intvw, it's already defined earlier in the map; third one in the file actually. There's no reason to look it up when it's already present.
defnames is the index for loading. If something references a group that doesn't exist, that's when you fall back to the object library in order to try to make it exist -- and defnames will tell you exactly how to load it.
Ok cool, I just misunderstood you. First try the current geobin, then try defnames, that's what worked last night
-
Finally I've got to the point where I can view a map in wireframe, here's a video https://youtu.be/D499ys9Gmtk (https://youtu.be/D499ys9Gmtk)
-
Nice.
-
Well done!
-
[quote
I have some ideas for an Icon 2.0 built on top of the Paragon Chat engine that is kind of a COH development swiss army knife. For that I want to try to go a step further and get the actual editor working, but that's purely hypothetical at this point.
[/quote]
A 'CoH Development Swiss Army Knife.' That sounds cool. It even has a cool name.
If you can get the editor working as part of an Icon 2.0 built on top of the Paragon Chat engine...
...*tries to imagine that. :o
Azrael.
-
Aaaand I've updated the documentation (https://github.com/spectre1989/thor/tree/master/documentation) to include the Geobin file format.