Author Topic: Technical side discussion  (Read 164659 times)

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #300 on: August 05, 2015, 09:02:33 PM »
For example its interesting to me that if you change an entity's location dramatically, the original devs decided the entity should move there at supersonic speed rather than just teleport there.  I know that it happens, but not what the thinking was.  Maybe they just thought it was cool.

The interesting thing is that it's intangible while it's doing it. I first encountered that effect when I was launching my bot across Atlas Park and then recalling it to its "origin". I ended up pretty much shrugging and saying, "Well, okay then." Despite it being a visual artifact of the bot's motion, there was *not* any of the collision that ought to have happened if the bot was really moving along that line. ErrBot had to move from a lower elevation to a higher elevation across a hundred yards or so of ground that included fences, sidewalks, walls and all kinds of obstacles that it ignored entirely.

Basically, the bot did teleport but it still left this visual effect connecting its start and end points. I guess it means that if you're going to literally jump from point A to point B all at once that you need to turn invisible first unless you want to appear to be moving so fast that you vibrate right through any solid objects in your way.


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #301 on: August 06, 2015, 02:00:36 AM »
Obviously, the bot's defiant belief its intangible combined with a client side predictor that doesn't believe that combine to create some very odd effects which are not obviously emergent.  Because the bot is visibly "rolling" over the wall in a weird sort of almost-hop over, and yet its only being told to walk straight through the wall.

I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #302 on: August 06, 2015, 02:44:36 AM »
For example its interesting to me that if you change an entity's location dramatically, the original devs decided the entity should move there at supersonic speed rather than just teleport there.  I know that it happens, but not what the thinking was.  Maybe they just thought it was cool.

That's the client doing that. All position updates are subject to interpolation by the client. It's not as obvious with small updates, but it does happen, because the server doesn't send 30 updates a second for every entity. Even clientside prediction is limited to 30 "frames" per second, and many systems can render faster than that. With larger changes, the interpolation gets spread out over multiple frames, though I haven't sat down to hammer out the exact details.

In the network protocol, there's a bit you have to explicitly set when sending the update to indicate that the client should not interpolate, but rather instantly move the entity.

Paragon Chat currently only sets that bit in a few circumstances, mostly when first creating the entity and moving it from 0,0,0 once the costume arrives and the first update is received. I plan to add an attribute to the update stanza to indicate other circumstances when this is needed; such as when taking an intra-zone door (it already sets it for the local player but doesn't communicate that over XMPP).

Speaking of which, I've been unable to reproduce the issue you mentioned where sending presence again causes it to think the initial position hasn't been received. I've been testing with PC doing costume changes and could not recreate it, even when being careful to hold still so that a <u> stanza is not sent right away.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #303 on: August 06, 2015, 02:49:48 AM »
Speaking of which, I've been unable to reproduce the issue you mentioned where sending presence again causes it to think the initial position hasn't been received. I've been testing with PC doing costume changes and could not recreate it, even when being careful to hold still so that a <u> stanza is not sent right away.

Hmm, I will try to see if I can come up with a reproducible way to generate that issue.  I might have to deliberately break the client to do it since I refactored that code to eliminate the possibility in the first place.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #304 on: August 06, 2015, 02:51:25 AM »
I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?

It seems unlikely because that would cause dramatically amusing errors in motion at other times but I can't say for certain such a bug isn't in there somewhere.  I will retry those tests making sure the bot isn't attempting to follow me while I am not in the same xz plane, just to see what happens.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #305 on: August 06, 2015, 03:07:53 AM »
Also, there's an important distinction that I think should be noted. When I'm talking about clientside prediction, I'm talking about the player. The COH client does motion prediction only for its own player. It never speculatively predicts the location of other players, NPCs, etc. Those are effectively physicsless and are not even processed except to draw them. It relies on the mapserver to send regular updates that it can interpolate between, but introduces some intentional time lag so that those updates seem to be happening in real-time, even when they're not being continuously received.

That's probably the cause of the latency that you're seeing even when the XMPP server is local. The movement 'buffer' as it were seems to be integrated into the client. I haven't yet investigated all the murky corners of the protocol to see if there's any way to disable it within the protocol itself assuming a near-infinite speed connection.

Now, Paragon Chat does do speculative prediction as well as apply collision mechanics to the simulated players (mostly so falling doesn't go through the ground), which is why you often see people rubber-banding back when they stop moving. Eventually that will probably go away and be replaced by a model much more similar to how the game handles it natively, but more research is needed in that area to make it work with real latency conditions.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #306 on: August 06, 2015, 03:21:09 AM »
Also, there's an important distinction that I think should be noted. When I'm talking about clientside prediction, I'm talking about the player. The COH client does motion prediction only for its own player. It never speculatively predicts the location of other players, NPCs, etc. Those are effectively physicsless and are not even processed except to draw them. It relies on the mapserver to send regular updates that it can interpolate between, but introduces some intentional time lag so that those updates seem to be happening in real-time, even when they're not being continuously received.

That's probably the cause of the latency that you're seeing even when the XMPP server is local. The movement 'buffer' as it were seems to be integrated into the client. I haven't yet investigated all the murky corners of the protocol to see if there's any way to disable it within the protocol itself assuming a near-infinite speed connection.

Now, Paragon Chat does do speculative prediction as well as apply collision mechanics to the simulated players (mostly so falling doesn't go through the ground), which is why you often see people rubber-banding back when they stop moving. Eventually that will probably go away and be replaced by a model much more similar to how the game handles it natively, but more research is needed in that area to make it work with real latency conditions.

Multiboxers probably noticed this more than other players.  If you multiboxed, movement in one session took a lot longer than people might expect to register in the other session, completely disproportionately to actual network lag.  Client-side prediction works because we can usually only detect the difference between our own inputs and our own character's response to those inputs, but we're usually poorer judges of "world lag" because we don't really know when the world "wanted" to do something verses when it did it.

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #307 on: August 06, 2015, 02:32:24 PM »
Well, my bot is finally reacting to PC stanzas.

And today's episode of Stupid Dyne Errors shall be called:
String Formatting Considered Harmful

When it came time to register the handlers for the custom stanzas, both Arcana and this example used a matcher like:

Code: [Select]
MatchXPath( '{%s}presence/{%s}pc' % (self.default_ns, pcPresenceStanza.namespace) )
That uses python's string formatting.  See, in the versions of Python 2 that I am familiar with, the %s patterns get expanded to the values of the variables after the % operator.  In Python 3 (and 2.7), this old way still works, but the preferred syntax for this has changed to "{}".format(variable) and -- having learned my lesson with the Byte slicing earlier -- I was just habitually converting this sort of thing as I went along.

Can you guess what I did?  Look closely at what is present in both the XPath call above and in Python 3's syntax, yet absent from my description of the old Python 2 syntax.

Yep.  It probably took you about 1/100th the time it took me. :-[

I saw the braces in the original version.  But, thanks to their presence in exactly the same spots in the new syntax, I didn't see them see them.  When I converted to the new syntax in situ, they were there already so I just called it a day.

My version ended up like:
Code: [Select]
MatchXPath( '{}presence/{}pc'.format(self.default_ns, pcPresenceStanza.namespace) )
Instead of "{jabber:client}presence/{pc:presence}pc", it was trying to match "jabber:clientpresence/pc:presencepc".  Until I managed to get around the blind spot induced by the syntax change, I couldn't even figure out how the original version could have produced anything valid.  Bad Dyne, no cookie.

(In my defense, I was distracted by other things that seemed more likely.  I kept glancing at that MatchXPath, frowning, and then checking something else.  After all, two different sources had used string formatting, and one would have to be a complete idiot to mess up a syntax conversion that simple, right?)


Unrelated, but I also discovered that joining the meta channel from Pidgin while also testing an invisible bot that chokes and dies (for a different issue) can confuse either Paragon Chat or Openfire, not sure which.  It resulted in a duplicate of my character; it looked like me and followed my movements, emotes, etc.  It's almost like I had a bot that would mirror my appearance and movements, without doing any of the work to get there. 8)

« Last Edit: August 06, 2015, 06:07:37 PM by Dyne »
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #308 on: August 06, 2015, 05:43:40 PM »
Unrelated, but I also discovered that joining the meta channel from Pidgin while also testing an invisible bot that chokes and dies (for a different issue) can confuse either Paragon Chat or Openfire, not sure which.  It resulted in a duplicate of my character; it looked like me and followed my movements, emotes, etc.  It's almost like I had a bot that would mirror my appearance and movements, without doing any of the work to get there. 8)

That's odd.  If I had to guess I'd say that Paragon Chat put an entry in an entity table for the pidgin client when it announced presence but because it didn't actually announce anything in the paragon chat namespace it wasn't visible.  Then when you bot tried to become visible and crashed it somehow convinced Paragon Chat that the pidgin entity was a Paragon Chat entity, and cross linked it with the position and animation data for your actual Paragon Chat character.  Whereupon you'd now have two entities with identical position and animation information.  Then capsule bumping would separate the two and you'd have two identical entities echoing each other's movements and separated by a few feet.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #309 on: August 06, 2015, 06:21:54 PM »
Did you use the same nickname in the channel by any chance? I've noticed that while Openfire normally disallows identical nicknames from separate resources as per the XMPP spec, it allows them if they differ by case only. Once you do that, the room occupant status gets really weird and both hang around as long as one of them stays in the room, and don't become "unavailable" until the last client on that base JID leaves.

As Paragon Chat treats JIDs as case-insensitive, I could definitely see that confusing it and causing it to only recognize one of them as "self", then creating a proxy player for the other.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #310 on: August 06, 2015, 06:29:09 PM »
I don't think that has anything to do with prediction or collision at all, actually. If you watch the video closely, you'll see that at 0:04 the bot has acquired quite a roll already, but it isn't even near the wall yet, and returns to normal by the time it reaches the wall. In fact, on rewatching it, to my eyes at least, the correlation appears to be that the bot rolls shortly after you jump, and again very slightly when its Y position differs from yours due to the curb. Are you perhaps getting pitch and roll swapped somehow?

Now this is strange.  I knew that my orientation calculations were working, because earlier I posted the results of my "faceme" command which actually worked.  Today, /faceme doesn't work, it rolls the character.  So you're right, the funny roll is due to the character trying to pitch up and rolling instead when running through the wall as my character jumps over it.

Except, I can't see what I'm doing wrong.  In fact, Paragon Chat itself says I'm sending the right thing.  When I do a /faceme where I'm above and in front of the character, instead of pitching upward the character rolls clockwise.  However, this is what Paragon Chat says it received when I do that:

xmpp DEBUG RECV: <message lang="en" type="groupchat" to="arcanacoh@core3770/Violet" from="atlaspark_meta@conference.core3770/ArcanaBot_meta"><u p="906.5038956 0.0818596208137 -490.958855515" xmlns="pc:u" v="0 0 0" o="-0.393764839539 -1.58960360374 0.0"/></message>

Notice roll is zero.  As far as I can tell, Paragon Chat confirms my roll in o= is always zero.  Did Paragon Chat change PYR order in orientation at some point?  I know this worked back when I posted this: http://www.cohtitan.com/forum/index.php/topic,10956.msg189244.html#msg189244.  The code to do that is literally identical between that version and my current one.

No, it can't be an order thing.  I just added roll controls to the bot that allow me to roll and nullify rolls.  Even when roll around and then force roll to be zero, when pitch is nonzero roll won't nullify.  I think there's something wrong with Paragon Chat's matrix calculations that are intertwining pitch and roll.  Its almost as if pitch is always zero and roll is always pitch+roll, if that makes sense.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #311 on: August 06, 2015, 06:45:11 PM »
Gotcha.  I added *pitch* controls to my bot so I can move freely.  The problem is that while roll seems to be working correctly, pitch is not.  Pitch doesn't factor in yaw.  Meaning, if your orientation is 0,0,0 and you try to pitch up and down, that works.  But if you turn 90 degrees, roll still works correctly about the axis you're facing in, but the pitch axis doesn't rotate with yaw, pitch now *becomes* roll because the pitch axis is now aligned with the roll axis, as if it were still pitching up and down relative to 0,0,0.  And I think this bug has been there since the beginning of time, since when I test with older version of PC I see the same behavior.  My pitch test from long ago worked only by coincidence: I'm facing close enough to 0,0,0 for pitch to appear to be working correctly.

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #312 on: August 06, 2015, 07:01:24 PM »
Did you use the same nickname in the channel by any chance? I've noticed that while Openfire normally disallows identical nicknames from separate resources as per the XMPP spec, it allows them if they differ by case only. Once you do that, the room occupant status gets really weird and both hang around as long as one of them stays in the room, and don't become "unavailable" until the last client on that base JID leaves.

I'm pretty sure I did end up in the room as dyne and Dyne, yeah.  In fact, I was just looking at a global chat muc that was doing exactly what you described (Paragon Chat was no longer logged in, but Pidgin saw two of me there).  I should change Pidgin's settings.
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #313 on: August 06, 2015, 07:17:26 PM »
And I think this bug has been there since the beginning of time, since when I test with older version of PC I see the same behavior.  My pitch test from long ago worked only by coincidence: I'm facing close enough to 0,0,0 for pitch to appear to be working correctly.

Sure enough, the function that recomposes the matrix from the "o" attribute is using PYR order instead of YPR. Well, easy enough to fix, I just swapped it and the fix will be in the next minor release. Just goes to show about how often COH uses pitch that it's gone unnoticed until now.

That probably explains the odd rotations I had seen when dual boxing and looking at myself "flying" around with nocoll turned on.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #314 on: August 06, 2015, 07:33:39 PM »
Sure enough, the function that recomposes the matrix from the "o" attribute is using PYR order instead of YPR. Well, easy enough to fix, I just swapped it and the fix will be in the next minor release. Just goes to show about how often COH uses pitch that it's gone unnoticed until now.

That probably explains the odd rotations I had seen when dual boxing and looking at myself "flying" around with nocoll turned on.

Even I didn't notice anything amiss until now, and I wrote it off.  When you're testing to see if something is working, you don't usually say "lets try that again, but facing in a different direction" although I really ought to know better, because that's *exactly* the test you should perform when testing orientation calculations.  But since I was mostly working in the XZ plane trying to get "normal" motion working, 99% of the time both pitch and roll are zeroed out.

At least now my bot can do something my City of Heroes characters could never do: an Immelmann.  I'm thinking of making it a flying primitive. 

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #315 on: August 06, 2015, 08:08:34 PM »
Code: [Select]
<frederick xmlns="FRONK-un-steen" age="young">From that fateful day when stinking bits of slime first crawled
from the sea and shouted to the cold stars, &quot;I am man.&quot;, our greatest dread has always been the knowledge
of our mortality. But tonight, we shall hurl the gauntlet of science into the frightful face of death itself. Tonight, we shall
ascend into the heavens. We shall mock the earthquake. WE SHALL COMMAND THE THUNDERS, AND PENETRATE INTO
THE VERY WOMB OF IMPERVIOUS NATURE HERSELF!</frederick>





Code: [Select]
<igor>He's going to be very popular with the ladies.</igor>
And now I'm going to bed.  *flop* :)
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner


slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #317 on: August 06, 2015, 08:52:07 PM »
Congrats. ;-)

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #318 on: August 08, 2015, 08:26:28 PM »
Dynebot is now serving a (hardcoded) costume in response to requests instead of just parroting a hash.  I wasted a bunch of time fighting the library, trying to get it to let me use the obvious solution of storing my desired appearance/part xml elements in a string and then stuffing that into the costume stanza.  The closest I came was using the iq._set_sub_text method, which solved my previous problem -- how to put the description into my iq.reply(). 

Unfortunately, the underlying objects appear to escape the text contents of an element whether you want them to or not ... which is fine for a description, but as far as Paragon Chat was concerned, it turned the costume to gibberish.  I could have delved into ElementTree or whatever, but at that point I just gave in and separated my string into objects.  I'd have been better served hooking Dynebot up to my actual costume loader, but I was just trying to get the thing to supply an independent costume ASAP.

Currently, Dynebot is at 745 lines; building the stupid costume manually is about 6% of that, and a large minority of the rest probably consists of either comments, whitespace, or logging.

It has multiple commands (help, restart, quit, resend presence, resend position, and speak), the latter of which is mostly pointless because it only joins the meta channel right now.  It keeps a list of nearby users for later use with local chat/text emotes whenever I get around to implementing them.  When someone joins the meta channel, the bot emulates Paragon Chat's random delay before refreshing presence/position (since it doesn't move or change animations yet) via an asynchronous thread.

While I was held up by the costume thing, I added classes and handlers for the chat bubble's <scale> and <dur> pseudo tags on a whim, even though Paragon Chat doesn't actually support them yet, and I'm not sure why the bot would care.  I suppose I could make it ask people using scale != 1 why they are whispering or yelling, and those with non-default durations why they're talking so fast or slow.  Considered going ahead and templating team, sg, league, etc. chat, but then I realized that these would probably need to be implemented as completely separate rooms or whatever, rather than messages in the zone room using the channel element (because they used to work across zones).

I haven't decided what I want to do next.  I really need to factor out as many bits of code that control the bot's behavior and appearance from the bits that are "generic PChat/XMPP stuff" as I can, but I expect that'll be fairly tedious.
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #319 on: August 09, 2015, 12:51:40 AM »
Dynebot is now serving a (hardcoded) costume in response to requests instead of just parroting a hash.  I wasted a bunch of time fighting the library, trying to get it to let me use the obvious solution of storing my desired appearance/part xml elements in a string and then stuffing that into the costume stanza.  The closest I came was using the iq._set_sub_text method, which solved my previous problem -- how to put the description into my iq.reply(). 

That's not necessary. If you're got a stanza object defined, and you're setting the attribute values via its defined interfaces, then it's already an XML object. The costume stanza is just an appearance stanza and a bunch of part stanzas - Create a new costume stanza variable, make an appearance stanza, append it to the costume, make each part stanza and append it to the costume, then append the costume to the IQ reply stanza.

The stanza methods take care of the xml magic. You can even do like I did and save out myCostume.xml to a file on disk. If you build the stanzas at the same time you are building your hash string, it all happens organically. Your costume stanza gets generated along with your hash, and then you stick it in a variable where you can pull it out whenever you need to append it to a presence or IQ reply.

You're going to want to be able to load a .costume file sooner rather than later, so you might as well make that your next order of business.

Quote
I haven't decided what I want to do next.  I really need to factor out as many bits of code that control the bot's behavior and appearance from the bits that are "generic PChat/XMPP stuff" as I can, but I expect that'll be fairly tedious.

I'm at this point myself. I have some other things going on that are demanding my attention and I'm having to consider some sort of direction that I'm taking this thing. I've identified three goals of increasing complexity that I believe I can accomplish with just the technology we currently have, though the implementation details will require some math that I haven't used since high school (and you don't want to know how long ago that was).

Pushing towards something that gives a tangible result gives me more motivation. At the moment I'm taking a break from the bot code proper, and I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up. I actually found that I could pull up practically an entire five or six year archive of the old CoH forums from the Wayback, but I don't think I've got the drive to write a parser that would spend the time to download each page, download all the individual posts, and then extract just the text of each post. What I do have is all of the lore bible docs that were released after the game closed, and all of the Paragon Times, and Know Your Adversary and other lore stuff that was on the official web pages. The initial results are sometimes rather hilarious but there's enough non-useful garbage in the story bible docs that I'm taking the time to edit out all the crap and make a "clean" data file for training the chatbot's "brain".

Once that's done, then I'll go back to the bot and add the code it needs to track everyone in whatever rooms the bot has joined, track their positions, respond to input on /Local or /Tell, and insure that local chat is only responded when the recipient is within 15 feet or so. Once all that works, I'll probably throw it online and see if anyone finds it amusing. ;)