Author Topic: Technical side discussion  (Read 164633 times)

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #220 on: July 29, 2015, 06:46:06 PM »
I agree, @Arcana, and there are steps I can take if that's a necessary thing. I'm pretty sure this is new behavior,though. I don't remember any previous versions of PC crashing just from listing a help menu in the bot. I wasn't actively testing for that kind of thing, so I can't state categorically that it worked previously but I've used those menus before this time and I have to think that at least once would have been in the Paragon Chat channel if not in Broadcast.

Could it be this is the first time you did this with a visible bot?  Some aspects of chat might be handled slightly differently when you are visible verses not visible or not on the map.  If I get a chance I might try this tonight, since I have a mostly-complete library of different Paragon Chat versions (all the numbers, not necessarily all the letters).

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #221 on: July 29, 2015, 08:13:59 PM »
Ah, you're thinking chat bubbles might be involved?

I guess that 0.98beta would be probably be the first time with a visible bot.

Chat bubbles are probably a good reason to consider limiting message size regardless.

I guess we'll see what Codewalker thinks next time he stops in.


slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #222 on: July 30, 2015, 12:37:09 AM »
Arcana, I need your math whizdom.

Let's say I want to order my bot to turn and face me, wherever I happen to be standing.

Simple, right? It would be for human. For a bear of little brain, it's a bit more difficult.

Here's what I think needs to happen:

1) I move to a random X,Y,Z point in the zone.

2) I send the command "Errbot, face me".

3) The bot now assumes a facing of zero. This seems more useful than trying to figure out where things are in relation to its own random orientation. For the sake of discussion, we'll call this facing "North". It doesn't matter what the compass on a map of Paragon City would call it.

4) Errbot draws a line segment between itself and my X,Z position. (In Paragonspace, Y is the height axis. We don't care about height for this.)

5) Errbot now draws another line segment of equal length towards "North".

6) Errbot now has three reference points. Itself and the two endpoints. Using those, it can compute the angle between North and me. Using that angle, it should be able to compute the number of radians in an arc between those two endpoints.

7) Errbot sets its facing equal to the results of step 6.

8) Fini

Does that sound right?

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #223 on: July 30, 2015, 12:55:28 AM »
Actually, you only need one directional vector (and rotation order) to calculate orientation.  You are at (X,Y,Z).  Your target is at (X',Y',Z').  Your directional vector is (X'-X,Y'-Y,Z'-Z) which we will abbreviate as (x,y,z). 

According to Codewalker, City of Heroes (and Paragon Chat) apply rotations in YPR order.  So we will do the same.  We start by computing the yaw angle which is in the XZ plane.  Given the projected vector (x,z) you can use basic trig to calculate the yaw angle for that vector in radians.  Then, given the triangle in that vector's plane composed of the length of (x,z) which is sqrt(x^2+z^2) and the inclination height y, you can calculate the pitch angle up or down. 

You never specifically need to change roll angle to face any target in space (which may be why the devs decided to use YPR order).  In YPR order, roll would spin you around the axis that connects you and your target without changing the fact you are continuously pointed at it (obvious, since roll is defined loosely as rotation about the axis you're facing, it doesn't change the direction you're facing).

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #224 on: July 30, 2015, 02:08:13 AM »
"Motion" interpreted to be trying to get your bot to travel from point A to point B in a way that is both smooth in terms of change of location and orientation, combined with the appropriate animations that make you seem like you are walking/running/flying and not just gliding there in mid-air like Magneto, is a little bit harder to do in the general case.  It involves combining acceleration (so you don't instantly start moving at full speed), changing orientation smoothly (so you face in the direction you're moving, assuming you want to do that), and executing the right animations at the right time to look like you're walking/running/flying and *also* you stop doing that when you stop moving so you don't keep moonwalking in place.

For myself, I'm going to attempt to adapt the basic acceleration logic of the elevator system I'm making in UE4 to handle some of that, since I already have that working.

In other news:

Code: [Select]
Hard-coded concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============

Generated concatenation:
=============
1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff
=============

Congrats!  Concatenations match!



PChat's Hard-coded Base64 Hash: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=

Arcana's Hard-coded Base64 Hash: ycY42S40EIPgvh6GaGWYI+H/tdk=

Arcana's Generated Base64 Hash (byteswapped): 2TjGyYMQNC6GHr7gI5hlaNm1/+E=

My Generated Base64 Hash (byteswapped): 2TjGyYMQNC6GHr7gI5hlaNm1/+E=
My Generated Base64 Hash (normal, for comparison): ycY42S40EIPgvh6GaGWYI+H/tdk=


A WINNER IS YOU!  Byteswapped Hash matches Paragon Chat.

Pro-tip: In Python 3.3, you need to slice byte objects like foo[0:1], not merely index them like foo[0].  Attempting to byte swap a 20 byte digest with the same logic that worked under Python 2.7 first required dealing with a type mismatch (the index version returns an int under 3.3) and then would only give me 2765 byte output.
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #225 on: July 30, 2015, 02:15:23 AM »
Well, Arcana beat me to it, but I was going to describe the process in a slightly different way.

Step 1: Subtract the bot's coordinates from the target's coordinates. That effectively gives you the target's position in what we'll call "bot-space" -- a coordinate system in which the bot is the center of the universe.

Step 2: Normalize the resulting vector. That means calculate the magnitude of it (sqrt(x^2 + y^2 + z^2)), and divide each coordinate by that magnitude. What that gives you is a unit vector pointing at the target. This is also known as the desired forward vector, since it's the direction you want to face.

Step 3: Use any of the well-known formulas to convert a unit vector to YPR. Assuming my trig is right (please feel free to verify and correct if necessary), you can get yaw and pitch from your vector like this:

P = asin(y)
Y = atan2(x/cos(P), z/cos(P))
R = 0

You'll need special handling for y=+/- 1 to avoid a division by zero. In that case just set P=+/- pi/2 (90 degrees) and Y=atan2(-z,x).

To deal with roll you would need a full rotation matrix, or at least both a forward and an up vector, but you probably don't have any reason to need it so you can simply leave it zero. I got the idea of dividing by cos(P) from this publication, which goes into a much more detailed discussion of methods for extracting Euler angles from a rotation matrix (of which the 'forward' vector we're discussing is the third row that defines the Z axis). Though be aware that it describes PYR order and needs some adjustment for YPR.

At some point in the future I'll probably add protocol support for optional alternative methods of specifying orientation that may be more convenient than Euler angles. Likely quaternions. In that case it's relatively simple to convert the forward unit vector into a quaternion; just a cross product to get a parallel vector and a dot product for the magnitude of the rotation.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #226 on: July 30, 2015, 02:38:30 AM »
Pro-tip: In Python 3.3, you need to slice byte objects like foo[0:1], not merely index them like foo[0].  Attempting to byte swap a 20 byte digest with the same logic that worked under Python 2.7 first required dealing with a type mismatch (the index version returns an int under 3.3) and then would only give me 2765 byte output.

One of the many reasons why I tend to still use Python 2.x.  Byte objects are strings in Python 2.x but integer sequences in Python 3.x.  That means you have to deal with the craziness that is the fact that foo[0:1] is a sequence of bytes (with exactly one item) but foo[0] is the first integer in the integer sequence that comprises the byte (type) object.  In Python 2.x, foo[0] is a character - which is essentially a byte - and foo[0:1] is a character sequence of length 1, which is also a byte.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #227 on: July 30, 2015, 03:13:38 AM »
Step 3: Use any of the well-known formulas to convert a unit vector to YPR. Assuming my trig is right (please feel free to verify and correct if necessary), you can get yaw and pitch from your vector like this:

P = asin(y)
Y = atan2(x/cos(P), z/cos(P))
R = 0

You'll need special handling for y=+/- 1 to avoid a division by zero. In that case just set P=+/- pi/2 (90 degrees) and Y=atan2(-z,x).

I *think* I did something like this (code not in front of me):

k = sqrt(x^2+z^2)
k' = sqrt(x^2+y^2+z^2)

Y = acos(x/k) * sign(z)
P = asin(y/k')

Technically I haven't tested my "go run that way" code yet because I was away from computer for an extended weekend, so hopefully my trig has survived thirty years of degradation.  I will probably be testing limited movement scripting this weekend (also, I'm putting off loading and collision checking maps because ick).

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #228 on: July 30, 2015, 10:13:57 AM »
I decided to do a quick hack to test orientation by adding a "/faceme" command to the bot.  As it turns out, there was some flipping and such required:

Code: [Select]
            x1 = self.position[0]
            y1 = self.position[1]
            z1 = self.position[2]
            x2 = float(self.cmdQueue[0][1].split()[0])
            y2 = float(self.cmdQueue[0][1].split()[1])
            z2 = float(self.cmdQueue[0][1].split()[2])

            xx = x2-x1
            yy = y2-y1
            zz = z2-z1

            k = math.sqrt(xx*xx + zz*zz)
            k2 = math.sqrt(xx*xx + yy*yy + zz*zz)

            yaw = math.acos(zz/k) * cmp(xx,0)
            pit = -math.asin(yy/k2)
            rol = 0.0

Should be fairly self-explanatory.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #229 on: July 30, 2015, 02:50:37 PM »
Fortunately (or not) I'm cutting my teeth on Python 3 so it's way is the only way, I know, heh.


Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #230 on: July 30, 2015, 05:37:33 PM »
Incidentally, there's what's mathematically true, and what's mechanically logical.  When you tell the bot "/faceme" the math says to turn directly towards the coordinates of the target, but that's not necessarily what makes sense to do.  For example:



A person would look up.  A person would not tilt backwards to stare upward.  So separate from the question of how to write the code, and separate from the question of how to do the math, is always the question of what you actually want to do.  Do you really want to face the target in three-space (maybe, if you were flying) or do you want to confine your orientation to the xz plane (which makes more sense if you are standing).

These are, in fact, the kinds of questions game programmers have to answer also.

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #231 on: July 31, 2015, 02:16:39 AM »
Fortunately (or not) I'm cutting my teeth on Python 3 so it's way is the only way, I know, heh.

I cut my teeth on Python zodknowswhat back in the early-mid 90s.  I code stuff so infrequently that I'm still thinking of parts of Python 2 as fancy and new-fangled.  :-[

But seriously, if you deduce from my previous post that I still haven't used Python 3 very much, you'd be correct.
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #232 on: July 31, 2015, 02:50:44 AM »
I've got an odd problem that's caused me some trouble.

I have two installations of Tequila/I-24/Paragon Chat on two different computer, each with its own Openfire server. They share my python folder via dropbox, so they typically have the same bot code on each (not always, but usually).

My bot works perfectly on the one. The other shows it sometimes and then stops showing it unless it's being viewed on a new account with no cache/history. Then once the bot goes into cache, if it logs off it is gone for good. No amount of restarting of the bot will cause it to show up for that client. Sometimes restarting the Paragon Chat/CoH clients will cause the bot to reappear, but just as frequently the bot will remain invisible through restarts of the entire software suite.

This was a big part of the reason I thought my bot was failing when it was actually working as intended. I just couldn't see it until I moved to the other computer to test on.

I don't really have a clue what's going on except that it seems like it could be peripherally related to the costume cache in the client. However; given that the other computer and its clients work beautifully, I can't really point to anything in Paragon Chat as the cuplrit, other than to note that it seems to happen most frequently after the bot costume is "known" by the client.

Since I'm not even positive that it is an issue with Paragon Chat, I'm not sure what to look at as far as fixing it. I might have to just stop using that computer for bot development and stick to just the one that properly displays things like they ought to be displayed.

Any suggestions, Codewalker? There's nothing in the debug logs of either the bot or of Paragon Chat that suggests that anything is out of whack.


Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #233 on: July 31, 2015, 03:45:01 AM »
One problem I've encountered with sleekxmpp is that if you create a bug that crashes a sleekxmpp thread, other threads can sometimes survive the death of the bot.  When that happens, I end up with zombie python.exe processes that maintain an XMPP connection.  Then when you try to restart the bot, Openfire doesn't let you log in properly with the same name and resource.  A complete implementation of XMPP would first ask if the resource was available, but I don't do that.  So you end up with the bot essentially trying to log in with credentials that are already in use.  Openfire effectively "kicks" you off the server by announcing you as "unavailable."  Paragon Chat sees that as a log out, and what happens is my bot appears briefly (because I send a valid presence and location stanza) and then disappears.

When that happens, I check to see if I have rogue python processes, and I also check to see if I stop the bot do I still see sessions in the Openfire console.  If I do, then I know zombies are interfering with the bot.  I have to kill those python processes and sometimes also reset those Openfire connections (killing usually does that for me, but once it didn't).  Then when I log in cleanly, the bot starts working correctly again.

That doesn't sound precisely like the problem you're having, but perhaps it might spark an idea or two.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #234 on: July 31, 2015, 03:18:52 PM »
Paragon Chat probably doesn't bounds check everything perfectly since normally the City of Heroes client itself has enforced limits on things like the maximum length of a tell.  Its probably a good idea to try to stay within those bounds ourselves, although given the fact that we're not the only people that will eventually figure out how to do this, its also probably a good idea for Paragon Chat to bounds check all XMPP messages just for anti-griefing purposes alone.

It is a maximum length issue, but it's not a bounds checking / buffer overflow problem per se, so things like remote code execution aren't possible. It actually is doing bounds checking on the buffer, however the default behavior unless otherwise specified is to assert() on exceeding the space available. That's a fail-safe as silently altering input when not expected could result in undefined behavior.

In this case, truncating overly long chat messages is harmless, and being able to remotely trigger an assert() is not so good, so I'll fix those cases to just chop off any extra.

I agree, @Arcana, and there are steps I can take if that's a necessary thing. I'm pretty sure this is new behavior,though. I don't remember any previous versions of PC crashing just from listing a help menu in the bot. I wasn't actively testing for that kind of thing, so I can't state categorically that it worked previously but I've used those menus before this time and I have to think that at least once would have been in the Paragon Chat channel if not in Broadcast.

Private tells and global channel messages were already being truncated if they exceed the maximum message length. Only broadcast and local were triggering the assert. That'll be fixed in the next incremental update.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #235 on: July 31, 2015, 04:01:12 PM »
Private tells and global channel messages were already being truncated if they exceed the maximum message length. Only broadcast and local were triggering the assert. That'll be fixed in the next incremental update.

In the meantime, I can configure ErrBot to break up messages that exceed a given size. What is a good message length to use to stay under the assertion/truncation limit?

I gave the bot an emote command last night and I was playing with having it perform lots of different emotes. (A strangely empowering experience. Dance, Puppet, Dance!) I wanted to double-check about the whole sequencer thing. All the bot ever sees from the game client on the wire is a MOV. That's what a bot will always choose to send on the wire itself in order to play an animation, correct? A bot doesn't know or care about the game client's sequencers, at least not at this point in time.


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #236 on: July 31, 2015, 04:23:54 PM »
Well, the in-game chat window doesn't let you type more than 256 characters I think.

Paragon Chat's internal buffer for chat messages is 1024 bytes and is what will assert (or truncate in future versions). That's what it uses to format the entire message "Character: text" or "@global: text", however, so the maximum message length varies depending on the length of the sender's name.

Global Channels operate over an entirely different system, using so-called shardcomm commands that work differently from the rest of the game. Theoretically, those can be up to 8192 bytes in length, so longer messages are possible there. That 8192 must also fit the text "ChanMsg", the name of the global channel, the sender's global name, the message, and quote marks around each of those. Various characters are also escaped to two-byte sequences, for example ' becomes \s and " becomes \q.

To be on the safe side, I wouldn't send messages longer than 900 bytes or so. 512 if you want to be conservative.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #237 on: July 31, 2015, 04:29:55 PM »
Just to be clear, you don't need to worry about any of the shardcomm or escaping stuff when writing a bot. Paragon Chat takes care of all those details; all you need to send it is well-formed XML.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #238 on: July 31, 2015, 06:01:41 PM »
I gave the bot an emote command last night and I was playing with having it perform lots of different emotes. (A strangely empowering experience. Dance, Puppet, Dance!) I wanted to double-check about the whole sequencer thing. All the bot ever sees from the game client on the wire is a MOV. That's what a bot will always choose to send on the wire itself in order to play an animation, correct? A bot doesn't know or care about the game client's sequencers, at least not at this point in time.

At the moment, all a bot needs to send are MOVs.  All a bot will ever see another entity perform are MOVs.  You don't need to understand sequencers to animate.  However you do need to understand sequencers if you want to know what an entity is doing right now.

For example, suppose your bot is logged into the same zone as my bot, and my bot decides to walk towards you.  You'll see me do a m="PLAYER_WALK_FORWARD" and then a lot of <u /> messages will show I'm moving towards you and when I stop you'll see me do a m="PLAYER_WALK_POST".  All good.  But what you won't see is that for most of the time I'm walking any Paragon Chat user will see my bot playing "PLAYER_WALK_FORWARD_CYCLE".  That's because according to the CoH sequencer system, PLAYER_WALK_FORWARD_CYCLE should be played immediately following the completion of PLAYER_WALK_FORWARD and should itself be continuously looped until interrupted.  Paragon Chat never sends such information because its not necessary due to its ability to execute the sequencer, and as a consequence my bot never sends that CYCLE MOV either.  Because there are animations that get played but never sent on the wire, *if* your bot needs to know if a player ever performs a specific MOV, its not enough to simply check for it on the wire.  You would technically need to make sure that no entity played an animation that eventually leads to that MOV also, and you would need to know how long each MOV takes from here to there, so you know *when* that MOV ultimately comes up.

If your bot is not going to react to specific MOVs played in its presence, then its not necessary to know if any specific MOV is being played at the moment, and thus its not necessary to understand how the sequencer affects future animations.  If at some point you do want to do that, then you would need to know at least some of that information.

Separately, while you don't *need* to understand the sequencer itself, knowing which animations follow which is useful if your bot moves, for economy.  Meaning, if I *tell* you that the walk sequence is PLAYER_WALK_FORWARD -> PLAYER_WALK_FORWARD_CYCLE  -> PLAYER_WALK_FORWARD_CYCLE, PLAYER_WALK_POST, then without necessarily knowing anything about the *specifics* of the CoH sequencer you could still use that knowledge to make your bot walk by knowing you only need to send the first and the last and not worry about the middle.  So if you want your bot to play animations with multiple sequence entries, knowing how that specific sequence works can be helpful.  But that doesn't explicitly require that your bot understand the sequencer.  Only that you do when you write the bot.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #239 on: July 31, 2015, 06:48:57 PM »
Yes, for starters the bot just needs to be able to walk around Atlas Plaza but the end game will need to be an emote database that knows the full sequence of any arbitrary animation it might play.