Author Topic: Technical side discussion  (Read 165046 times)

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #160 on: July 18, 2015, 11:56:41 PM »
Reading back now it looks like that's pretty much what Arcana said, so, good. I'm going to have to come up with some sort of testbed for costume hashes, though. If we can't just compare our hash to the hash presented by PC, then there's no practical way to tell whether your hash is actually correct or not. I only felt fairly confident about my hashing function after Codewalker certified Arcana's results and I was able to replicate her results. When you look at your function spit out a string of base64, it isn't intuitively obvious whether that string is correct or incorrect.

Back up.  I did this in a particular order to boot strap the process.  If you want your bot to be visible, it actually doesn't have to process any costume stanzas at all as long as you recognize you can cheat: if you advertise a costume has that is identical to one everyone already knows about then they won't ask you for your costume hash because they already know what the costume corresponding to that hash looks like.  In particular, if you are just testing, if your bot presents a costume hash that is identical to the one your Paragon Chat client presents for its logged in character (assuming you are testing on your own server and you have Paragon Chat logged in so you can actually attempt to see your bot) then because you're echoing back Paragon Chat's character hash, it will never ask you for your costume, because it will always know that costume and obviously it can't timeout its own costume in its hash table.

But there's no actual reason why your hashes must match Paragon Chat hashes in the specific.  Your hashes must match the costumes you *send* to Paragon Chat.  Even if the costume you send is a costume Paragon Chat would never actually create on its own, so long as your hash matches your costume stanza, Paragon Chat will at least validate it as correct and then use that costume data.

Unfortunately, there's no way to know if your hash is correct just by looking at it.  In fact, that's the whole point of how hashes work: you can't predict what they should look like without just plain computing them.  Which is why rather than use incremental SHA (as Codewalker did in Paragon Chat) I actually compute the long-form string.  The reason for doing that is that in theory the place you're going to screw up is not in computing the hash, but determining what to feed to the hash engine.  You and I and even Codewalker are just feeding OpenSSL's SHA1 algorithm: we have to generate the same hashes given the same input.  By recording the string that goes in, I can compare that with the costume stanza data that Paragon Chat sends out for costumes.  If ever I think something has gone wrong with my costume code, I can feed that costume to City of Heroes and ask Paragon Chat (by running it in debug mode) to tell me both what the hash is for that costume *and* what data it would send for that costume (in the costume reply stanza).  If the costume data matches my costume file, I can then compare hashes freely.  If the data doesn't match, as my costume did not, I need to modify my costume file to match what Paragon Chat sent so that my data and its data are identical, and then compare hashes.  Or alternatively, I could conduct a detailed investigation into why the costume data changes inside of City of Heroes: there are a lot of weird redundancies in costume files, and City of Heroes often seems content to randomly fiddle with its don't-cares (which is a sign of extremely demented programmers).

In summary:

1.  A specific set of costume data always generates the same (SHA1) hash.

2.  Two sets of costume data that differ by any amount however small generate different hashes (ignoring cosmicly rare collisions), even if the only difference is upper/lower case differences.

3.  A specific costume *file* is not guaranteed to generate costume data that is exactly identical within the game itself.

4.  Two different costume files and two different costume data sets can generate the same appearance.  For example, a costume data set in which all of the alpha characters are lower case generates the same appearance as one that is identical but all the alpha characters are upper case.  However, these two costume data sets generate different hashes.

5.  Paragon Chat compares the hash you send with the hash it computes for the data you send.  Paragon Chat does not (and SHOULD not) do anything to the data you send before computing its hash, so if you send the exact data you used to compute your hash, it should work.

I should point out that I hit the bullseye on essentially the first try: this is not hard to do.  If it wasn't for the endian issue (see: endianness can be an issue :p ) and the loss of two color elements (still need to track that down) I would have had it on the first go around.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #161 on: July 19, 2015, 06:58:22 AM »
Hmm.  I'm not sure if SleekXMPP is doing the wrong thing, or I'm using it the wrong way.  No matter what I do, I cannot seem to actually receive the Iq costume request sent by Paragon Chat.  I even registered a stanza callback to show me *all* Iq requests, and it goes so far as to show me the Iq traffic that sets up the SSL handshake.  It shows me the Iq traffic for rosters.  And yet it does not ever show me the Iq costume request.

Completely separately, I seem to be not understanding substanza semantics quite right.  I tried to do this:

Code: [Select]
class pcAppearanceStanza(ElementBase):
    namespace = 'pc:costume'
    name = 'appearance'
    plugin_attrib = 'appearance'
    interfaces = set(('bodytype','skincolor','scale','bone','head','shoulder','chest',
                      'waist','hip','leg','head3d','brow3d','cheek3d','chin3d','cranium3d',
                      'jaw3d','nose3d'))

class pcPartStanza(ElementBase):
    namespace = 'pc:costume'
    name = 'part'
    plugin_attrib = 'part'
    interfaces = set(('n','geom','tex1','tex2','fx','color1','color2','color3','color4'))

class pcCostumeStanza(ElementBase):
    namespace = 'pc:costume'
    name = 'costume'
    plugin_attrib = 'costume'
    interfaces = set(('hash', 'appearance', 'parts'))
    subitem = (pcAppearanceStanza, pcPartStanza)

    def getParts(self):
        parts = {}
        for pt in self.xml.findall('{%s}part' % pcPartStanza.namespace):
            part = pcPartStanza(pt)
            parts[part['n']] = part['value']
        return parts

    def setParts(self,parts):
        for part in parts:
            self.addPart(part)

    def delParts(self):
        parts = self.xml.findall('{%s}part' % pcPartStanza.namespace)
        for part in parts:
            self.xml.remove(part)
        return

    def addPart(self, part_dict):
        part_obj = Part(None, self)
        for element in ['n','tex1','tex2','fx','color1','color2','color3','color4']:
            if element in part_dict:
                part_obj[element] = part_dict[element]
        return

    def delPart(self, n):
        parts = self.xml.findall('{%s}part' % pcPartStanza.namespace)
        for partXML in parts:
            part = pcPartStanza(partXML)
            if part['n'] == n:
                self.xml.remove(partXML)
        return

registerStanzaPlugin(Iq,pcCostumeStanza)

That should register <costume /> as a substanza of Iq, and <appearance /> and <part /> as substanzas of <costume /> but that doesn't quite seem to work right either, because this doesn't work:

Code: [Select]
cStanza['costume']['appearance']['bodytype'] = pcCostumeDict['BodyType']
That throws a unicode error, implying I'm indirecting incorrectly off of that Stanza object.  I'm beginning to wonder if doing this from literal raw TCP traffic wouldn't have been faster.  It gets off the ground quickly, but past the 50% mark I've been fighting the framework almost as often as using it.  Something simple like "show me every Iq stanza that arrives on the wire" should be bulletproof.  It ain't.

JSON would have, in fact, been quicker.  :p

FloatingFatMan

  • An Offal
  • Elite Boss
  • *****
  • Posts: 1,178
  • Kheldian's Forever!
Re: Technical side discussion
« Reply #162 on: July 19, 2015, 07:15:41 AM »
JSON would have, in fact, been quicker.  :p

VINDICATION!! :P

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #163 on: July 19, 2015, 08:23:49 AM »
One problem solved: the reason why this doesn't work:

Code: [Select]
cStanza['costume']['appearance']['bodytype'] = pcCostumeDict['BodyType']
Is because while some stuff implies it works, it has no chance of working.  Looking at the way the code works carefully, its now clear that cStanza['costume']['appearance'] cannot be further subscripted because you can only assign it a substanza.  So you're supposed to do this:

tiq = xmpp.make_iq(id="abc123")
ap = pcAppearanceStanza(None, tiq['costume'])
ap['bodytype'] = '1'

That generates <iq id="abc123"><costume xmlns="pc:costume"><appearance bodytype="2" /></costume></iq>

I'm pretty sure I can now send the correct response to Iq requests I'm still not getting.  The problem was figuring out what the heck the library writer was trying to accomplish, and doing that particular gyration with substanza classes proves to me the library writer was completely insane.  It only makes sense if you write romantic love letters as XMPP xeps.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #164 on: July 19, 2015, 08:32:59 AM »
VINDICATION!! :P

How many times did you hit F5 before that happened?

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #165 on: July 19, 2015, 03:24:39 PM »
Unfortunately, there's no way to know if your hash is correct just by looking at it. 

Yes, and that's pretty much the reason why a testbed with a reference implementation is a useful, even necessary tool. The only real test for correctness is that you prove that someone else using your input achieves the same output.

To discover whether you were hashing correctly, Codewalker had to run your costume through his encryption routines. He even modified his normal process to do things the same way that you did them, even though technically it wasn't necessary to do that. Doing so revealed that *he* had accidentally introduced an unintended factor into the hash, that he otherwise would not have found.

When I was testing my hashing function, I knew your string and your results; your input and output. That meant that using your input I should get your output. When I didn't, I was able to run through my code and discover the typo where I'd used a dict index of 'Chest ' instead of 'Chest'.

With access to your costume, your string, and your final hash, I debugged that in a couple of minutes. Without those things, it would have taken considerably longer and I would have been obliged to wait until I was running the bot on the wire, realizing that Paragon Chat was not displaying my bot, and then trying to figure out which of the moving parts wasn't moving quite right.

That's all I was saying - That a lone developer without the support of a conversation like this one would be unable to verify his costume hash. The best he could do is run his bot and say, "Well, that didn't work..."

A reference program with a bunch of known costumes and known hashes, though, would give a bot writer the inputs and outputs he needed to prove the accuracy of his own hash routine.

As for the costume hash, it's true that I initially misunderstood its purpose. That's mostly because I didn't see much point in having a costume hash if it wasn't some sort of global identifier for a given combination of costume parts. Trusting anybody's costume hash to give you a "valid" costume is the same as trusting a self-signed security certificate. The only validation happening is that you received all the data in the costume stanza. TCP/IP and your XMPP framework already handle that validation for you.

If the value of a costume hash is that it gives you a unique identifier for filing the costume in your cache, then you could dispense with all of the encoding and just provide "mybot=costume_french_maid_123"

In any case - a reference implementation would avoid the need for a developer to test against a live server with a dozen other possible complications affecting the test.


FloatingFatMan

  • An Offal
  • Elite Boss
  • *****
  • Posts: 1,178
  • Kheldian's Forever!
Re: Technical side discussion
« Reply #166 on: July 19, 2015, 04:00:49 PM »
How many times did you hit F5 before that happened?

I would never hit F5.











I caress it, lovingly. :P


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #167 on: July 19, 2015, 06:06:15 PM »
I'm pretty sure I can now send the correct response to Iq requests I'm still not getting.  The problem was figuring out what the heck the library writer was trying to accomplish, and doing that particular gyration with substanza classes proves to me the library writer was completely insane.  It only makes sense if you write romantic love letters as XMPP xeps.

That's the main reason I went with libstrophe for Paragon Chat. Yes, it's a bit more work since I have to implement a lot of the higher level XMPP logic myself. I can't just tell it to join a MUC room, I have to actually build the stanza to do so and handle the results. But I have complete control over every bit of XML sent or received and don't have to worry about figuring out what the library author was trying to do to 'protect' me. The library just handles login and very basic stanza construction / parsing to make things easier.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #168 on: July 19, 2015, 06:52:52 PM »
That's the main reason I went with libstrophe for Paragon Chat. Yes, it's a bit more work since I have to implement a lot of the higher level XMPP logic myself. I can't just tell it to join a MUC room, I have to actually build the stanza to do so and handle the results. But I have complete control over every bit of XML sent or received and don't have to worry about figuring out what the library author was trying to do to 'protect' me. The library just handles login and very basic stanza construction / parsing to make things easier.

If I was writing in C, that's what I would have used.  I tend to prefer minimal libraries, unless I am literally doing only what the library explicitly is designed to do. SleekXMPP was written by someone who would have been a LISP programmer if they could only find a good parenthesis autocompleter.  There's a lot of cleverness in it that gets confusing if you're not the author of the cleverness and your primary documentation is sample code.

I suspect there's just one last bit of cleverness I have to figure out.  Although there's also some XMPP issues I can loop back around to figure out while I'm thinking about that.  I need to read the RFCs carefully because I'm still not sure I'm doing Presence quite right.  If the bot enters second, its fine.  If the bot enters first, its fine.  If the bot leaves and comes back, it doesn't always reappear.  And if I try to update Presence, that update doesn't actually get sent to PC.  I seem to recall something in the spec about Presence updates being sent to roster instead of the room, but my memory is a little hazy there.

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #169 on: July 19, 2015, 07:15:01 PM »
And if I try to update Presence, that update doesn't actually get sent to PC.  I seem to recall something in the spec about Presence updates being sent to roster instead of the room, but my memory is a little hazy there.

Yes, if you broadcast presence to the server with no 'to' attribute, it only goes to the roster. It doesn't automatically update the presence in the rooms that you're in. You have to send directed presence to the room to update everyone who's in it. You'll notice that when you do a costume change in Paragon Chat, it re-sends presence several times to different destinations. A couple of those could probably be avoided now that <character> isn't sent to global channels, but are harmless as it is.

Keep in mind that MUC is usually implemented as a standalone service on a subdomain and doesn't necessarily have any special privileges, so it can only see stanzas that are explicitly sent to its subdomain.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #170 on: July 19, 2015, 07:31:00 PM »
Yes, if you broadcast presence to the server with no 'to' attribute, it only goes to the roster. It doesn't automatically update the presence in the rooms that you're in. You have to send directed presence to the room to update everyone who's in it. You'll notice that when you do a costume change in Paragon Chat, it re-sends presence several times to different destinations. A couple of those could probably be avoided now that <character> isn't sent to global channels, but are harmless as it is.

Keep in mind that MUC is usually implemented as a standalone service on a subdomain and doesn't necessarily have any special privileges, so it can only see stanzas that are explicitly sent to its subdomain.

Actually, the problem is more subtle than that.  I don't broadcast presence without destination anymore, in fact, I only broadcast presence to the room(s) I'm in.  But sometimes, I notice when Paragon Chat logs in, it gets an abbreviated Presence for the bot that is missing the "pc:xxx" namespace stuff.  I am wondering, though, if the problem is specific to the way I'm testing the bots: sometimes the bots crash, and when they do they leave zombie connections on Openfire.  Its possible Openfire is getting confused by the bot constantly relogging without doing any of the appropriate protocol to even discover if nickname is currently occupied (which is another disco I have to implement).

I should have just made that ski slope bot.

FloatingFatMan

  • An Offal
  • Elite Boss
  • *****
  • Posts: 1,178
  • Kheldian's Forever!
Re: Technical side discussion
« Reply #171 on: July 19, 2015, 08:21:33 PM »
I should have just made that ski slope bot.

And 5 minutes after you had, you'd be doing this bot! You know you're never happy with the easy route. *grins*

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #172 on: July 19, 2015, 08:41:30 PM »
For backward compatibility shouldn't newer Paragon Chat clients advertise the old style hash as costume= and the newer hash as newcostume= or costume6=?  That way older clients will simply ignore the newcostume attribute and default to using the older one.

That was my first idea and would be simpler. However after some thought I realized that it would cause an OCD violation if the name of the attribute for the life of the project was changed due to a previous implementation briefly existing during what is effectively open beta.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #173 on: July 20, 2015, 03:09:26 AM »
That was my first idea and would be simpler. However after some thought I realized that it would cause an OCD violation if the name of the attribute for the life of the project was changed due to a previous implementation briefly existing during what is effectively open beta.

The flip side is: what happens if you have to change it again?

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #174 on: July 20, 2015, 07:47:54 PM »
No question now sleekxmpp does something a little different when it comes to iq/get queries.  Even when I try to register a callback for *all* Iq stanzas (i.e. <iq />) all I see in the debug logs are the replies to Iq queries (like replies to my own requests for roster).  I never see *any* iq/get stanzas at all.  Not even the trivially simple iq/appearance one.

Looking at ClientXMPP, BaseXMPP, and XMLStream, I don't see any reason why a simple registerHandler wouldn't work on Iq stanzas no different than any other, so it must be something not in the actual event handler but something in the way the library registers its own handlers to Iq.  Maybe their own handlers somehow override all iq/gets and thus they never get passed to me, although that *also* appears to violate the actual documentation of the library.  It also seems to violate all of the sample code for custom iq.

Anyone wanna jump right on in and start reading the iq source with me, I won't mind, honest.  I suspect its also possible this isn't in the base handlers, but a side effect of something going on in the xep plugins.  Oh well, maybe I'll have it working by next weekend.

FloatingFatMan

  • An Offal
  • Elite Boss
  • *****
  • Posts: 1,178
  • Kheldian's Forever!
Re: Technical side discussion
« Reply #175 on: July 20, 2015, 07:50:20 PM »
I'm probably going to regret this but... Wouldn't it be an idea to get your bot running on the same XMPP server as Titan are using for now, and then see about alternatives later?

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #176 on: July 20, 2015, 08:16:53 PM »
I'm probably going to regret this but... Wouldn't it be an idea to get your bot running on the same XMPP server as Titan are using for now, and then see about alternatives later?

SleekXMPP is a python library for making XMPP programs.  Its not an XMPP server.  I'm using Openfire as my server, the same server software Titan uses.  And it works perfectly fine with minimal set up actually: Codewalker even programmed Paragon Chat to automatically create any conference room it needs to represent a zone if it doesn't already exist (your Paragon Chat account becomes the admin of that room as a side effect, but who cares on a dev server).

If you want a test environment, on Windows you can run Openfire, then run Paragon Chat, then run Paragon Chat a second time with -localdb and -localhost switches (so it uses different databases and different IP addresses to keep them separate), and have an entire server all to yourself with just two "players" logged in and your bot, so you can test players interacting with bots and bots observing players interacting with each other.  It helps to have two players logged in sometimes because that way I can see precisely what Codewalker programmed PC to do - even if he makes a typo in his descriptions, Paragon Chat itself will tell you exactly what it was told to do.

Also it just occurred to me that I'm an idiot, because instead of looking at sample code or even the source code of sleekxmpp, what I should have been doing this weekend is realizing that xep 0199 (XMPP Ping), which I enable to satisfy Codewalker's xep prereqs, is actually a sleekxmpp plugin.  Now I have to wait until I get home to look at that (I don't have my full sources at work, because if I did I'd never get anything done at work).

FloatingFatMan

  • An Offal
  • Elite Boss
  • *****
  • Posts: 1,178
  • Kheldian's Forever!
Re: Technical side discussion
« Reply #177 on: July 20, 2015, 08:18:17 PM »
See, that's what I get for not looking up what SleekXMPP was. :P  I presumed it was a different server you were using for a test bed.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #178 on: July 20, 2015, 08:46:06 PM »
See, that's what I get for not looking up what SleekXMPP was. :P  I presumed it was a different server you were using for a test bed.

Should have been reading my dev commentary in between all those F5s.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #179 on: July 21, 2015, 05:44:03 AM »
Success!  Sort of!

Apparently the reason why none of my stanza filters were working was because I was never getting the stanzas in the first place.  Weirdly, the bot was logged into the meta channel as "unavailable."  I'm not sure why, but it survived multiple stops and restarts.  Must have been something sticky in Openfire, even though I have no code anywhere that ever sends unavailable Presence.  Once I unstuck that, I actually started to get Iq/costume stanzas.

Which of course meant for the first time ever, the code I was using to parse them was tested, and of course absolutely none of it worked.  It took a while to realize that sleekxmpp really, really wants you to reply to Iq stanzas, and not generate replies to those stanzas.  Meaning: if you get an iq@type=get stanza X, bad things will happen if you try to create a reply Y that happens to have the right data, because sleekxmpp will never know what happened to X.  It wants you to construct a reply Y that includes everything *except* the Iq wrapper, and do a x.reply().setPayload(Y.xml).  Once I fixed that, and a few more squirrely nested stanza bugs, I started to get iq costume gets and replies working.

Its such a little thing, but an important one.  Without any cheats whatsoever, I now have a visible bot:



The bot no longer relies on the cheat of using a costume identical to the player to be visible.  It loads its costume from an I24 costume file, converts it into Paragon Chat data format, generates a hash, and sends the computed contents upon request, with no hardcoding whatsoever.  There's a lot I want to experiment with, but first I'm going to clean up my code.  There are now more DEBUG statements in the bot than actual code, all trying to track down my inability to process messages I was never getting in the first place.  I should also loop back and try to figure out how many dozens of steps I skipped over in the XMPP protocol that might be causing issues, like sometimes logging in unavailable.

So let's see, from release to now ... thirteen days to make a visible Paragon Chat bot.  That's embarrassingly long even averaging only a couple hours per night.  I must be getting senile.  Still, a bot in about thirty hours means I'm not totally useless yet.  And the roadmap worked fairly well.  So congratulations, Codewalker, your documentations works.

Man, if we had gotten this working when the game was still alive TopDoc would have farmed the crap out of the game with it.