Technical side discussion

[Codewalker]

Weird. It appears to be a cut and paste error somehow, because I still have the notepad window open that I copied it from, and it definitely says "...1_Maskcapes/CapeLongFem.fx1ff0058..." there. But in the post the slash is missing.

[Arcana]

The byteswap seems to fix the mismatch.  I won't be able to check it in-game until tonight, because I need to finish up the actual Iq response stanza to see if I can send this to anyone.  But with luck that's the last impediment to getting this particular element working.  Nice to see I didn't fail at writing a simple parser.  Parsers are like 75% of my programming history.


Wait, now I have to support a version number?  Aw man I was totally ignoring that attribute. 

[Codewalker]

You don't have to, if you support the V2 format when it comes out (I'll post details ahead of time), you can just switch to that and your bot simply won't be visible to anyone running old versions. It's mostly incremental improvements that are not backwards compatible, but shouldn't be hard to adapt what you already have to.

The update check / nag feature will happen before the hash change happens, so hopefully there won't be many people running old versions anyway.

Most likely case is that clients with protocol=6 will use the new hash format in their costume attribute, while also (for a time) providing the old hash format in "oldcostume". protocol=5 clients will only know about the old format, but versions after 0.98 will know to check for "oldcostume" and use it if present, so that things will still work while there's overlap of 1.0 and 0.98 - 0.999 (repeating of course). Versions older than 0.98 won't be supported at all and will lose the ability to see people running 1.0 or later.

I don't expect the transition period to last extremely long, and will probably drop the oldcostume support in 1.1.

[Arcana]

You don't have to, if you support the V2 format when it comes out (I'll post details ahead of time), you can just switch to that and your bot simply won't be visible to anyone running old versions. It's mostly incremental improvements that are not backwards compatible, but shouldn't be hard to adapt what you already have to.

The update check / nag feature will happen before the hash change happens, so hopefully there won't be many people running old versions anyway.

Most likely case is that clients with protocol=6 will use the new hash format in their costume attribute, while also (for a time) providing the old hash format in "oldcostume". protocol=5 clients will only know about the old format, but versions after 0.98 will know to check for "oldcostume" and use it if present, so that things will still work while there's overlap of 1.0 and 0.98 - 0.999 (repeating of course). Versions older than 0.98 won't be supported at all and will lose the ability to see people running 1.0 or later.

I don't expect the transition period to last extremely long, and will probably drop the oldcostume support in 1.1.

For backward compatibility shouldn't newer Paragon Chat clients advertise the old style hash as costume= and the newer hash as newcostume= or costume6=?  That way older clients will simply ignore the newcostume attribute and default to using the older one.

Incidentally, did you figure out why Paragon Chat appeared to be dropping Color3 and Color4 from Part 15?  I can get the hash to match now, but only by editing my costume file to zero out those two colors.  If I load the original costume with nonzero Color3 and Color4, Paragon Chat seems to ignore those two colors; it doesn't send them as part of its <costume /> stanza, and its hash is consistent with those two colors simply vanishing.

[slickriptide]

Okay. I'm clearly doing something wrong with the SHA1 stuff because I put in Arcana's string and get entirely different results than Codewalker posted. (I'm testing Arcana's string because it's a known quantity at this point. Needless to say, my own costumes' hashes are also entirely different than what my XMPP traffic logs say they should be.)

Here's my snippet.py:

Code: [Select]

import hashlib

# Arcana's costume string

snippet = "1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_MaskcapesCapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff"

h = hashlib.sha1() # initializes a new sha-1 digest

h.update(snippet.encode('utf-8')) # the update barfs without some sort of string char encoding

print(h.hexdigest())


The result of this is: 21151b461e3ef228d0c21ce94e9a984761730dfb

Never mind bit swapping. That looks nothing like c9c638d92e341083e0be1e8668659823e1ffb5d9.

Despite this appearing to be straight forward, I'm clearly missing something important, especially since Arcana is working with the identical software and she's getting the right results.

What am I doing wrong?

***edit***

and now I'm really confused - I pasted Arcana's costume string into an online sha-1 encoder and got the identical result that Python's hashlib gave me.

What is it that I'm not "getting" about the instruction to "take the SHA-1 of the combined string"?

[Arcana]

What am I doing wrong?

You're not pasting my string, you're pasting Codewalker's quote of my string, which somehow ended up missing a character.  Note the "capesCapeLongFem.fx" sequence in your string.  It should be "capes/CapeLongFem.fx".  Remove the "/", and the hex hash ends up being 21151b461e3ef228d0c21ce94e9a984761730dfb which is I believe what you are getting.

[slickriptide]

You're not pasting my string, you're pasting Codewalker's quote of my string, which somehow ended up missing a character.  Note the "capesCapeLongFem.fx" sequence in your string.  It should be "capes/CapeLongFem.fx".  Remove the "/", and the hex hash ends up being 21151b461e3ef228d0c21ce94e9a984761730dfb which is I believe what you are getting.

Right. You even posted that later in the thread.

/headsmack

Well, at least that means that any problems in my costume hashes are my own fault rather than my being an idiot at following simple instructions like "put tab A in slot B".

[slickriptide]

Codewalker - Am I understanding correctly that when you put this sha-1 digest: d938c6c98310342e861ebee023986568d9b5ffe1

through a base64 encoding, what you get out the other end is this: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=?

When I encode that I get - ZDkzOGM2Yzk4MzEwMzQyZTg2MWViZWUwMjM5ODY1NjhkOWI1ZmZlMQ==

That's what the online coder/decoder websites I've tried also get. Attempting to decode the base64 costume hash results in non-displayable binary data.

Is there an intermediate step that I'm missing?

[Codewalker]

Codewalker - Am I understanding correctly that when you put this sha-1 digest: d938c6c98310342e861ebee023986568d9b5ffe1

through a base64 encoding, what you get out the other end is this: 2TjGyYMQNC6GHr7gI5hlaNm1/+E=?

When I encode that I get - ZDkzOGM2Yzk4MzEwMzQyZTg2MWViZWUwMjM5ODY1NjhkOWI1ZmZlMQ==

Don't base64 encode the hex string. The hex digits only represent a human-readable form of the "non-displayable binary data" that is the real SHA-1 hash. That's what you need to encode. With hashlib I think it's digest() rather than hexdigest().

[Arcana]

Don't base64 encode the hex string. The hex digits only represent a human-readable form of the "non-displayable binary data" that is the real SHA-1 hash. That's what you need to encode. With hashlib I think it's digest() rather than hexdigest().

Yep.

ch = hashlib.sha1()
ch.update(costume_string)
ch_hash = binascii.b2a_base64(ch.digest())

That will do it.

Note that ch.digest() is a binary string, which means you can't do anything with it but process it with binary-aware functions.  It can include zeros (nulls) with all that implies.

[Codewalker]

Incidentally, did you figure out why Paragon Chat appeared to be dropping Color3 and Color4 from Part 15?  I can get the hash to match now, but only by editing my costume file to zero out those two colors.  If I load the original costume with nonzero Color3 and Color4, Paragon Chat seems to ignore those two colors; it doesn't send them as part of its <costume /> stanza, and its hash is consistent with those two colors simply vanishing.

Are you sure that it's Paragon Chat that's removing those and not the game client when you load the costume? Easiest way to determine that is to see if those colors are 0 or the color in the sqlite database.

[slickriptide]

Groovy!

Thanks to both of you for the advice.

[Arcana]

Are you sure that it's Paragon Chat that's removing those and not the game client when you load the costume? Easiest way to determine that is to see if those colors are 0 or the color in the sqlite database.

Weirdly, those colors are blank in the Paragon Chat database, *but* the costume file was actually *written* by the game client.  What I did was enter the character creator, import the costume, then re-export it.  Maybe the creator itself isn't stripping the colors, but entering the game is what triggers the color strip.  Not sure.  It seems the character creator can both import and export costume files that it doesn't actually apply in full to actual character entities.

Also, multiple <part /> stanzas?  More work than I thought to make those work in sleekxmpp.  Or rather, not so much work as a lot of typing.  I probably won't even get the whole thing in a testable format until tomorrow.  Fortunately, I actually have a stretch of a few hours tonight that nothing is demanding my time, so I'm going to see if I can focus on this for more than an hour.

Its hokey, but I really want to get this working because I really want to see what happens when I try something, because I'm curious what the entire system of client, xmpp server, and paragon chat will do.

[slickriptide]

Paragon Chat seems to be advertising the hash as: Lxdw1TQl8g/dEYfAJNh7LE7oe6c=
I am calculating it as: ycY42S40EIPgvh6GaGWYI+H/tdk=

For the record, the string of data being hashed looks like this:

Code: [Select]

1f6bbaaff-0.85-0.53-0.780shortsskin_tightsminiskirt_11ff0000ff20045ccff1Tightskin_tightsTop_51ff0058ff2d40000ff2V_fem_Head.GEO/GEO_Head_V_Asym_Standard!v_sf_face_skin_head_101ff65cdff2ccca00ff3Bracer_02Skin_Bracer_02aSkin_Bracer_02b1fffd00ff2d40000ff4Hi_Heels_02skin_Hi_Heels_02askin_Hi_Heels_02b1990031ff20000ffff5standardMARTIAL_ARTS_011990031ff21f0000ff6Long_04Long_01aLong_01b1ff0000ff2aa3800ff8Tiara_01Tiara_01aTiara_01b1ccca00ff2d40000ff9TightBaseStar_102fffd4cff15FullCape_Top_011ff0058ff2ff0058ff3ff0058ff4ff0058ff17!X_Valkyrie_Cape_01!Cape_Valkyrie_01_Maskcapes/CapeLongFem.fx1ff0058ff2ff0058ff3ff0058ff4ff0058ff19ShortSkirt_01pleatedplaid_01b1ff0058ff2ff0058ff

Note - Paragon chat version referenced here is 0.97f

So, what's up with the hash that Paragon Chat is using? How "sacred" is it?

I'm asking because we never really addressed this issue of Paragon Chat computing a different costume hash than what Arcana was calculating for the identical costume.

We seem to have established that for the costume shown above, that Arcana's hash is correct (or, rather,the byteswapped version 2TjGyYMQNC6GHr7gI5hlaNm1/+E= is technically correct for now).

Despite that, Paragon Chat was sending her something else entirely when she logged in a character using that costume, as opposed to the bot being the one wearing it.

It gets murkier - since Arcana posted Arcanabot's costume file, I used it as a test case for my own parser. My parser can open Arcanabot's costume and end up with the exact same hash that we've denoted as "correct". So far, so good.

But when I login a female character and dress her up as Arcanabot, Paragon Chat is sending me a costume hash of i7yXq1nBm+PltncTA+pg50DfGRA=.

Clearly, that's different than the costume hash spec and different from what Aracana's own Paragon Chat client sent to her own server.

So, what's going on here? Should we be using the Paragon Chat client as a yardstick for judging what's correct from what's incorrect? If not, what yardstick do we use?

****EDIT****

Just pointing out that I haven't forgotten what Arcana was mentioning about the color discrepancy on part 15. However, if that was itself a consistent thing then my chat client ought to be producing the same hash that her chat client was producing, even if it differed from the "vanilla" hash.

[Codewalker]

I'm asking because we never really addressed this issue of Paragon Chat computing a different costume hash than what Arcana was calculating for the identical costume.

No, Paragon Chat was computing a hash for a different costume. The COH game client does not maintain costumes in a particularly stable format, often changing the case of strings or changing / removing colors that "don't matter" (i.e. aren't actually used by the costume part in question). Paragon Chat will calculate different hashes for these variations, because while they're visually identical to each other, they are not the same costume.

The hash that Arcana calculated, once adjusted for the byte swapping issue, is valid for the exact costume file that she posted. The costume that she was receiving after loading that file into the game was not exactly the same as the costume files.

Just calculate the hash for the costumes your bot is sending and you'll be fine. Don't worry about whether or not they exactly match one loaded into the game.

[Codewalker]

Weirdly, those colors are blank in the Paragon Chat database, *but* the costume file was actually *written* by the game client.  What I did was enter the character creator, import the costume, then re-export it.  Maybe the creator itself isn't stripping the colors, but entering the game is what triggers the color strip.  Not sure.  It seems the character creator can both import and export costume files that it doesn't actually apply in full to actual character entities.

It's quite possible that it's not the creator that's doing it, but rather the network protocol it transits over to make it to PC and back. It's quite... prematurely optimized IMO. I mean I know they were probably worried about zoning in and needing a hundred costumes sent over a 56k modem, but once you're in game how often do you really need to receive new costumes...

Alternatively, those colors may be getting inserted by the client somehow after the costume is received, but before it's written to the .costume file.

[Arcana]

So, what's going on here? Should we be using the Paragon Chat client as a yardstick for judging what's correct from what's incorrect?

Well, if you want your bot to work, yes (since you have to agree with Paragon Chat clients to be visible to them).

Just to amplify what Codewalker said, what I noticed is a curious anomaly, but its not a fatal issue for bots.  The issue I'm seeing is that if I give a bot a costume file and also give that costume file to a City of Heroes / Paragon Chat character, somehow they don't always end up with exactly the same costume data.  The bot gets literally what's inside the file.  But the CoH entity sometimes gets something different.  Which means the CoH entity will send a different hash for its version of that costume than what the bot would.  But the important thing is that so long as your hash matches *your* costume data, the costume data you literally send, Paragon Chat will validate the hash against what you send correctly and display it.

To put it another way, suppose I had a costume file with the word "cape."  But I noticed that when I make a character in CoH/PC and load that costume file, somehow "cape" becomes "Cape" with a different capitalization.  The Paragon Chat database shows "Cape" and when I watch the debug logs Paragon Chat sends "Cape" in its costume stanza.  The hash is consistent with "Cape" and not "cape."  That doesn't really matter to my bot, because if my bot sends "cape" even though its doing something different than what CoH itself does, so long as my hash matches "cape" and I send "cape" the hash will be consistent and Paragon Chat will be happy.

Why its happening is one of life's mysteries, and it means you cannot predict what the hash will be for a specific costume *file* when *City of Heroes* (via Paragon Chat) sends it.  But you can predict what the hash will need to be when your bot sends it, because you know how your bot will send it.

[Arcana]

Bah, another problem, this time with Iq stanzas.  I can now compute my own hashes and send them accurately, but when Paragon Chat requests them...

conn DEBUG SENT: <iq id="0000001d-4d5a05cf" to="arcanabot@core3770/ArcanaBot_meta" type="get"><costume hash="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" xmlns="pc:costume"/></iq>

xmpp DEBUG RECV: <iq id="0000001c-4d5a05cf" to="arcanacoh@core3770/Violet" type="error" from="arcanabot@core3770/ArcanaBot_meta"><costume hash="Lxdw1TQl8g/dEYfAJNh7LE7oe6c=" xmlns="pc:costume"/><error code="503" type="cancel"><service-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></iq>

Its probably a bug somewhere in how I'm registering Iq request stanza callbacks (almost certainly because I don't even see them show up in any of my message handlers), but I'm not sure where that is yet.  Because I registered them exactly the same way I registered Presence and Message stanzas, I presume the problem is that I *shouldn't* register them in precisely the same way because of something unique to Iq handling.  Its possible one problem is that sleekxmpp appears to want queries to show up in <query> blocks by default, although I should be able to override that with the appropriate stanza matching.  But that ain't going nowhere and I've exhausted every method of matching.    I might be forced to RTFM again.

[Codewalker]

The XMPP spec requires that an Iq 'get' MUST be responded to, by either a 'result' or an 'error'.

It wouldn't surprise me if sleekxmpp has a special form of registering to handle Iqs, so that it can take care of sending back the 'error' for otherwise unhandled ones.

[slickriptide]

Just calculate the hash for the costumes your bot is sending and you'll be fine. Don't worry about whether or not they exactly match one loaded into the game.

That sounds good.

I can see how future bot programmers are going to find it very helpful to have a pre-made test bed that includes Max Slider Man and Crazy Slider Girl and Red Eyes Guy and so on as a way of validating their costume hashing code.

The important thing then is not that every given costume has a unique hash(because one costume can be "compiled" in an assortment of ways that can affect the hash), it's that a given costume stanza received off the wire can be validated by the hash presented in the associated <Character> stanza. Does that sound correct?


***EDIT***

Reading back now it looks like that's pretty much what Arcana said, so, good. I'm going to have to come up with some sort of testbed for costume hashes, though. If we can't just compare our hash to the hash presented by PC, then there's no practical way to tell whether your hash is actually correct or not. I only felt fairly confident about my hashing function after Codewalker certified Arcana's results and I was able to replicate her results. When you look at your function spit out a string of base64, it isn't intuitively obvious whether that string is correct or incorrect.