Author Topic: Technical side discussion  (Read 164723 times)

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #320 on: August 09, 2015, 01:07:19 PM »
The costume stanza is just an appearance stanza and a bunch of part stanzas - Create a new costume stanza variable, make an appearance stanza, append it to the costume, make each part stanza and append it to the costume, then append the costume to the IQ reply stanza.

That is precisely the solution I was trying to avoid, as it took a lot of fiddly reformatting to turn my already valid xml string into a bunch of code ... which just told the library to put my original xml back out.  It was kind of overkill for what amounted to a short-lived test on the road to supporting full costume loading.  But needs must.

Quote
The stanza methods take care of the xml magic.

No real xml magic should have been necessary.  I just needed the library to let me use the xml that I already had.  And it would have ... as long as I was content with having that xml escaped automatically, rendering it useless.


Naturally, I quickly found out how to do this as soon as I no longer needed to.  Sleek may or may not provide support for what I was wanting, but python's xml library (which sleek is built on) does.  It's actually pretty simple, so for posterity:
Code: [Select]
from sleekxmpp.basexmpp import Iq
from xml.etree import ElementTree

bob=Iq()  # Just for a demonstration of the idea, since there's no actual stream here
tom = "<costume><appearance nosejob='1' /><part n='1' tex='Costume Data!' /></costume>"

# this works as long as the xml string has a root element.  take out <costume></costume> and it complains.
tomxml = ElementTree.fromstring(tom)
bob.setPayload(tomxml)
print(bob)

The snippet prints:
<iq xmlns="jabber:client" id="0"><costume><appearance nosejob="1" /><part n="1" tex="Costume Data!" /></costume></iq>

(Which is obviously not full-fledged Paragon Chat format, but there's nothing that I know of preventing me from sticking pc:namespaces or whatever in the string.)

The reason I couldn't figure it out before was that I was looking in the wrong place; I was expecting sleek to have this capability.  While I had a suspicion that I could get around it, as previously mentioned, I didn't want to go Standard Library delving in detail to sort out how. I figured I'd end up having to do something like reimplement _set_sub_text, only without the escaping, and that was even more work than the method that sleek wants.  I wasn't expecting it to take literally five minutes to just dodge the entire issue, including the time to type the above test code.


Quote
You can even do like I did and save out myCostume.xml to a file on disk.

Are you serializing the xml objects or dumping the xml text to a file?  If it's the latter, I suspect you'll need to do some parsing like the above to make use of the files.


Quote
You're going to want to be able to load a .costume file sooner rather than later, so you might as well make that your next order of business.

That is true.


Quote
I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up.

That sounds like fun.
« Last Edit: August 09, 2015, 07:32:55 PM by Dyne »
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #321 on: August 10, 2015, 05:19:15 AM »
Naturally, I quickly found out how to do this as soon as I no longer needed to.  Sleek may or may not provide support for what I was wanting, but python's xml library (which sleek is built on) does.  It's actually pretty simple, so for posterity:
Code: [Select]
from sleekxmpp.basexmpp import Iq
from xml.etree import ElementTree

bob=Iq()  # Just for a demonstration of the idea, since there's no actual stream here
tom = "<costume><appearance nosejob='1' /><part n='1' tex='Costume Data!' /></costume>"

# this works as long as the xml string has a root element.  take out <costume></costume> and it complains.
tomxml = ElementTree.fromstring(tom)
bob.setPayload(tomxml)
print(bob)

The snippet prints:
<iq xmlns="jabber:client" id="0"><costume><appearance nosejob="1" /><part n="1" tex="Costume Data!" /></costume></iq>

(Which is obviously not full-fledged Paragon Chat format, but there's nothing that I know of preventing me from sticking pc:namespaces or whatever in the string.)

The reason I couldn't figure it out before was that I was looking in the wrong place; I was expecting sleek to have this capability.  While I had a suspicion that I could get around it, as previously mentioned, I didn't want to go Standard Library delving in detail to sort out how. I figured I'd end up having to do something like reimplement _set_sub_text, only without the escaping, and that was even more work than the method that sleek wants.  I wasn't expecting it to take literally five minutes to just dodge the entire issue, including the time to type the above test code.

The sleepxmpp documentation does note this in the documentation for Stanza Objects:  http://sleekxmpp.com/api/xmlstream/stanzabase.html.  If you search that page, at the bottom you'll see the listed methods for Stanza objects, including setPayload().  It probably complains if you don't have the root element because without it the string might be valid XML but it cannot be converted to a valid (single) sleekxmpp XML object.  You'd then have two independent XML objects (attribute and part) that need to be passed as a sequence (with each object in a different element), not a single string.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #322 on: August 10, 2015, 05:26:45 AM »
Pushing towards something that gives a tangible result gives me more motivation. At the moment I'm taking a break from the bot code proper, and I'm working with a Markov Chain-based conversation bot that's seeded with all of the City of Heroes lore that I can dig up. I actually found that I could pull up practically an entire five or six year archive of the old CoH forums from the Wayback, but I don't think I've got the drive to write a parser that would spend the time to download each page, download all the individual posts, and then extract just the text of each post. What I do have is all of the lore bible docs that were released after the game closed, and all of the Paragon Times, and Know Your Adversary and other lore stuff that was on the official web pages. The initial results are sometimes rather hilarious but there's enough non-useful garbage in the story bible docs that I'm taking the time to edit out all the crap and make a "clean" data file for training the chatbot's "brain".

You're making a UniqueDragon bot?  That's very meta.

I myself was forced to take a break this week, owing to work detonating on me.  I'll probably have dug myself out of vendor-hell by mid-week next week and be able to resume work on the bot.  In my never-ending search for ways to duck trying to make a map parser and space tree search algorithm, I'm thinking of building up my library of bot animation primitives.  Given what I have, and given lack of support for things like ragdoll, I'm going to make a list of animation and animation combinations that replicate some of the behavior I want the bot to be able to perform.  I "know" what I want and mostly how to find it, but I still need to track it all down and test it.  I suspect it won't all exactly work the way I want without some kludging.  I'm thinking of a way to make animation "libraries" composed of sequences of moves correctly timed, so my bot can call them without having to "think" about how to do them once I've hand-tested them.  That's time consuming, but has immediate payoffs as I work, which I kind of need after the past week.

Dyne

  • Lieutenant
  • ***
  • Posts: 54
  • Gluer of Gears
Re: Technical side discussion
« Reply #323 on: August 10, 2015, 10:26:14 AM »
-= Virtue Server - 2004-06-13 to 2012-11-30 =-
hortis publicis gemini in aeternum

Usurper Dyne, Still Heart, River Elemental, Saul Invictus, Grim Grinner

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #324 on: August 10, 2015, 06:11:13 PM »
UniqueDragon! Wow, that takes me back!

I might have to train that into the chatbot, LOL.







Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #325 on: August 10, 2015, 06:22:44 PM »
Yeah, in hindsight, I can see what they were going for.  It wasn't so helpful from the other side, though; I read that at least a dozen times while trying to work this out.  (Mostly it read like a reference to StanzaObject.xml, which I already knew that method took.)

It's quite possible that I'm actually losing the ability to make connections like this, probably due to lifestyle factors.

Honestly, I'm still figuring out sleekxmpp.  Some things work the way I expect, some things don't, and its therefore difficult at time to tell the difference between a bug and library goofiness.  For example, as I mentioned upstream, don't ever do this:

interfaces = set(('id'))

if you want to preserve your sanity.  Sleekxmpp will decide that interfaces is not a sequence with one entry but a single string sequence of entries, and will happily make interfaces with names "i" and "d".  That has to be a bug in the library since its a set**, but not a bug I particularly want to hunt down at the moment.


** If I had to guess, they make it a set to prevent dups, then try to make the entries into an array one element at a time, and when there's only one entry that happens to be a sequence (string), the code gets confused and starts derefing the string.

blacksly

  • Elite Boss
  • *****
  • Posts: 513
Re: Technical side discussion
« Reply #326 on: August 10, 2015, 08:40:51 PM »
UniqueDragon! Wow, that takes me back!

I might have to train that into the chatbot, LOL.

/jranger

No, just kidding, go for it.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #327 on: August 25, 2015, 07:44:17 PM »
Why no updates lately (from me):

1.  Work got busy and complicated, slowing me down from a crawl to whatever it is snails do.

2.  I'm currently at a point in my bot research that involves more research and less coding, so there's less interesting things to talk about.  However, I'm hoping to have something interesting to show in maybe another week.  Thread not dead yet.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #328 on: August 26, 2015, 08:33:38 AM »
This should be obvious to my fellow bot writers in this thread, but for any new bot writers and for the purposes of completeness, there's a new stanza to support the hacks necessary to get some of the travel powers working.  Codewalker can confirm or correct, but the new stanza is:

<hack xmlns="pc:hack"><power name="powername" state="[off|on]" /></hack>

powername currently is either fly or superspeed.  It is almost certainly used to communicate to other Paragon Chat users that entities with these powers enabled should have the appropriate VFX attached to them.  Walk and Sprint do not send hack power state messages, probably because neither power requires any other Paragon Chat user to know it is turned on, because there's no visual difference for characters with these powers toggled on (except the animations they play which is already handled with standard <u /> stanzas.

The current apparent intent is to use <hack /> stanzas to signal to other Paragon Chat users when a particular continuing state is true or false for a particular entity to signal to them that Paragon Chat should visually turn something on or off for that entity.  Unless this model changes, expect eventual extensions to the hacks stanza to support enabling state for anything other than the sender - for example, to turn on continuing VFX for spawned entities (when those become available) or to signal environmental changes (if that ever becomes a possibility).

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #329 on: August 26, 2015, 01:16:31 PM »
Quote
The current apparent intent is to use <hack /> stanzas to signal to anyone who wants to interoperate with Paragon Chat that anything inside of it is a temporary stopgap at best, and the protocol is subject to change as soon as something better is designed.

Fix'd.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #330 on: August 27, 2015, 05:31:02 PM »
Fix'd.

LMAO

I have been busy with some other projects, and then Frontier decided that when my internet service broke that they didn't need to fix it for the better part of two weeks. I'm comfortable with all of the tools by now but I still need access to online reference material and using my phone for that was less fun than playing some of the games I bought from Steam and GoG over the last year and never bothered to play before my Internet went out. (Beyond Good and Evil is pretty good, if you're curious.)

I might get back to the bot project sometime this week.


slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #331 on: August 27, 2015, 06:19:18 PM »
Out of curiosity - is "oldcostume =" going to be the byteswapped version of the costume hash after a version of Paragon Chat comes out that doesn't byteswap it any more?


Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #332 on: August 28, 2015, 01:11:09 PM »
Yes. 0.98 and later prefer oldcostume if it's present, so that when the switch is made of the 'costume' attribute, they will continue to work so long as newer versions still send oldcostume.

Given that the travel capabilities in 0.99 are a strong incentive to upgrade to at least that version, I think we're on track for performing the switch to the new hashing algorithm (which I'll post details about shortly for review / comment) with the 1.0 release.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #333 on: September 05, 2015, 09:18:40 AM »
So its been another week, and I'm still plodding along.  Sorry, nothing much to show yet, but I can say what I've been working on.  I decided to see what it would take to write a generic animation engine for bots (which would have uses beyond bots).  In other words, given a set of inputs, the engine would tell you what animation if any you should be playing/sending to Paragon Chat.  For example, feed it a bunch of sequence bits and a time index, and then for any future time the engine will tell you based on the game sequence database what animation should be playing now.  That kind of thing.  I actually once had an almost fully functioning set of code to do that, but the last time it worked was in I20 and the code is also a mess.  I mean literally a mess.  Take a look at this:

Code: [Select]
def fixedpoint4byte(s):
    fp1 = ord(s[3])
    fp2 = ord(s[2])
    fp3 = ord(s[1])
    fp4 = ord(s[0])
    fpsign = 1.0

    if fp1 == 0 and fp2 == 0 and fp3 == 0 and fp4 == 0:
        return 0

    if (fp1 & 0x80) != 0:
        fp1 = fp1 & 0x7f
        fpsign = -1.0
##        sys.stderr.write("Debug: negative sign triggered in fixedpoint routine\n")

    fpE = fp1 * 2

    if (fp2 & 0x80) != 0:
        fpE = fpE + 1

    fpE = fpE - 0x80

    fpM = (fp2 & 0x7f) * 65536.0 + fp3 * 256.0 + fp4
    fpS = 2 ** (22 - fpE)

#    sys.stderr.write("Debug: fpE:"+str(fpE)+" fpM:"+str(fpM)+" fpS:"+str(fpS)+"\n")
#    print "Debug: fpE:"+str(fpE)+" fpM:"+str(fpM)+" fpS:"+str(fpS),
    fpval = ((2 ** (fpE + 1)) + fpM / fpS) * fpsign
    return fpval

Seriously.  That's the function I wrote, way, way, WAY back in I9 when I was first trying to reverse engineer pigg files.  All I remember was trying to figure out how those numbers were encoded and trying out different theories until I found one that worked (trying to do this before you know what the numbers are supposed to represent yet makes it a bit more interesting).  Of course, this is doing things the really hard way, treating the problem like code breaking.  There are vastly simpler ways to do this, if you're willing to, say, trace the program itself (which didn't occur to me at the time).

This function was the one I was looking for, for a while:

Code: [Select]
def loadAnimTables(piggdict,subdir):

    # subdir = "player_library/animations/male" for male players
    global global_dver
   
    piggfilename = piggdict['PiggFileName']
    piggfilenameshort = piggfilename.split('/')[-1]
    pf_debug = open('C:/CoH/decode/working/'+piggfilenameshort+'.sizes.'+global_dver+'.txt','wb')
   
    anim_dict = {}

    piggf = open(piggdict['PiggFileName'],'rb')
    for pfile in piggdict.keys():
        if pfile[0:30].lower() == subdir:
            piggf.seek(piggdict[pfile]['foffset'])
            fbz = piggf.read(piggdict[pfile]['fsizecomp'])
            fraw = zlib.decompress(fbz)
            fcount = fixedpoint4byte(fraw[520:524])
            sys.stderr.write(pfile+","+str(fcount)+"\n")
            anim_dict[pfile[26:].lower()] = {'frames':fcount}
            pf_debug.write(pfile+": "+str(fcount)+"\n")
        #sys.stderr.write(pfile[0:30].lower()+"\n")
           
    piggf.close()
    pf_debug.close()
    return anim_dict

What does this do?  This scans the pigg files looking for all of the *.anim files, which are animation files.  This part: "fcount = fixedpoint4byte(fraw[520:524])" - that's some very hard-fought for knowledge.  It took about a week for me to figure this out back in I11ish, but that's how long that animation takes to play from beginning to end, in frames.  Why is that important?  Because many animation sequence entries basically say "play until the end, then move to this sequence."  That means the data in the sequence tables itself is insufficient to know how long to play sequence X before moving to sequence Y.  But if you know how long the animation associated with sequence X is, you can time that accordingly**.

With that piece of information, plus a sequence decoder, you could write a set of functions that could load the sequence data and animation data from the CoH client piggs, then perform basic calculations on animation sequences.  Paragon Chat itself must have the ability to do this to do what it does, more or less.  Once I untangle all my old (really, really horrible) code from my pigg diving days, and make it more readable and less embarrassing, I should be able to replicate that functionality in a way that would be useful to bots.  And also other things***.  But its been slow going, first because my pigg decode code is organic, includes code for every version of pigg from I9 to I23 in a giant mish-mash, and I forget what the purpose of some of it was.  And my free time is still fairly limited on what I can spend on it.  But I'ma keep cranking away, and we'll see if I can't make something useful eventually.


** Actually, for things like emotes this is often not necessary because those kinds of animations usually contained explicit start and stop frame counters.  Interestingly it tended to be powers-related animations that had stop=0 meaning play to the end.  Which is why I needed that bit of information to calculate animation-based root times and compare them to powers-based cast times, which oddly no one before me ever seems to have thought to do.

*** Hypothetically speaking, it could be used to spit out custom emote.cfg files for people, based on what sorts of animations they wanted to have.  With a little extra work, it could be used to spit out an emote.cfg that contained every animation connected to every power in the powers database (whether the animation actually worked or not).  Stuff like that.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #334 on: April 19, 2016, 04:00:48 PM »
Well, engaging in a bit of thread necromancy here.

Life happened, I put my bot aside back in August, and never really got back to it. Judging by the state of this thread, that may have happened to some other folks also.

A few days ago I updated to PChat 1.0. I had a bit of time this morning, so I fired up the bot. After re-familiarizing myself with the right way to launch errbot, I fired it up and of course it was invisible.

However, a bit of fiddling to set oldcostume="TheByteswappedCostumeHash" and costume="" set matters right.

Now I just have to re-learn everything I knew about SleekXMPP and Paragon Chat. :-p

Anyway, the reason for the post - Are there any other technical details that a bot should be aware of, given the advent of PChat 1.0?

On another note - I was reading the thread where someone asked about how Null the Gull worked, and I was intrigued.

Maybe this would be crossing a fine line into behavior that constitutes "acting like a game server", but I'm wondering about the difficulty of modifying the NullClick() function to broadcast a message to the metadata room for a clickable object that identifies the object and requests a response to the click.

A bot configured to listen for such requests could respond with an ACK and an XML packet describing what the player should see as a response.

I have no clue about whether this is something that would be able to leverage Null's existing code or if it would require a new "dialog interpreter" module added to Paragon Chat. In the latter case, I'm sure that Codewalker has more than enough on his plate already.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #335 on: April 22, 2016, 03:09:47 PM »
I noticed that my bot is sending its version number as "0.98a". Are there any pluses or pitfalls to updating it to whatever the current protocol version number is? (Probably 1.0?)

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Re: Technical side discussion
« Reply #336 on: April 22, 2016, 07:37:02 PM »
The version number in the presence is only for troubleshooting and for display purposes (on the player info tab). It is not used by Paragon Chat to make any decisions about how it communicates with that player. That's what the "protocol" attribute is for, and it is incremented whenever non-backwards compatible changes are made to the XMPP protocol that other clients need to be aware of.

Probably the best thing would be to put something to uniquely identify your bot in there instead of a real paragon chat version number.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #337 on: April 25, 2016, 07:22:55 AM »
Small victories -

Firstly, having fallen two major versions behind on errbot, I updated my installation and it predictably broke the bot five ways from Sunday.

After a day of tracing down error messages and altering code to accomodate changes in naming conventions and functionality, I once again had my bot hanging around in Atlas Park. In the end it was not a picnic but it was easier than I feared.

Secondly, after another day of debugging both my code and the assumptions the code was originally based on, the bot is now able to load a costume on command.

I need to try loading a costume that is not in my client's cache to really test it, but for the moment, it looks like one more feature checked off of the list. Next up is tracking room occupants and their position relative to the bot.

slickriptide

  • Elite Boss
  • *****
  • Posts: 356
Re: Technical side discussion
« Reply #338 on: April 25, 2016, 02:02:09 PM »
So, I'm not surprised to learn that a NPC costume is an illegal costume for a bot, but I had my hopes.

Question - Is it well understood how the Halloween costume powers functioned? Is there a possibility of duplicating that function in Paragon Chat? I'm only suggesting this if it's more or less a switch inside of the client where the server sent a message that said "replace your current model with this NPC model".

On another note - I learned two things while experimenting with NPC costumes (from the collection that is linked elsewhere in the forum in the thread about  hacking NPC costume bits into the sqlite character database).

One is that launching the bot with a NPC costume as its initial costume caused it to be invisible, but commanding it to change to a legal costume made it visible. Not a surprise, really. I was sort of hoping that if it was invisible that I could still see its chat bubbles in local and have it pretend to be a static NPC, heh. No dice there.

Two, and more interesting, is that starting with a legal costume and changing to an illegal costume caused no visual change for the bot. I would have expected the bot to become invisible with an illegal costume exception in the /whoall. Instead, it retained its previous costume, from a visual standpoint, even though it's presence was being advertised with an illegal NPC costume.

I'm not sure if this is a bug or a feature? (I wouldn't eliminate the possibility that I'm causing it somehow but my review of the bot logs looks like it's sending presence for illegal costumes identically to the way it sends presence for legal costumes.)

It means that if the bot is currently visible and I want to deliberately turn it invisible, that I should leave the zone entirely (leave atlaspark_meta and maybe also leave atlaspark channels) and rejoin with no costume or with an invisible one at any rate.

« Last Edit: April 25, 2016, 04:04:00 PM by slickriptide »

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Technical side discussion
« Reply #339 on: April 25, 2016, 09:25:26 PM »
I suspect that your inability to turn invisible by announcing an illegal costume is due to the various things Codewalker did to try to explicitly prevent weird errors from making players invisible.  My guess is that if you announce a legal costume and everyone in the zone acquires it, announcing an illegal costume causes all of the other players to simply ignore that illegal command.  That means you effectively continue to look like what you last looked like to them.  But if you start by zoning in with an illegal costume, no one has a previously legal cached costume for you, so you have to be invisible.

Unfortunately, I've had to shelve my own work on this project.  I'm currently trying to figure out why FreeBSD's LSI driver (both of them) stall their writes to Intel 3510s under high random load. Work has me chasing a number of really complex systems problems at the moment.