Author Topic: programmatically obtaining badge checklist for character  (Read 14059 times)

wilcoxon

  • Underling
  • *
  • Posts: 7
programmatically obtaining badge checklist for character
« on: May 29, 2009, 05:38:07 AM »
I looked at the Private XML Feeds but they just provide basic character info.  An XML checklist would be awesome.

A long time ago (under previous version of CIT), I had a script that grabbed the HTML checklist and parsed it.  I just updated it so it should work with the current CIT.

However, login seems to be an issue.  Whenever I try to grab http://cit.cohtitan.com/manage/checklist/xxxx, I end up with the login page instead of the checklist.  This happens whether using LWP in Perl or wget and when supplying a cookie file (exported from Firefox while logged in) and/or username/password auth.  Any ideas how to be able to programmatically grab the checklist?

SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #1 on: May 29, 2009, 06:26:39 PM »
I looked at the Private XML Feeds but they just provide basic character info.  An XML checklist would be awesome.

A long time ago (under previous version of CIT), I had a script that grabbed the HTML checklist and parsed it.  I just updated it so it should work with the current CIT.

However, login seems to be an issue.  Whenever I try to grab http://cit.cohtitan.com/manage/checklist/xxxx, I end up with the login page instead of the checklist.  This happens whether using LWP in Perl or wget and when supplying a cookie file (exported from Firefox while logged in) and/or username/password auth.  Any ideas how to be able to programmatically grab the checklist?

Which cookie are you passing in?  Also, have you tried programmaticly logging in via the LWP or wget?  Might be worth something there. 

I have added the XML badgelist to "The List", but as you can see, the list is a bit long.


wilcoxon

  • Underling
  • *
  • Posts: 7
Re: programmatically obtaining badge checklist for character
« Reply #2 on: May 29, 2009, 07:44:31 PM »
I'm passing in whichever cookie(s) it asks for.  I dumped my Firefox cookies and then removed all non-cohtitan cookies from the file (leaving 5-6 cookies).

I'll probably look at programmatically logging in next if nobody has simpler, more direct suggestions.  It was way too late last night when I quit working on it as it was.

wilcoxon

  • Underling
  • *
  • Posts: 7
Re: programmatically obtaining badge checklist for character
« Reply #3 on: May 30, 2009, 08:18:17 PM »
Cute.  CIT uses Flash to handle the login.  As far as I know, there is no way to programmatically interact with Flash.

Is there any possibility of not using Flash for the login or of allowing username and password to be passed in as GET arguments so that it is possible to retrieve the checklist outside of a manual browser?

I still have no idea why the cookies don't seem to be working in LWP or wget.

Then again, XML badgelists (per character) would alleviate the need to login or parse HTML.

SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #4 on: May 30, 2009, 10:09:30 PM »
Cute.  CIT uses Flash to handle the login.  As far as I know, there is no way to programmatically interact with Flash.

Is there any possibility of not using Flash for the login or of allowing username and password to be passed in as GET arguments so that it is possible to retrieve the checklist outside of a manual browser?

I still have no idea why the cookies don't seem to be working in LWP or wget.

Then again, XML badgelists (per character) would alleviate the need to login or parse HTML.

It isn't flash, it is AJAX.  The only bit that might be flash is the loading wheel.


wilcoxon

  • Underling
  • *
  • Posts: 7
Re: programmatically obtaining badge checklist for character
« Reply #5 on: May 30, 2009, 10:26:23 PM »
Hmm.  That's odd.  The loading wheel definitely looks like the Flash one and the FlashBlock Firefox addon offers to block the popup login window (which I've never seen it do for anything that isn't really Flash).

That's even odder.  I just tried it again and the FlashBlock options only show up in the context menu sometimes.

AJAX isn't much more program friendly than Flash typically (I'll look at the JavaScript just in case) so I guess I'm stuck manually downloading the checklist to parse until there is XML checklists.  :(

Steiner

  • Elite Boss
  • *****
  • Posts: 1,602
    • Steinerd.com
Re: programmatically obtaining badge checklist for character
« Reply #6 on: May 31, 2009, 04:01:55 AM »
lol, it's Javascript jQuery Ajax calls.

Try sending the call to http://cit.cohtitan.com/login/ajax or http://cit.cohtitan.com/login.html.

It's set to use javascript to forward a successful log in to manage, but since you're just trying to set sessions... should bounce back the right stuff.
~Steinerd

SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #7 on: June 08, 2009, 04:41:05 AM »
It's only a start, but the checklist no longer requires you to be logged in.  It also has a new address.  You can find it at http://cit.cohtitan.com/checklist/{charid}.  The checklist also respects the character's hidden status.  If the character is hidden, you will need to be logged in to see the list.

I will still be putting out an XML version, and the shorts version, just later.


wilcoxon

  • Underling
  • *
  • Posts: 7
Re: programmatically obtaining badge checklist for character
« Reply #8 on: June 08, 2009, 05:21:06 AM »
Thanks.  I'll check it out tomorrow (when I'm more awake).  That sounds like a great start (and should be sufficient for what I want until an XML checklist is done).

Steiner

  • Elite Boss
  • *****
  • Posts: 1,602
    • Steinerd.com
Re: programmatically obtaining badge checklist for character
« Reply #9 on: June 08, 2009, 03:22:10 PM »
Yea, we tried that, as Nicster said, "... it's making the server cry". lol

The idea was to have one grand XML feed for all the characters on the account using the same key the private character listings use... I have like 3 characters up on CIT right now (Q&A reasons) and even that took a bit to load.

We'll figure something out... I'm thinking along the lines of JSON, it's way more lightweight and any OOP language can decode it, including PHP parse.
~Steinerd

SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #10 on: June 08, 2009, 04:22:53 PM »
Yea, we tried that, as Nicster said, "... it's making the server cry". lol

The idea was to have one grand XML feed for all the characters on the account using the same key the private character listings use... I have like 3 characters up on CIT right now (Q&A reasons) and even that took a bit to load.

We'll figure something out... I'm thinking along the lines of JSON, it's way more lightweight and any OOP language can decode it, including PHP parse.
I plan on having individual feeds for each character.  Just a matter of figuring out the best way to pseudo-secure it (similar to the other XML feeds), and creating a UI that doesn't suck.


Shorts

  • Minion
  • **
  • Posts: 47
Re: programmatically obtaining badge checklist for character
« Reply #11 on: June 10, 2009, 02:11:59 PM »
It's only a start, but the checklist no longer requires you to be logged in.  It also has a new address.  You can find it at http://cit.cohtitan.com/checklist/{charid}.  The checklist also respects the character's hidden status.  If the character is hidden, you will need to be logged in to see the list.

I will still be putting out an XML version, and the shorts version, just later.
Awsume.  ;D
Excreta ergo sum.



SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #12 on: June 10, 2009, 02:39:24 PM »
Awsume.  ;D
I guess I should also point out that I have the initial "short" version up at http://cit.cohtitan.com/checklist/short/{charid}

The "tip" listed will still need to be adjusted, but this is a start.


eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,291
Re: programmatically obtaining badge checklist for character
« Reply #13 on: June 10, 2009, 02:47:51 PM »
Ooo.  I like that.
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Shorts

  • Minion
  • **
  • Posts: 47
Re: programmatically obtaining badge checklist for character
« Reply #14 on: June 11, 2009, 07:57:32 PM »
I guess I should also point out that I have the initial "short" version up at http://cit.cohtitan.com/checklist/short/{charid}

The "tip" listed will still need to be adjusted, but this is a start.
A good start indeed. Nice work.  :)

I hope you'll be working in a short description of the requirements. I'd prefer them over tips to be honest. Some tips can be a bit redundant, but if they can be fitted I have no quarrel.  ;)

Eg: Devoted (331) - 27 months
      Exalted (1144) - Pay off 10,000,000 Debt - Keep Dying and working off your Debt.
Excreta ergo sum.



SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #15 on: June 11, 2009, 11:04:40 PM »
A good start indeed. Nice work.  :)

I hope you'll be working in a short description of the requirements. I'd prefer them over tips to be honest. Some tips can be a bit redundant, but if they can be fitted I have no quarrel.  ;)

Eg: Devoted (331) - 27 months
      Exalted (1144) - Pay off 10,000,000 Debt - Keep Dying and working off your Debt.


The "tip" listed will still need to be adjusted, but this is a start.
That's what I meant ;)

It is just going to take a bit for us to sit down and actually change every tip entry for basically every badge in the DB.


Shorts

  • Minion
  • **
  • Posts: 47
Re: programmatically obtaining badge checklist for character
« Reply #16 on: June 12, 2009, 07:52:07 AM »
It is just going to take a bit for us to sit down and actually change every tip entry for basically every badge in the DB.
If there's anything I can do to help, let me know. :)
Excreta ergo sum.



Tazhyngarth

  • Elite Boss
  • *****
  • Posts: 2,085
    • Titan Network
Re: programmatically obtaining badge checklist for character
« Reply #17 on: June 15, 2009, 01:47:23 PM »
Changing them isn't so hard, just a little lengthy for time usage, but we can get it done no sweat.

SaintNicster

  • Elite Boss
  • *****
  • Posts: 865
Re: programmatically obtaining badge checklist for character
« Reply #18 on: September 23, 2009, 09:34:57 PM »
*blows the dust away from this thread*

XML and JSON lists are now available.  Also, I have linked the "Short" version of the checklist from http://cit.cohtitan.com/checklist/{char_id}

Still haven't really done anything for the missing/long tip text, but hopefully that'll be fixed soon.


wilcoxon

  • Underling
  • *
  • Posts: 7
Re: programmatically obtaining badge checklist for character
« Reply #19 on: September 24, 2009, 04:25:29 AM »
Wow!  That's great!  I'm glad I've been distracted and didn't get any further on my HTML list parser - now I can just trivially use the XML or JSON checklist.

Thanks much.