Author Topic: Interwiki Transclusion and TOCleft  (Read 47750 times)

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Interwiki Transclusion and TOCleft
« on: June 22, 2009, 03:19:50 PM »
I'm noticing some interesting behavior over on the Ourportal when I transclude Paragon Wiki content in conjunction with a TOCleft template.

Let's consider my User page on Ouroportal the control for this little experiment.  If you look at the source for the page, you'll see that it is composed almost entirely of transclusion calls to Paragon Wiki.  (You may have to hit reload or add a "?action=purge" to the end of the URL if you get a template render error.)

Taking that same source, I added a TOCleft at the top of the page for Test 1.  Notice that when this renders, the Table of Contents does not appear at the top of the page as I would expect.  Instead, I end up with two ToCs in subsections - one for Works in Progress and one for Personal Info.

In Test 2 I replaced the individual transclusion calls with one single transclusion to pull in my entire User page from Paragon Wiki.  The results are pretty ugly.  The Oportal logo and the left column have been sucked inside the content and there seems to be something strange going on with one of the linked images down in the table of Pinnacle characters.

I'm sort of mystified by this since the User page on Ouroportal looked just fine up until a couple of days ago.  (The source was previously identical to the Test 2 example.)

Anyone have any ideas?

EDIT:  Just a little less than an hour later Test 2 appears to display normally.  Now I'm really scratching my head.  Test 1 still doesn't look right, though.
« Last Edit: June 22, 2009, 04:08:58 PM by eabrace »
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Steiner

  • Elite Boss
  • *****
  • Posts: 1,602
    • Steinerd.com
Re: Interwiki Transclusion and TOCleft
« Reply #1 on: June 22, 2009, 04:24:43 PM »
It has something to do with the CSS. I was taking a look at it and it just smells of CSS foul play. Possibly give the O-Portal the same theme as Pwiki with a different color scheme and logo?

I can help with that if need be.
~Steinerd

Sekoia

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 1,828
Re: Interwiki Transclusion and TOCleft
« Reply #2 on: June 22, 2009, 04:37:36 PM »
I'm not too surprised about Test 1. The interwiki transclusion process pulls the page content from PW and inserts into your local page. If the __TOC__ tag were directly in the referenced pages, it would likely end up carrying over as multiple __TOC__ tags in your OP article, which would result in multiple full copies of the TOC at those spots.

However, since the __TOC__ is wrapped in a template, it might get pre-expanded before OP scoops it up--which would result in the behavior you're seeing.

The simplest solution would be to avoid using the __TOC__ tag in anything you want to transclude. Let the wiki auto-TOC those pages instead. :)

When I first viewed Test 2 it looked fine. Then I looked again and it looked ugly like you described. I'm not sure what exactly is happening there without poring over the rendered HTML, but my guess is that it's dropping a closing tag somewhere for some unknown reason. I've seen unrelated cases where something rendered without a closing tag and caused similar odd results. I don't know why it would do that here though. :/

Interwiki transclusion isn't really a strongly supported feature. Its configuration isn't well documented, nor are its mechanics. So it's also possible that some of this is Mediawiki bugs.

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #3 on: June 22, 2009, 04:48:47 PM »
I figured interwiki transclusion would fall in the bucket of "Really?  People do that?"  That's why I'm trying to stress it and find its limitations.

I think we found one.  Now if someone else stumbles into this we can at least say, "Oh yeah.  We've seen that."  :)
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Sekoia

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 1,828
Re: Interwiki Transclusion and TOCleft
« Reply #4 on: June 23, 2009, 01:08:44 AM »
Fair enough. :D

I might actually be wrong on what I said, slightly. I just remembered that we had been having issues with transclusions that involved images because, originally, images were in the HTML as (for example) /w/images//1/11/Knives_of_Artemis_logo.png. When you try linking to that exact path on OP, there's no such image. So I had to fix it to render the images as http://wiki.cohtitan.com/w/images//1/11/Knives_of_Artemis_logo.png in order to have them show up right under interwiki transclusions.

So, my suspicion is that the interwiki transclusion might just be ripping the HTML directly from the page and using it as-is.

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #5 on: June 23, 2009, 01:58:13 AM »
That makes more sense to me and would seem to fit what I've seen from playing around so far.  The behavior I've seen is very close to "Hey, you.  Go render this and pass me back the HTML so I can just copy your results."  That's sort of how I approached getting the Userboxes for PW and O-portal both working on the same page.

But that's why I was confused to see that it seemed to be dropping a </div> somewhere in the whole process and cause the sidebar to shift over into the content.
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #6 on: June 23, 2009, 04:33:10 AM »
Aha.  I may have a new hypothesis to examine in more detail tomorrow.

I notice in the gooned up Test 2, that it cuts off the content here:

Code: [Select]
<tr>
<td style="width:130px"><a href="http://cit.cohtitan.com/character/451" class="external text" title="http://cit.cohtitan.com/character/451" rel="nofollow">The Tainter</a>
</td><td style="width:20px">25
</td><td><a href="http://paragonwiki.com/wiki/Mutant" title="Mutant"><img alt="Mutation" src="http://paragonwiki.com/

The end of that last line

Code: [Select]
/w/images//thumb/3/37/Originicon_mutation.png/15px-Originicon_mutation.png" width="15" height="15" border="0" /></a>
and about 88 more lines are being dropped - including the </td> </tr> tags needed to close out the table that the sidebar is getting sucked into.

Perhaps there's a character limit or similar size limitation to interwiki transclusion.
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

TonyV

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,175
    • Paragon Wiki
Re: Interwiki Transclusion and TOCleft
« Reply #7 on: June 23, 2009, 05:20:35 AM »
Wow, great catch.  Can you tell how many characters/lines are being included?  Also, if you run the test again, is it consistently cutting off at the same place?  (I'm wondering if maybe it's trying to do it asynchronously.)

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #8 on: June 23, 2009, 11:29:22 AM »
From what I've been able to see, it does seem to cut off in the same place every time.

Figuring out how many characters it's getting before the cutoff is the "more detail" I was planning to check out today.  :)
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #9 on: June 23, 2009, 05:31:24 PM »
OK, here's the information I've dredged so far.

In the transcluded result on O-portal, the resulting HTML appears to contain 65511 characters.

However, it's worth noting that the transcluded HTML is not exactly a match for the HTML as it's generated for Paragon Wiki.  A side by side comparison between the two very quickly points out that every link to an article or image that appears as "/wiki/Filename.png" on PW appears as "http://paragonwiki.com/wiki/Filename.png" when transcluded on O-portal.  Whether or not this contributes to a potential character limit is unclear at the moment.  However, without the extra "http://paragonwiki.com" at the beginning of each hyperlink, the character count would be 61639.

No obvious 2^n correlation there, so I'm not entirely convinced just yet that I've found the limit.  Maybe if I try something that's just a giant, single paragraph string of nothing but pure text to simplify the experiment...
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Steiner

  • Elite Boss
  • *****
  • Posts: 1,602
    • Steinerd.com
Re: Interwiki Transclusion and TOCleft
« Reply #10 on: June 23, 2009, 05:51:50 PM »
I'll take a look at the code, because I know MediaWiki's classes use buffer and sometime cURL... and when they do you have to tell it how big the information is... meaning the default 1024 bytes might not be big enough... 

EDIT:
Nopers... though it does look like that tool is wrapping it in another table... I ever tell ya guys how much I hate wiki, lol... too much going on at once, lol. Not to mention a syntax that makes worse sense than html markup, lol.
« Last Edit: June 23, 2009, 06:00:34 PM by Steiner »
~Steinerd

Sekoia

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 1,828
Re: Interwiki Transclusion and TOCleft
« Reply #11 on: June 23, 2009, 06:31:00 PM »
I think I figured out where it's pulling the page from. It's probably using the action=render method. If you look at this: http://paragonwiki.com/w/index.php/User:Eabrace?action=render You'll see that it expands the urls like you described, Eabrace. So the domain names would contribute to the character limit.

Interestingly, 65511 is just 25 bytes short of 2^16 (65536). Might the HTTP response headers be exactly 25 bytes? (EDIT: I looked at some of the response headers and this seems unlikely...)
« Last Edit: June 23, 2009, 06:39:15 PM by Sekoia »

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #12 on: June 23, 2009, 08:26:55 PM »
Interestingly, 65511 is just 25 bytes short of 2^16 (65536). Might the HTTP response headers be exactly 25 bytes? (EDIT: I looked at some of the response headers and this seems unlikely...)

Taking the source from the "render", 65536 cuts off freakishly close to where the transclusion is cutting out, even if that's not it.

Strangely enough, I just refreshed the Test 2 page and it's displaying properly now.  I suspect you're in there tweaking something on the backend, Sekoia.  ;)

EDIT:  Oh, nope.  Now it's gone all schizo on me again.  :P

Here's where 65536 characters would end based on a copy all from the "render":

Code: [Select]
The Tainter</a>
</td><t

Here's where it cuts off the transclusion on the O-portal:

Code: [Select]
The Tainter</a>
</td><td style="width:20px">25
</td><td><a href="http:

This time I get a count of 65552 characters instead of 65511 and appear to be 16 characters over the 2^16 boundary.  65536 characters might fall within a margin of error depending on where my selection boundary was placed when I grabbed the rendered source.

I wonder if this little block might be hinting at something useful now that I look closely at it:

Code: [Select]
<!--
NewPP limit report
Preprocessor node count: 2/1000000
Post-expand include size: 37/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
-->

I'm also wondering if putting the final O-portal HTML and the render HTML side by side in BeyondCompare might tell me anything.

Quote
I ever tell ya guys how much I hate wiki, lol... too much going on at once, lol. Not to mention a syntax that makes worse sense than html markup, lol.
Yeah, I've started to notice that over the last couple of months.   :D
« Last Edit: June 23, 2009, 08:46:47 PM by eabrace »
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

Sekoia

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 1,828
Re: Interwiki Transclusion and TOCleft
« Reply #13 on: June 23, 2009, 08:56:06 PM »
Strangely enough, I just refreshed the Test 2 page and it's displaying properly now.  I suspect you're in there tweaking something on the backend, Sekoia.  ;)

EDIT:  Oh, nope.  Now it's gone all schizo on me again.  :P

I had a similar experience this morning. I had cleared the interwiki cache, and when I visited Test 2 and purged it, it displayed normally. Then when I hit refresh, it was back to displaying wrong. I chalked it up to browser caching weirdness, but... maybe not?

In any case, I haven't poked at the backend since this morning around 9:30 AM EST.

I wonder if this little block might be hinting at something useful now that I look closely at it:

Code: [Select]
<!--
NewPP limit report
Preprocessor node count: 2/1000000
Post-expand include size: 37/2097152 bytes
Template argument size: 0/2097152 bytes
Expensive parser function count: 0/100
-->

You can read more about that report here. All of the numbers are well below their limits, so I don't think there's anything in it that's going to help solve the problem unfortunately. :/

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #14 on: June 23, 2009, 09:18:32 PM »
I was just thinking that if - for example - Post-expand include size number was an exact match for the difference between the number of characters at the cutoff and 65536 that it might tell us something.

But it's not.  :P
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,230
Re: Interwiki Transclusion and TOCleft
« Reply #15 on: June 24, 2009, 04:33:16 AM »
OK, for my next trick, I pulled a lion outta my hat.

Musta been the wrong hat.

(I don't know why a Bullwinkle reference suddenly popped into my head, but there we have it.)

Then I went back to testing this.

Just to eliminate as many variables as I could think of, I went for an extremely simple example of transclusion.  I just created a series of integers from 1 to 19200, each separated by a single space, and created a page here.  I then jumped over to the O-portal and transcluded that page here.

Character counts:

Text entered:  104093
Text rendered:  104101 - <p> and </p> tags and /n (or whatever the proper notation is in this case - an actual newline, not a visible tag) added by rendering
Text transcluded:  65550 - <p>< br /> and /n added at the beginning, <p> from first render retained, all text from the number 12774 on (including the first render's </p> tag) dropped

If we consider all code (including the <p>,</p>, and /n) from the first rendering as having been transcluded on the second render and subtract the characters added by the second rendering (<p>< br />/n</p>/n), then a total of 65535 characters were transcluded into the O-portal.

(If you're counting along at home, I have to use put an extra space after the < in the br tags because the forums will translate it into an actual line break if I remove that extra space.)

I think it's pretty safe at this point to say that we have an effective (if not exact) transclusion limit of 2^16 characters.   :)
Titan Twitter broadcasting at 5.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov