need help mirroring Paragon Chat files

[cohRock]

This message is directed at anyone with a strong background in web server configuration.

I've been working in conjunction with Leandro to offer a mirror for the i24 beta files and whatever other file(s) may be necessary for Paragon Chat.  The (un-validated) files have been available at my domain since 29 June.  It is a virtual domain on a Linux Apache server, hosted by midphase.com.  (My package offers "unlimited" storage and "unlimited" bandwidth, so I'm not worried about being hit with extra fees due to too many downloads.)

On 7 July, Leandro reported a snag, though:

I ran into a problem with the pc.r0k.us mirror; the server seems to have HEAD requests (to request the file information without the contents) disabled. That makes it impossible to validate if the files are there without actually downloading them, which would be a massive pain to do future verification.

He went on to show me the error messages he was getting.  They are all of the form:

Code: [Select]

HEAD request failed for http://pc.r0k.us/i24beta/cg.dll
I added a .htaccess file to the subdomain.  On first try it contained a single line:

Code: [Select]

Options +Indexes
When that did not help, we tried replacing that with:

Code: [Select]

<Limit HEAD>
order allow,deny
allow from all
</Limit>
That did not work either, and I don't think Leandro has had any more time to try to get my mirror going.  Since it has been several days now with no advancement, it seemed useful to see if there are any experts out there who could tell me, "Dummy, you just need to do this."

BTW, a web browser is able to show all the filenames just fine, but apparently any HTTP HEAD info beyond that is being blocked.

I decided to add a CPanel summary of server details.  Most of it is Greek to me, but an expert might find some of it useful:

The complete kernel version (displayed as hover text) is 2.6.18-498.el5.lve0.8.80

[BitLoadR]

I've been working with Apache servers and coding PHP sites for some time (6+ years) but have never run into anything like this.
I will try and find a cause / solution on Stack Overflow, maybe somebody there known about this.
If anything comes up I'll let you know.

[Leandro]

I am planning to work around the problem by sending a GET request with Range: 0-0 instead of a HEAD request for your server. The result should be pretty much identical to a HEAD request, with an extra byte.

For the record, this is what a HEAD request to pc.r0k.us returns:

Code: [Select]

Connection: close
Date: Tue, 14 Jul 2015 14:15:02 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Client-Date: Tue, 14 Jul 2015 14:15:05 GMT
Client-Peer: 174.127.119.148:80
Client-Response-Num: 1

Edit: well, this is interesting. Sending a GET request with Range: bytes=0-99 results in a 403 Forbidden. At this point I am wondering if it's blocking the User Agent for not being a standard browser? Sending a GET request with no range results in the same error.

Code: [Select]

Connection: close
Date: Tue, 14 Jul 2015 14:23:15 GMT
Server: Apache
Content-Length: 342
Content-Type: text/html; charset=iso-8859-1
Client-Date: Tue, 14 Jul 2015 14:23:17 GMT
Client-Peer: 174.127.119.148:80
Client-Response-Num: 1
Title: 403 Forbidden
Edit 2: yep, it was the User Agent. I set the header to clone my browser's ("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2453.0 Safari/537.36") instead of using the default for LWP::UserAgent, and now both GET and HEAD work fine.

Code: [Select]

Connection: close
Date: Tue, 14 Jul 2015 14:33:53 GMT
Accept-Ranges: bytes
Server: Apache
Content-Length: 5667432
Content-Type: application/x-msdownload
Last-Modified: Fri, 04 Feb 2011 08:03:42 GMT
Client-Date: Tue, 14 Jul 2015 14:33:56 GMT
Client-Peer: 174.127.119.148:80
Client-Response-Num: 1

The host is picky about what software can access it, apparently. The problem is that I don't know what header Tequila and Island Rum send when they're requesting content, so now my validation tool works, but the patcher itself might not. I'll have to test.

[cohRock]

Bitloader and Leandro, thanks.  We're free to modify the .htaccess file however it might be useful for the pc.r0k.us subdomain.  Beyond that, the tech support at midphase.com has been very knowledgeable and supportive in the past.  If the first layer of support can't help, they escalate the issue, probably to their actual server engineers.  If there is some setting in the physical server that can alleviate problems for my virtual domain, it is quite likely they would be willing to change the behavior of my domain to be more "normal".  I would just have to know what to tell them.

[Leandro]

I tested the Windows version of Tequila and Island Rum and both seem to work. Manga will test the Mac version tonight, and if that works, it's a non-issue.

The Forbidden response to unknown user agents might be their attempt at flood/DDOS prevention, but considering how easy it is to spoof user agents, it's really useless.

[cohRock]

It looks like my site has started acting as a mirror.   

You can see that since this morning monthly bandwidth transfer has increased from 1.31 Gbytes to 11.62 Gbytes.  My domain normally does not transfer 10 GBytes in an entire month.

It concerns me somewhat that CPU usage has increased from 0MB/100MB to 8MB/100MB.  Actually I don't remember ever seeing it at anything other than 0MB.  I don't know what CPU usage actually measures, nor how often it is reset.  My concern is that it will act as a back-door brake on the ∞ bandwidth I thought my domain could offer.  If the 100MB is a daily limit, then there should not be a problem.  If the usage accumulates for a month, though, then in several days my mirror will probably stop serving files until August.

I'll keep an eye on it, hoping that the value resets to 0, or at least stabilizes.

[edit add]
Checking cPanel again, the CPU usage has dropped to 0, and the bandwidth is still at 11.62 GBytes.  So the usage seems to be some sort of dynamic measurement, not an accumulative one.  That should mean my site can reliably serve files throughout the month, unless there are so many simultaneous downloads going on that the CPU usage reaches its limit.
[/edit]

[cohRock]

Out of curiosity, I checked my server logs.  From 14 July until the end of the month, almost 230 Gbytes were downloaded.  Assuming everyone needed the full set of files (about 4 Gbytes), that is almost 57 installations from my server alone.  I don't know how many servers there are.  If there are 4 and the load were equal, then 230 new folks were trying out Paragon Chat in the last half of July.

With 23 Gybtes in August so far, that adds another 23 of us (and counting).  Cool!   I'm sure there are more folks too, who transferred the files from an old computer or backup files or got them directly from a friend.

[Leandro]

We have between 6 and 8 mirrors for the files, actually, so more like 50 full installations. Some of the mirrors died under load, though, I will run a cleanup script on the 10th to remove dead ones.