Author Topic: Titan Server "Sidegrade"  (Read 5441 times)

TonyV

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,156
    • Paragon Wiki
Titan Server "Sidegrade"
« on: June 25, 2013, 03:31:38 AM »
So as you can tell, this maintenance is taking a lot longer than I expected.  Nothing's broken, it's just that what I thought was going to be about an hour and a half process turned into a bit of a slog.  So here's the deal.  Some of it technical, some of it not, but all of it honest.  :)

Our hosting provider is Linode.  (This isn't a Linode rant, they're awesome!)  Before around an hour ago, we were running on their Linode 4096 plan, which runs $80 per month.  Since City of Hero's shutdown, though, our traffic is a fraction of what it was, so we really don't need that much horsepower to run the site.  (But we'll still have it, I'll get to that in a minute...)  So for a while now, I've been wanting to downgrade the server to a Linode 2048 plan instead, which is $40 per month.  I knew that would be kind of an ordeal, so I put it off until tonight.

So tonight, I buckled down and started working on it.  The first order of business is that I would have to shrink some of the filesystems.  No problem, we had a lot of space not being used, so that should easy peasy.  But I got to thinking about it, and I didn't like the layout of our filesystems.  For you Unix geeks out there, I had a root filesystem (/), /home, /var/www (all of our web site data), /var/database (all of our mysql databases), and /var/vcs (our Mercurial code repositories).

At first, I thought, what the heck, let's just put the whole kit and kaboodle on the root filesystem.  That was the first thing that extended the outage time.  It's not like there's any huge advantage to us for having them split up, and it's all redundant virtual storage anyway so it's not like we'll ever just have one filesystem fail, so I started copying /var/www over to our root filesystem.  Around 45 minutes later, it was copied.  Great!  So I started copying the others, starting with /var/database.

Around that time, I got to chatting with Codewalker, who mentioned that it might be worthwhile to have the root filesystem by itself and /var separate, so that if the /var filesystem fills up, the server won't crash to the point where we can't log in remotely.  He's right, so all of a sudden, I was almost back to square one.  We decided to consolidate our web directories, database directories, user home directories, and version control system directories all on /var, and have that be a monster filesystem, with root and everything else that's not /var on its own, much smaller, filesystem.

So with that, I deleted what I had copied over to /var on the root filesystem, resized it back down, and increased the old /var/www filesystem, moved www down a level under /var, and started copying the other stuff.  After some configuration changes and filesystem resizing, another two hours had gone by.  I fired up the server, and I started getting a bunch of error messages.  Crud!  What happened?  I had stupidly forgotten to copy the files from /var on the root filesystem to the new /var filesystem I had created.  Okay, no sweat, back into single-user mode, copy over, boot back up, everything works.

Now, it's about five hours in, and I'm only now getting to the whole reason I wanted to do this stuff to begin with: to downgrade the server.  So I pop into Linode and choose the new sizing option.  It tells me that I need to use my free upgrade before I can downgrade the server.  Um... free upgrade?

Turns out, at some point in the not-too-distant past, Linode changed their plans around, doubling the RAM and increasing the disk capacity on all of their servers.  But it didn't just happen automatically, since it required a server reboot and some down time to migrate everything.  So at least for a while, we've been on a Linode 4GB RAM plan, but have only had 2GB of physical memory allocated to us.  That's right, folks, we've only been using half of our allocated resources, grrr!

So I had to upgrading the server (for free!) to a Linode 4GB server, a bit ironically, so that I can complete the downgrade of the server (to half price!) to a Linode 2GB server.  Done, then a quick kickoff of the downgrade (which actually took a lot less time than I thought, probably since I had already resized the filesystems and Linode's request queue was much smaller by then), and now we're finished, booyah!

I'm really sorry that it has taken this long to get everything straightened out, but I'm also really glad I'll be saving $40 per month and we'll be operating with the exact same resources we had before.

Sekoia

  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 1,804
Re: Titan Server "Sidegrade"
« Reply #1 on: June 25, 2013, 04:09:06 AM »
Before around an hour ago, we were running on their Linode 4096 plan, which runs $80 per month. [...] I'm really sorry that it has taken this long to get everything straightened out, but I'm also really glad I'll be saving $40 per month and we'll be operating with the exact same resources we had before.

I just wanted to highlight those numbers briefly, since we don't often mention it. Tony has been paying $80/month out of pocket to keep Titan running up until now. That's just shy of $1000/year. With this change, it's dropping to about half that, but it's still a lot of money.

Titan does still welcome your donations if you want to help keep the site running. Even at our peak, we rarely received enough donations to cover the cost of keeping the site up. Now that City has shut down, we rarely receive donations at all.

I don't want to scare anyone into thinking that Titan might shut down for lack of funding. Tony's pretty darn committing to keeping the site up and running. And I know Tony doesn't want anyone dropping large chunks of cash on him; he'd rather that kind of thing go to other projects.

All the same, just a reminder: Titan does accept donations. At the bottom right corner of our home page, there's a PayPal button you can click. Alternately, you can donate by clicking through from this page on the Paragon Wiki. (Both go to the same place.) If you want to toss a few dollars in so that Tony isn't footing the whole bill himself, I'm sure he'd appreciate it. :)

eabrace

  • Titan Moderator
  • Elite Boss
  • *****
  • Posts: 4,091
Re: Titan Server "Sidegrade"
« Reply #2 on: June 25, 2013, 04:32:38 AM »
I'll likely be making a "Christmas bonus" donation toward the end of the year when I see how my first year as a contractor plays out.  :)
Titan Twitter broadcasting at 4.000 mWh and growing.
Titan Facebook

Paragon Wiki admin
I was once being interviewed by Barbara Walters...In between two of the segments she asked me..."But what would you do if the doctor gave you only six months to live?" I said, "Type faster." - Isaac Asimov