New card installed!

There have been some upgrades across the janky cluster that have caused a few issues like 500 errors and site downtime.  Because of the volume of support this has created, we thought it would be a wise idea to roll these issues up into a blog post in order to address everything that has been going on.  It’s not a perfect solution by any means — especially since this should have been addressed sooner — but I’m hoping that most of the folks on janky can benefit from the solutions I will outline below.

For the customers who have been seeing temporary site outages between the hours of 10pm and 10am PST, you have been witnessing the upgrade happening itself.  We’re taking all of the servers on your clusters and giving them the latest and greatest version of Debian.  This means that services might error out for a few minutes while new key packages are put into place of old ones.  This is also being done in controlled batches in order to minimize outages and incoming support.

Yes, this strategy isn’t optimal.  As a company that hosts customers around the world, it is hard to pick a time that won’t inconvenience somebody.  As a large percentage of our customer base is in the United States, the times listed above were picked.  This also allowed for a minimal amount of disruption.  Still, people are seeing problems — and we’re sorry if you’re one of them.

If the outages weren’t enough, there have been 500 errors that have been cropping up.  These can be caused by a couple of things:

1. Software (PHP, Python, etc.) that was compiled against the old 32-Bit libraries on our server.  Since the new version of Debian (Etch) is 64-Bit, this has caused locally compiled binaries to fail.  A quick way to fix this is to recompile any custom software against the new libraries on our server.  Unfortunately, since custom software falls under the “unsupported” category, the best we can do is point you at the following DreamHost Status blog post from a few night ago and wish you luck:

http://dreamhoststatus.com/2008/09/02/debian-upgrades-and-custom-php/

Good luck!

2. Recent changes to Apache 2 caused anybody using AuthGroupFile in their .htaccess to get errors.  It turns out this was an oversight.  The module was recently added to Apache 2 and our servers have been reconfigured — so this should be fixed now.  If you continue to see errors tho, you might want to attempt commenting out any AuthGroupFile lines in your site’s .htaccess files (adding a “#” to the beginning of the line should do the trick) and reloading your page.  If your site comes up after commenting that line out, please drop me a line so I can fix your Apache service correctly.

Should you find that your problem was not addressed here and you still need assistance, please let us know right away by contacting Technical Support.  One of our techs will be glad to assist you with any issues that you might be seeing.  We want to make sure that everything is how it should be, after all…

Update 5:33 PDT:

Actually since its much quicker and I am pretty sure the issue is just ram I went ahead and just replaced the ram in the system and it is now back up and running. Sorry for any inconvenience this caused!

bubs buck cheshire felix fin flipper morris nobnin riffraff tigger

Update 09/04 2:35 AM PDT:
All services should be restored. Everything went off without a hitch and took less than 40 minutes. Each sever was down for less than 5 minutes. Please contact support if you are still having any connecting to MySQL and your database is on any of the above servers.

php.cgi: error while loading shared libraries: whatever.so.15: cannot open shared object file: No such file or directory

You are going to need to recompile your php against the new 64bit toolchain, since it was more than likely compiled against the old 32bit libraries. This is not something that we can offer support with, but you can find more information about how to compile PHP in our wiki:

http://wiki.dreamhost.com/Installing_PHP5#Compiling_a_Customized_PHP_5

Update @01:05

Fozzie is three quarters of the way through the upgrade process. Thog is at 10%. Both should be back up by 02:30. After which, we’ll start on gladys. For those on the frisky cluster, once thog is done, your upgrades will be complete.

Update @01:33

Fozzie is back up. Thog is chugging along and is 70% through the update process.

Update @02:08

Thog is back up and running! Last on the list for tonight is gladys, which will begin around 02:30.

Update @04:55

Gladys is serving files again! Good night!

Affected machines: barracuda carrera chevelle corvette dart delorean esprit fairlane frisky-vserver10 geep hornet jetta knocker model-t montecarlo olds passat rambler riviera scion skylark tempest touareg

Update: Okay! Things are getting back to normal. Sorry about that. Please let your happy DH tech support person know if you’re still having problems!

Check back here for updates!

Update @23:06

Zoot is progressing nicely. It should be back up and running in about an hour and a half. We’re starting on bumper now.

Update @00:09

Zoot is back up and serving files. Progress on bumper is moving along as expected. We’ll start kermit within the next 30 minutes.

Update @00:58

Bumper is back up and running. We didn’t start kermit as planned. That will happen now.

Update @04:17

Whew! After some issues with it’s boot archive, kermit is finally upgraded/patched/running! That’ll do it for tonight. Fozzie, thog, and gladys tonight will round out the upgrades for the next couple of weeks.