Clifford blew a Drive!
The MySQL server ‘clifford’ has blown a hard-drive and I had to replace it and the array is rebuilding as we speak. This should take approximately 30 minutes but the server will be down/inaccessible in the meantime. This should only effect people with ‘clifford’ as their MySQL server. You can find this out from Goodies -> Manage MySQL as your server will be listed in the format of ’servername:service’ and if your ’servername’ is ‘clifford’ than this is effecting you.
From the Happy DreamHost NightCrew!
Update: 8:18 PDT.
Clifford is having problems with building the array so one of our senior admins has been contacted and is working on resolving the issue.
Update: 11:25 PDT.
We are still working on fixing clifford and apologize for the continued outage. There’s currently no eta on when it will be up, however our admin team is hard at work to get it going as soon as possible.
Update: 12:11 PDT.
We’re now in the process of moving clifford to new hardware and restoring from backups. Databases hosted on clifford should start coming online soon, though the entire process may take a few hours.
.
May 13th, 2008 at 7:20 am
It’s ‘A’ffecting you. The clifford server going down would be the ‘E’ffect of the hard drive blowing out.
Yes, I am a snob.
May 13th, 2008 at 7:21 am
My site has been very slow for 2 days, Can you please let me know when is going normal?
I’m on peach, mysql pichu
May 13th, 2008 at 7:31 am
The grammar in this status update was atrocious.
May 13th, 2008 at 7:49 am
Thanks dreamy guys….
May 13th, 2008 at 8:07 am
An update on the ETA would be nice. All of my sites are down.
May 13th, 2008 at 8:15 am
My sites are also down. Please at least be courteous enough to update the ET to fix.
May 13th, 2008 at 8:20 am
Downtime total 2 hour(s) 14 min(s) 5 sec(s).
May 13th, 2008 at 8:34 am
Why my sites are DOWN!!!
!!! Please !!! FOUR DAY very bad!!!
May 13th, 2008 at 9:08 am
All of my sites are down as well. Hoping for a quick recovery…
May 13th, 2008 at 9:14 am
DF’s hope is lost!
Quick recovery does not equal 3 hours.
May 13th, 2008 at 10:15 am
I’m showing over 5 hours downtime on my db assisted sites :-/
May 13th, 2008 at 10:35 am
This is a very critical time for my site as it is new and traffic is just now starting to pick up! I’m going to be pretty bummed if this causing the traffic I’ve been building to slow down again. Argh! ETA anyone?
May 13th, 2008 at 10:36 am
Labeling the “Severity” of this problem as “Medium” is highly understating the problem. It might be a “medium” problem for Dreamhost as it is not affecting all customers — but my website is DOWN. I have clients and others calling us all morning for updates on the status. The “Severity” should be classified as “URGENT” or “EXTREME.”
May 13th, 2008 at 10:59 am
Please im need some information for my visitors and others Radio Producers, please fixed a little more quickly that mysql server problem.
From UN Happy DreamHost Customer
May 13th, 2008 at 11:11 am
DreamHost really should adopt the moniker “Hosting on the You-Get-What-You-Pay-For Principle!”
May 13th, 2008 at 11:19 am
It’s been down for nearly 6 hours, I think a severity of High is waranted at this time… We need an update as to what is going on, 6 hours is far too long with out an update.
This should have been resolved within 2 hours according to the problem description’s severity.
May 13th, 2008 at 11:24 am
But they have one of their SENIOR ADMINS on it!
May 13th, 2008 at 11:26 am
Please update the ETA at least guys..
May 13th, 2008 at 12:38 pm
YAY! The error loads right away now instead of waiting for 2 minutes of attempting to load!
Wait a second, that’s still a problem.
hehe
Thanks for working on it guys. I put up a temporary page letting people know what’s going on.
May 13th, 2008 at 2:34 pm
My site is finally back up. Just running slowly. I’m assuming it will pick up speed after this is all done? I hope so.
May 13th, 2008 at 2:49 pm
Well, that really takes a long time to restore from backup - hope you don’t use a tape drive for backups……that would be ancient technology….mysite gives an error message saying “Error establishing a database connection” - I guess it isn’t completed
May 14th, 2008 at 11:05 am
one of my sites has been down for over a day now. this is getting ridiculous. is there any sort of idea as to when the clifford server will be fully functional??
May 14th, 2008 at 3:23 pm
You think you guys have got problems. My sites on york have been down for 3 days now, apparently due to a problem with the filer wayne.
There is no mention on here of problems with york or wayne, so I can only assume that for some reason it isn’t affecting too many other people ? I was told on Monday that it might take 24 hours to sort out and then no response from anyone since then.
Time to move my sites to somewhere vaguely reliable I think.
May 15th, 2008 at 12:02 am
http://xkcd.com/349/
May 15th, 2008 at 12:04 am
Oh.. and @Andy Norman
“If you are experiencing a problem not reported here, check our web panel for more information.
(Please remember, posting in the comments here IS NOT an official way to contact DreamHost.)”
May 15th, 2008 at 4:29 am
I wasn’t attempting to contact DreamHost, I have already raised a ticket via the web panel. I was just venting because I wasn’t getting any sensible reply via the proper channels and there was no mention of any issues with wayne or york here. My issue is fixed now, but of course no one at DH bothered to tell me that and I had to go an tweak the domain settings to fix the normal “bad_httpd_conf” error that you get after your home directory has been unavailable for a while.
Now busy moving most of my sites elsewhere
May 16th, 2008 at 3:13 pm
http://digg.com/tech_news/Dreamhost_Attempts_to_SCAM_Their_Customers
May 19th, 2008 at 1:23 pm
So, why does a drive failure (in this day and age) result in server outage? I think the DH guys need to hire some real server architects/senior admins to resolve their problems.
I wish I was running the show over there.