RESOLVED! Tib hardware failure
The mysql server “tib” suffered a hardware failure around 1 hour 30 minutes ago, at 5am Pacific time. It rebooted itself, and spent the next hour checking its backup drive, only to hang. It will not remotely reboot so I am heading in to diagnose the problem! Estimated further downtime is one hour.
Update 7:33AM Pacific: Unfortunately, it turned out to be checking the raid volume, and fsck was eating the files. I am moving the backup drive to another server right now and will begin the restore process.
Update 945AM Pacific: The backups took a little coaxing, but they are now restoring as fast as the drive will read! I’m not certain exactly how long it will take, but your databases will come online as they are restored. Ballpark time to completion is 2 hours.
Update 1251PM Pacific: Looks like the backups finished restoring at 12:17pm. Please write technical support if you are having any other problems with your “tib” databases!
.
August 18th, 2007 at 6:34 am
hmmm…. my site’s been ‘not loading’ for awhile now =\
August 18th, 2007 at 6:42 am
Why the hell Tib is my MySQL server… It has suffered several problems these days…
August 18th, 2007 at 7:10 am
Sigh, so this is the third time http://ww2db.com has gone down in the last four days.
C’mon Dreamhost, you can do better than this. Hell, I think my home Windows server has been having longer up time more consistently…
August 18th, 2007 at 7:23 am
My site and all my email accounts have been down continuously since Thursday.
August 18th, 2007 at 7:23 am
This is not too good. Tib hosts most of my dev database. Tough luck. Send 2 trouble tickets in as many days.
August 18th, 2007 at 7:29 am
This is great.
The site that I host with Dreamhost that I use the most has its DB hosted off of Tib.
August 18th, 2007 at 7:39 am
I suppose if I must dig deep to find one good thing that I can find because of the Tib problem, I can say that I finally found the motivation to build the proper error handling for when that database goes down…
Anyone know how I can change timeout settings in PHP? I want it to fail faster.
August 18th, 2007 at 7:54 am
“Severity: Low”
That’s rather subjective, isn’t it? To those of us whose sites are down, I don’t think we consider the severity to be “low”.
August 18th, 2007 at 8:19 am
this sucks… im considering moving hosts.
August 18th, 2007 at 8:21 am
My favorite part is “Estimated further downtime is one hour”… 3 hours later we are still f***ed up!
August 18th, 2007 at 8:23 am
Re: Pete
The severity is pretty subjective, honestly. Our primary goal is to get the information out there, and sometimes assigning an appropriate severity is hard to do. Our general rule of thumb is one server = low, handful of servers = medium, and the-world-is-ending-hope-you-brought-emergency-rations-and-coffee = high.
I’ve got the backup drive in the new server, the mysql instances are being configured now, then the restores can commence. Should be about 5-10 more minutes, then they will begin.
August 18th, 2007 at 8:27 am
3 days my sites have been down. Dreamhost is a f****** joke. Worst web hosting company ever. I should have moved all my sites to another host a long time ago.
August 18th, 2007 at 8:53 am
All of the announcements make it sound like the system occasionally comes back online but then shuts down again. Yet I’ve had uninterrupted downtime since Wednesday p.m. — it hasn’t come back once, not even briefly, so none of these announcements seem to apply to my situation at all.
August 18th, 2007 at 9:10 am
Thanks for the update, Kelly. How long will the restore take? Just wonder b/c it’s been 30 min since your last update.
August 18th, 2007 at 9:42 am
To those whose sites are down because of the single mysql server being offline -
When I was actually trying to use dreamhost for a site that needed to have some sort of reliability, one thing that was key was backing up my DB to a second mysql server on a regular basis. When the primary mysql server croaked (which they will here), I failed over to the backup server.
Write a little cron job that fires off every hour that backs up your db to the backup. It’ll make your life a bit better when DH has a problem. If your site is interactive, there will be a bit of overlap that you’ll have to merge when the primary comes back up and you’ll want to disable your cron job after a fail over occurs, but it’s better than not being up at all and you can do all this automatically if you’re a programmer.
- Roach
August 18th, 2007 at 9:55 am
still down…. mmmmm
August 18th, 2007 at 10:30 am
For those of you who have repeated issues with Tib like I do, you can backup your databases with phpMyAdmin, and then import that data into a database on a different server.
If you have trouble with doing this, try having the dreamhost support staff help you with it. Just do a dump of all your SQL stuff, then using ftp, put the dump file onto your server somewhere that you can get shell access, and use mysql to access your new sql server like so:
mysql –username=blah –password=blah –server=blah
August 18th, 2007 at 10:31 am
hmm. it cut my reply. I think because I used the < sign without using the html code.
Anyways, it should look like this:
mysql –username=blah –password=blah –server=blah < backup-file-name.sql
There are a lot more details to this procedure, but this should get you on the right path.
August 18th, 2007 at 10:32 am
my site been acting strange since 2 days now plz try to fix soon thanks
August 18th, 2007 at 10:36 am
Both friend of mine’s sites have been down since yesterday morning. Congrats DH!
August 18th, 2007 at 10:44 am
sorry but if this issue is not fix by tonight im moving im loosing to many members cuz of this
August 18th, 2007 at 11:03 am
I don’t believe that Dreamhost is the best host ever, but I do believe that any intelligent site admin should learn the tools necessary to handle supporting their own site. Everything you need to make your site work is available to you if you want it to be.
But if you have the money to spend on another host, then do the American thing and just go spend more money so you don’t have to learn anything.
August 18th, 2007 at 11:28 am
The site being down isn’t half the problem. The biggest problem is having no email.
August 18th, 2007 at 11:33 am
Haha, my site just came back online.
And guess what? It’s all data from 30 hours ago. A lot has happened on my site in 30 hours too. Man, that really sucks. *sigh* I got a lot of work to do to get back to where I was last night.
August 18th, 2007 at 11:42 am
well that sucks hope i dont have to do the same thing. im debating wether i want to extend my hosting with them. they stay with some issue.
August 18th, 2007 at 11:44 am
My website http://ww2db.com seems to be working now… hope it’ll stay working for a while this time…
August 18th, 2007 at 11:49 am
Ugh, I lost a day or two worth of data!!!
Thanks, DH…
August 18th, 2007 at 11:49 am
Well, I’ve hosted with them for a few years now, and this is the first time I’ve been screwed like this. I don’t blame them for the hardware failure, but a 30 hour old backup… that means they’re probably restoring from a tape drive or some other swappable backup device.
The problem I have with this is that means either their mirror failed, or they didn’t have a proper raid mirror setup in the first place. They said FSCK was the culprit. The crappy thing about that is that if fsck was doing anything to the files, then those changes were getting mirrored too, and wiping out the immediate backup options. The only option they would then have is to go for a 30 hour old backup.
I’m shooting in the dark here though. Just remember that these guys aren’t trying to screw their customers. It’s just not good business practice
August 18th, 2007 at 12:48 pm
I’m having issues with the temp drive being full? Wierd.
August 18th, 2007 at 3:00 pm
Tutto rotto !
August 18th, 2007 at 3:15 pm
“streaky” mysql server is still down. It’s been down since 9am PST at least.
August 18th, 2007 at 3:45 pm
I’m glad you posted, Jim. I thought I was the only one having trouble with “streaky.” I have two database-driven sites that haven’t been working.all day.
August 18th, 2007 at 6:29 pm
One of my sites is dying with not being able to be accessed these last few days on and off. Is there a way to monitor where in the world a website cant be viewed?
August 18th, 2007 at 6:30 pm
One of my sites has been down like 95% traffic and sales but I can see it most of the time the last 4 days.. what in the world.. lol
August 18th, 2007 at 6:46 pm
OMG OMG I AM LUSING S00 MUCHH PESOS BECUZ OF TEH D0WnT1M3!
TIME 2 SWICH HOSTSES TO NO UPTIME HOSTING
(I found that link rather amusing in the context of this week’s Dreamhost downtime and ensuing litany of hilarious angry comments)
August 19th, 2007 at 1:46 am
panel.dreamhost.com not loading.
my sites are not loading.
please fix.
August 19th, 2007 at 1:51 am
panel.dreamhost.com not loading.
my sites are not loading.
+1
August 19th, 2007 at 2:01 am
now working again.
August 19th, 2007 at 2:17 am
panel and sites are down for me as well.
August 19th, 2007 at 2:17 am
My account panel is still not work now! And my site does the same thing.Fix the problem quickly
August 19th, 2007 at 5:53 am
Nothing is resolved as far as I am concerned, all five of my sites are down as of 8:52 am.
August 19th, 2007 at 6:11 am
I have four accounts I manage, and as always happens when there’s a problem, three are working fine and the busiest one is down. It was alternately slow and down for ten minutes at a time yesterday but seemed to have been fixed by last night. Now it’s down completely, and has been since I first checked it an hour and a half ago. At least Sunday morning is a slow time for us, but I hope it’s fixed soon.
August 19th, 2007 at 3:58 pm
Our site is down since one hour ago…
August 20th, 2007 at 8:36 pm
Two of my sites just went down.
August 20th, 2007 at 8:44 pm
Guys,
Common… they had a hardware failure, if that happens to your own computer… if you are not a technician like me and don’t have any knowledge at all you ill have to wait at least about 4 days to get your computer repaired at 100% (off course depends on the issue). If you don’t like dream host services, you should just cancel your own account and go find another one elsewhere, and you ill see that happens too in any other host… hardware someday ill fail and is needed to be replaced, like your car, like your own life… needs a doctor sometimes. Stop being a child a stop whining and complaining, its not the end of the world.
Regards
August 20th, 2007 at 9:20 pm
Site was down for about an hour, just came back up.
August 20th, 2007 at 10:39 pm
now even the freakin’ comments cannot be posted here!?????? AAAAARRRRRRRRRGHHHHHHHHHHHHH
August 20th, 2007 at 10:39 pm
Hi there. ALL the sites I am hosting with you are either ridiculously SLOOOOOOOOOOOOOOOW or cannot be accessed. Even panel.dreamhost.com and status.dreamhost.com do not work - so I cannot contact customer support. Great! SMTP and POP also just went down. Please sort it out, guys and gals!!! Twice in a week?!?!
(And do please make a customer service email available again! ……….)
August 21st, 2007 at 5:46 am
Hey ya all, have been monitoring these alerts and the comments.
Highly suggest you change hosting to IPower.com they give you 200gb of storage for the same charge (7.95) as this hosting and are never down like this. at least they do not screw up every bodies website. Also have a much better CO (vdeck) that you can actually understand and navigate.
The cp of this hosting looks like a 5 year old built it. Very unorganized and not user friendly. also the vdeck at Ipower will let you edit your files on line. no ftp if you don’t wish.
Been watching hosting sites since 2001 and this hosting company has grown and become one of the most economic and user friendly hosting on the web.
No I am not staff or connected with this company… Just a consumer that researches thoroughly before buying.
August 21st, 2007 at 6:13 am
I’ve moved most of my stuff to a new host now - had enough of the crap. Database driven soooo much faster too. And it’s cheaper! Site5.com.