Sanspoof database server down (and back up)
If you’ve noticed your databases being down, it’s likely because one of our database machines, sanspoof, simultaneously lost 2 drives shortly before 5am this morning. We have it on new hardware and are restoring all of your databases from backups as we speak. We expect everything to be back up sometime around 9am PST.
Update: Everything was back up by approximately 10am PST on Sunday the 10th. Only about 2-3 hours of data was lost on average.
Happy DreamHost Why-Must-It-Be-2-drives-At-5am Team!
.
December 10th, 2006 at 8:37 am
Interesting.
They should have a secondary database server for each operational one, so it goes right to the backup. Losing our database is pretty annoying, especially if your whole website depends on it.
December 10th, 2006 at 8:39 am
Is this outage still going on?
December 10th, 2006 at 9:07 am
I am pretty upset that mine has been down for so long also, especially since I was having database problems all week and then they moved it somewhere new, and now a total outage. My websites don’t run at all with out it.
December 10th, 2006 at 9:12 am
My site is still down. I’m totally dependent upon my database as well. What really ticks me off is that support has sent 2 emails to me telling me the issue was resolved when it clearly was not.
December 10th, 2006 at 10:01 am
Hopefully it will be up soon! My databases isn’t on Sanspoof, but good luck others!
December 10th, 2006 at 10:03 am
Isn’t it 10am over there now? How is it going? Site is still down. BTW, fellow Wordpress users - if you don’t want the default Wordpress error message, which exposes your database host, you can find and adjust the text in the wp-db.php file within the includes folder.
December 10th, 2006 at 10:08 am
Still down here …. I haven’t gotten an email about it being up or down yet.
December 10th, 2006 at 10:13 am
Still waiting I sent ticket 3 hours ago had no reply from them at all.
December 10th, 2006 at 10:26 am
I keep getting emails from them telling me it’s been resolved when it clearly has NOT been. I really am getting sick and tired of this crap.
December 10th, 2006 at 10:38 am
i am down too…that would be nice to get some credit for downtime
December 10th, 2006 at 10:44 am
I’m losing money to, I highly doubt they can compensate the amount I’m losing. But they should.
December 10th, 2006 at 11:43 am
Wow, all the hints at how “much” money you guys are loosing makes me wonder why you don’t upgrade and get off of a pennies a day hosting. Go with one of the big boys, the $XXX per month dedicated 99.9999999% uptime gauranteee hosting companies.
Grow up and stop whining. Do you guys think that the tech support people at Dreamhost sit on their butts and only fix stuff when the babies cry and whine? Are you all so stupid to think that they want stuff like this to happen?
Some of my sites are down too, but they’ll be back up as soon as they can get them up. And no amount of sissy baby remarks will speed that up. *burp*
December 10th, 2006 at 11:45 am
Zak, what do you mean? Is your site back online? I am curious and worried about the age of the backup. Guess from now on I will need by own daily backup, so that I can migrate the database to an alternative server within minutes if this ever happens again.
December 10th, 2006 at 11:59 am
Just for the record: my site is back online. And a note to Rick. It’s funny, that some people still feel they have to defend Dreamhost. But I also agree, what are people who seriously earn money with their sites doing here at all? Buy or rent your own server at a professional hoster. The only reason I am here at Dreamhost is the cheap bandwidth, which I am giving away for free (creative commons downloads etc.) as a hobby. I don’t mind paying more for better quality but my other much more reliable and slightly more expensive provider can’t offer me the bandwidth I need for an acceptable price. I am still dreaming of my own server in a nice 19″ rack, but that’s probably how this ‘Dreamhost’ once started and it is turning more and more into a nightmare host.
December 10th, 2006 at 12:20 pm
Rafael I know how it feels
December 10th, 2006 at 1:46 pm
My sites down this sucks
December 10th, 2006 at 1:52 pm
I just lost a million dollars in revenue. NOT. Sucks to be you guys.
December 10th, 2006 at 2:23 pm
Most of my stuff is back up now. Of my webapps back up it doesn’t seem like I have lost any data. Good job fellas.
December 10th, 2006 at 2:35 pm
Scott, why do you run a webhosting business off a webhosting business?
December 10th, 2006 at 3:41 pm
Because $8 a month is the cheapest overhead he could manage while making someone else actually do the hard work.
December 10th, 2006 at 6:13 pm
Another suck-ass Dreamhost moment. They say they backup, but they don’t. An entire day’s worth of phpBB posts, PMs, registrations, all gone. The 9th has ceased to exist.
December 10th, 2006 at 7:00 pm
My sites still down.
December 10th, 2006 at 7:50 pm
once a day backups are pretty standard - for off machine backups anyway - which are needed when two disks die at once in most cases.
December 10th, 2006 at 8:05 pm
Don’t worry, you’ll still be taxed for the 9th.
December 10th, 2006 at 8:32 pm
ToS, Limited Liability, section 2:
“Under no circumstances, including negligence, shall DreamHost Webhosting, its offices, agents or any one else involved in creating, producing or distributing DreamHost Webhosting’s Server service be liable for any direct, indirect, incidental, special or consequential damages that result from the use of or inability to use the DreamHost Webhosting Server service; or that results from mistakes, omissions, interruptions, deletion of files, errors, defects, delays in operation, or transmission or any failure of performance, whether or not limited to acts of God, communication failure, theft, destruction or unauthorized access to DreamHost Webhosting’s records, programs or services. Customer hereby acknowledges that this paragraph shall apply to all content on DreamHost Webhosting’s Server service.”
The terms of service effectively state that backups are *your* responsibility. Under the agreement you signed, they are under *no* obligation to do them for you, so I’d be thankful that they do backups for youi at all.
December 10th, 2006 at 10:49 pm
My sites still down.
December 11th, 2006 at 1:43 am
Dear all of you who have complained about your site being down or your massive loss of revenue caused by this outage,
Shut up.
If you think you could do better, shove off and do it. If your revenue is dependant on something you can’t control, more fool you. If you expect what you expect for what you pay, more fool you.
Stop whining.
I have nothing further to add.
Thirsty Ferret
December 11th, 2006 at 5:25 am
I couldnt agree more with Ricky and Thirsty Ferrett, why do people whine so much when they are paying tuppence for their hosting? If you are that unhappy with it, try other providers, although I could possibly tell you about each and everyone, as I have worked with nearly 99% of hosting companies.
The guys at Dreamhost are doing a great job, especially with the number of servers they have to look after, and as always, Technology is never 100% guaranteed!
If you are 100% dependant on your hosting for your “business” blow the cobwebs off the credit card and pay for hosting that you are guaranteed 100%.
December 11th, 2006 at 9:19 am
Don’t use pages that depend on the “sql” then? Or use a site that got a backup page that u can use. And don’t whine about lost credit and so on I think the one that takes this hardest is DH
December 11th, 2006 at 9:28 am
Why are you defending dreamhost for there downtime and flaming everyone else who complains about 2 days straight of downtime. You 3 must be the stupidist sacks of shit I have ever seen.
“The guys at Dreamhost are doing a great job” What? They could have done a number of things. Move our stuff to another server and our data as well and we would be back up in less then an hour. Our sites have been down for 2 days (48 hours for you 3 dumbasses)
It’s pretty clear, the staff of dreamhost is sitting around. Show you appriciation by wacking them off for there great accomplishment of being the worst host ever. Oum oum oum yumm…
December 11th, 2006 at 9:50 am
Well, you know where the door is…. (although I doubt you will go to a different provider, somehow the cost thing probably wont agree with you).
“They could have done a number of things. Move our stuff to another server and our data as well and we would be back up in less then an hour.”
You´re obviously one of those people that things Hosting Technology is that easy to operate and that it can be done at the flick of a switch - So easy that you get a hosting company to take care of it for you.
December 11th, 2006 at 11:19 am
I was just thinking about not using the mysql for webpages… I have read few articles about it and I found no good arguments to run website on mysql…
sqlite is a lot faster and eats less resources (so the sqlite site says :D).
In this way, you are worried only about your own server and you dont have to care about database server at all…
Unfortunately most CMS and similar things are completely focused on mysql…
Just an idea…
I read this problem, because my jabber dont work and I was searching something about this
December 11th, 2006 at 1:02 pm
Oh drats… I just discovered that my site is down! Any new news on when it should be up and running again? It says 9am… but seeing as it’s noon… I thought it would be nice to know.
December 11th, 2006 at 1:05 pm
It is about time for a status update. Our database is still down.
In the meantime we all could help collectively by sending some Ducktape to Dreamhost!
December 11th, 2006 at 1:28 pm
It’s one thing to have an outage — it’s another to totally ignore your own “official” status site and have a claim that “this will be fixed in a few hours” up for two days. Embarrassing.
December 11th, 2006 at 2:07 pm
When two disks in a RAID5 set goes down and you have to recover from backup, it really isn’t as easy as pulling in extra hardware. If you have a spare RAID5 set laying around it doesn’t take that long, but if not it takes 4-8 hours just to build the new RAID (depending on controller type, drive type and size), and THEN you have to recover the database from backup. Depending on backup solution, which is probably tapes and probably incremental, you have to build from an initial backup followed with the daily incrementals, which due to the search time for tapes takes quite some time as well. Hours would not be unusual, depending on how many tapes the backup spanned (that is, how long ago it is a full backup was made, and how much data we’re talking about).
Losing two drives of a RAID5 set at the same time is really a once every 50 years occurrence (again depending on how many drives we’re talking of), but when you’re running very many servers, it will happen from time to time.
From what was reported here, the database went down at 5:00 and was back up around 11-12:00, which is really in the lower end of what I’d estimate as the minimum recovery time. I’m guessing the rest of you are suffering from unrelated problems.
December 11th, 2006 at 2:57 pm
Is the “Customer Login” still down??
December 11th, 2006 at 4:20 pm
I can’t get to webftp or panel.dreamhost. I know this is unrelated, but since there isn’t a status post for those items, I was wonder if anyone else is having this problem?
December 11th, 2006 at 4:21 pm
Pfft, figures. I finally comment about it, and they come back. Nevermind!
December 11th, 2006 at 8:00 pm
Hmmm. Monthly hosting for the price of a big mac fries and a coke. I wouldn’t expect that much.
Two days of straight downtime? I don’t have that problem (and my sites are mysql backed). I don’t deny that it’s bad luck for you. (and to have two drives fail at the same time is seriously bad luck too).
I understand the therapeutic value of bitchin here.
But I also understand that it makes totally diddly-squat difference to when your site comes back up.
Make sure you’ve reported it and watch your mail for responses from support so they can ask you any questions necessary to get you back online. But you already know that.
Dana - it was a trick to make you look funny!
December 12th, 2006 at 1:02 am
I was considering switching my hosting over to DreamHost from Site5 due to the amount of bandwidth they provide at the cost. They’re really cheap. Until reading all of this…
Is DreamHost totally unreliable? Does this sort of thing happen often? A little history here would be great.
December 12th, 2006 at 6:19 am
Dustin,
I’ve been with Dreamhost for about 7 months now, I host a half dozen domains with as many subdomains, all running websites, databases, mail and SVN repositories. I’ve had my one website dugg and had close to 70,000 unique visitors hit my site in little less than 4 hours and my site running WordPress stayed up just fine. It was not an insane amount of traffic, true, but moderately so. My machine is gobstopper, and in all the time that I’ve had it I have had maybe 15 minutes of scheduled downtime. There is also http://blog.dreamhost.com or http://discussion.dreamhost.com if you want some another places to read up about dreamhost.
Just remember, there are over a thousand machines ( increasing daily ) and over 300, 000+ customers. What you’ll see here are the very vocal, very anonymous minority who are angry and frustrated that their service was interrupted.
Dreamhost has an excellent referral program and a 97 day refund period. I highly suggest you give them a try. If you decide to try, I suggest signing up for a 1 year, pre-paid L1 plan using a coupon code that gives you the full $97 discount ( like REM97* ) . That will end up costing you about $22.40 US for the first year of hosting. I think that’s pretty decent.
Signed, A happy customer
* in the interests of full disclosure, that is a code i created, if you use it, I get nothing because I’m passing the full discount on to you. But if you refer someone using your own code and they stay past the 97 days, I get $5 applied to my hosting bill.
December 12th, 2006 at 7:05 am
They’re not *THAT* anonymous.
December 12th, 2006 at 7:57 am
Am I the only one with their site STILL down?
December 12th, 2006 at 8:24 am
Our site is still down (Tuesday 9:25am).
December 12th, 2006 at 8:26 am
dreamhost is a company which is loosing quality while growing:( and there is NO TIME about solution. they dont dont when they going to solve this problem:(
i get an hosting accout from dreamhost and dreamhost id getting big problem for every 2 months:mad:
December 12th, 2006 at 8:32 am
Im down now
December 12th, 2006 at 8:39 am
Still down :’(
December 12th, 2006 at 8:51 am
@George: There is something wrong with your DNS - look here: http://dnsreport.com/tools/dnsreport.ch?domain=buttonboyinc.com
December 12th, 2006 at 8:55 am
mysql.airnews.pl (208.113.130.219) down
December 12th, 2006 at 9:49 am
When I first signed up a few months ago I had lots of issues but since then I’ve had none. Dreamhost is a great hosting provider. Everyone has their problems. I like the people that run this thing and I’ll take a little downtime if that’s what it takes to stick with the guys who aren’t worried about a corporate identity.
December 12th, 2006 at 10:20 am
Henrik: Thank you. Could you help me? I really don’t know what that means other that something is wrong.
December 12th, 2006 at 10:39 am
My site is down.
December 12th, 2006 at 12:23 pm
@Dustin:
Don’t have too many worries. I’ve been w/ DH for 8 months now and haven’t had any real downtime or server issues. I think there was one once for about 15 minutes one day. You shouldn’t take the comments on these boards into consideration too much, as the only ones you see are the ones pissed off. You don’t see the thousands that are happy and extrememly satisfied with their service. Everyone always posts complaints, never compliments.
I have been very happy with DH, and I hope you’ll continue to consider their great service in your hosting hunt.
–Kurt Kohle
December 12th, 2006 at 12:26 pm
sorry, but DH has downtime every other day, its unacceptable stop overloading ur servers and provide a proper server.
December 12th, 2006 at 12:47 pm
Half of my sites are down - again : (
December 12th, 2006 at 1:12 pm
@George: If you want, contact me through the contact window on my webpage.
What you should do is to check with your domain registrar which nameservers (DNS) you have linked to your domain name. That’s a start.
December 12th, 2006 at 1:54 pm
brisk.dreamhost.com ,, not work alsol , or it have something wrong , please try to solve it quickly ,
December 12th, 2006 at 2:07 pm
@KING MASTER
You can’t reach the servers directly. My sites are on planters and work just fine, but planters.dreamhost.com will not work, as theres nothing to show you there
December 12th, 2006 at 4:44 pm
my website is down, yet i can access my mail etc. First time this has ever happened. I was able to ftp items to server but my person cannot view them because my site is down. Any help is very welcome.
December 12th, 2006 at 5:18 pm
So is it common for them to have a SERIOUS problem like this affecting this many people and completely ignore all of their customers? My (income producing) sites have been down for WAY too long now, and we get nothing from DH. This is ridiculous. The least they could do is give us a quick update.
December 12th, 2006 at 5:22 pm
snewp , i didn’t mean that . i only want to say my site on this server ..
and also i think this machine is down
December 12th, 2006 at 7:09 pm
egon -
Do you know how many people are actually affected? I don’t and I would bet that you don’t either. *SOME* of these problems for some of these users (like George and King Master) are likely either misunderstanding or user mistakes. Sorting these messages and reports out for 3-4 hundred users is not a 5 minute task. I agree that it shouldn’t be too hard to get a read on what’s up and what’s not but in the THOUSANDS of machines supported by DH it’s still a pretty big haystack.
I can assure you that the DH staff that I have dealt with have been very concerned about uptime and very concerned about getting back to people.
Many people here (including myself) are sympathetic that you are down. I *DO* agree that it seems unreasonable to not have an update or something when it does appear that there is an issue affecting more than just a handful of people. dreamhoststatus.com really needs to be as up to date as reasonable. It would be way nice to have a site that indicates the actual apparent status of machines or even sites. Maybe we need to put together a shared monitoring site.
Carlo -
The mail servers are NOT on your hosting machine. I’m a bit surprised that FTP works since that really is on your machine, but it could be that the apache process has failed. I’m assuming you reported it to DH support through the panel.
December 12th, 2006 at 10:02 pm
I understand that there are a lot of people they support, but it’s pretty apparent that there isn’t only one or two having an issue. Any global issues should be at least somewhat quick to resolve. My day job is monitoring the data center for yellowbook.com so I understand high traffic and tons of users. I also understand failures, and there should NEVER be an issue this severe that lasts as long as it did. Servers fail all the time, it’s expected. It should also be expected that there are fail-safes in place, and updates be available to those who need it.
That being said, my sites are now back up, and the support rep that helped me was very helpful, although it took a long time to respond to the issue initially, as it always does with DH. Although understandable with so many customer’s sites being down, critical issues should not be ignored, and a universal update would have been very nice.
December 13th, 2006 at 12:53 am
is nash.dreamhost.com server works fine ?
myy site on it still slow, the database server is Sanspoof.
December 13th, 2006 at 6:16 am
My Mysql is offline now
December 13th, 2006 at 6:18 am
MySQL on vanowen.dreamhost.com still down
December 13th, 2006 at 6:22 am
egon -
Thanks for following up. I agree that finding that monitoring should be able to provide notification to DH staff that there are issues with a machine enough to address it within an hour. I do agree that even at the low prices DH uses, it’s not unreasonable to expect less than 24 hours in a row of downtime. But I also don’t want administration to think that the solution to all problems is a reboot. These aren’t Windows(tm) machines.
DH is certainly working on the balance between cost and service (and if you were to look at their jobs page you’d see that they are NOT paying a whole lot to these poor guys).
December 15th, 2006 at 2:33 am
So DreamHost is totally unreliable… so what? You get what you’ve paid for, and that’s it.
I wonder if all of you screaming guys ever had any serious job, and would you do it for free? Because that’s what you’re asking DreamHost to do, to give you reliability you haven’t paid for.
(btw, it’s kind of foolish to ask for less than 72h downtimes at this price; DreamHost is actually overachieving)
April 28th, 2007 at 7:30 am
I met a lot of webmaster which hosted websites at dreamhost and aren’t satisfied. They moved their sites because dreamhost offer coupon codes for really cheap hosting solution but users aren’t satisfied.
July 19th, 2007 at 3:03 am
Good job fellas.