RESOLVED DNS Outage Identified
Posted (August 17th, 2007 at 12:52 pm PST) by DallasWe have identified a runaway process that has been erroneously eating domain DNS records and that has been killed. The missing DNS data is being restored as quickly as possible now so any domains still down should be restored in the next couple of hours. We have already pushed out new software with safeguards against this situation happening again. The network intermittence yesterday caused database connection problems which triggered a race condition in our code.
We will be checking all domains for missing DNS records and restoring all of them so there is no need to contact us to let us know your domain is down. We will update this site when we believe all domains should be working again.
UPDATE @ 2:39PM PDT
Okay, we believe all domains DNS should be regenerated and working again now. Please contact support if you have any problems still!
80 Responses to “RESOLVED DNS Outage Identified”
Its too late Dreamhost hella people switching now.
So that’s why I couldn’t reach my db server.
What happened to mail sent to one of the malfunctioning domains during the outage?
Maybe it is just me but this sounds like they are making things up.
That’s why I have DNS records on nameservers elsewhere….
@zorg And just why would it be in their interest to do that?
I can’t help but laugh my arse off @ that excuse! We have identified a runaway process that has been erroneously eating domain DNS records and that has been killed. BWAHAHAHHAHAHAHA. Thats a good one! Next time I need off work, I know who to ask for an excuse!
@Les, what external DNS service are you using? I’ve been looking into that, but don’t really know where to look.
Michael - I can tell you for certain that some mail was backed up and came through later, and other mail was bounced.
It would be nice to be able to slave your records to another server
According to SiteUptime my site has had about 99.2% uptime which, all things considered, isn’t too bad for something costing me less than five quid a year. It’s just that today I was expecting a few hits and I’ve had maybe two hours uptime. Not good, but I can live with it. Hope it’s all solved soon.
Yes - seems like DNS is one of DH’s weaknesses at the moment.
Interesting that the DNS server only is not serving one of my zones, out of about 15.
Great to have you back…
I for one will miss Nigerian Scam Baiter and his $1490 dollars. (It’s okay though…he’s just in a tough spot right now and has to go into hiding…if I let him transfer his money into my bank account for a few months, I get to keep most of it.).
Thanks for making things work again Dreamhost… Let’s get a little fault-tolerance in the systems now…
@Michael Froomkin and Yodu, no mail should have been lost during this problem. Mail servers will hold mail and attempt to deliver it generally for 24 hours or more. I sent my self an E-mail from yahoo while my domain was down and it came in with others once DNS was up for my site.
“We have identified a runaway process that has been erroneously eating domain DNS records and that has been killed.” In other wards someone was actively hacking your DNS, errant processes do not delete DNS entries unless it was maliciously planted!
The dog ate my DNS records.
Try ZoneEdit.com for DNS hosting. Your first five domains are free. There are other providers out there, but I’ve used them for years, and like them. We only have about five domains that had DNS with Dreamhost (out of about 60 or so that we host). And those five sites used Dreamhost’s servers for primary site operations — whereas the remainder of our clients are hosted on a leased dedicated server with another provider. I have loved Dreamhost. And the first few years with them we had absolutely no issues whatsoever… But the last year or two has become very worrisome. Today was the last straw for us, and while we’ll continue to use Dreamhost for “extras” like Mailman and one-click installs — we will no longer have any of our client sites run their their primary operations (web, email, etc.) with Dreamhost. We already switched DNS this morning and will be moving content and other data over the weekend. It’s a sad thing, really. I hope Dreamhost can rebound. I’m rooting for you guys!
Maybe Dreamhost should investigate providing some DNS service OUTSIDE their server farms so that there is no longer a “single point of failure” that can take everything else down.
Reminds me of an engineer who designed a fully redundant, completely backed up, high priority communications system for one of our clients when I worked for XX_XXXXX. He thought he had it all covered until the janitor broke one wire to the redundant controller in the middle of the night. The military base got rather excited.
Seriously, though, Dreamhost should take the time to examine their layout for possible single points of failure (such as their DNS) and do what they can to reduce the number and severity of those points.
In fact they should do that as soon as they get my service restored… :o)
Tom
Dreamhost is a dying breed. They’ve had more negative press of any hosting company I can recall in years. This is a time when press is hurting them. Slideshowpro made the switch off of DH. When choosing a host… don’t choose dreamhost.
The thing about DNS is that, if it is setup correctly, there should NEVER be a single point of failure. Are all three Dreamhost nameservers really just one box? Or are they using the same load balancer? Are they all in the same location? One should reasonably assume that with multiple nameservers, means you have redundancy… so if one, or in the extremely unlikely event, even two, of your nameservers go down, you can still provide DNS service. So either I don’t fully understand Dreamhost’s perhaps unique DNS setup or they don’t fully understand the concept of DNS redundancy.
We switched hosts. Done with dreamhost. This kind of stuff happens WAY too often. We were scared to go through the switch because we didn’t want down time but this is it. Done.
I’m back - now to rectify that lost $30,000 during the downtime. I mean geez, my webcomic is a multi-gabillion dollar business… the loss of productivity during this time has been STAGGERING.
Thanks Dreamhost, you help me beat more flash games.
XOXO
Jamie
Why do people insist on coming on here and announcing how they hate dreamhost and are leaving. Are they trying to get others to agree with them and leave too? It’s like dreamhost’s majority of customers are 5 year old andgry bitter children! If you’re going to leave, then leave and be done with it. You aren’t speeding up the recovery process, and you aren’t convining anyone else to leave. Please stop spamming the forums/comments with your malcontent.
A Loyal and Happy DreamHost customer.
I have a feeling that DH probably has an automated process to populate the DNS from the databases that make up the control panel. And it was probably that which went awry. But I might be wrong.
@JH Thanks for the link to ZoneEdit.com you rock!
Pingback:
DreamHost Status running again…
@Matt - mail was indeed bouncing for a good portion of this DNS outage. That’s how I found out about it (a client called me to say that a mail had bounced back as undeliverable), otherwise I just was a strangely quite morning on email.
@Kevin: Because we are frustrated with Dreamhost’s spotty service and lack of communication about what’s actually going on. Because using ANY Dreamhost service on dreamhoststatus.com is just dumb. Because once this outage is over, Josh or someone is going to post a long, funny, apologetic message on the blog and everyone’s going to say “Thanks, you guys are so great for being up front with your customers.” Because, as someone said earlier in this thread, Dreamhost sometimes behaves like the only sites hosted on its servers are personal gaming blogs. We have to say something; nobody’s trying to make you leave, but if I do, and if I announce that, you can’t blame me.
“…This kind of stuff happens WAY too often.”
really? when was the last time?
@microcars:
How about yesterday?
I’m mostly a happy customer, but it does seem like the vast majority of outages I’ve experienced over the last couple of years could be rectified by better backup and/or redundancy. I know it’s an inexpensive service so I don’t expect 100% multiple levels of redundancy, but if the DNS is going down I’d at least like it to point to a Dreamhost page that says something like, “Dreamhost, the host for the website you are trying to reach, is currently experiencing technical difficulties. Please try visiting the page you were looking for in a couple of hours. We’re sorry for any inconvenience.” That way at least people don’t think pages are gone and remove bookmarks, email me to complain that the pattern they wanted is gone, etc.
Is that kind of thing possible? I’ve asked before but never seen any kind of official answer.
Thanks.
How many other cheapo hosts have the guts to host a transparent status blog like this?
If you think the grass is greener, just google “1&1 sucks”..
kimberly: DNS is such a low level protocol that virtually every function of the internet depends on it… when DNS is down (or malfunctioning), you can’t even *get* the error page…
My website is slow again. This is really getting old.
already bought a media temple account . beginning a slow and annoying move of my sites. I understand problems occur, but honestly, that’s why you prepare for it happening. I second the feeling that dreamhost seems to think it’s personal blog sites etc.
I’m not saying, look at me I’m leaving it’s important. I’m saying “you’re n00bs, this is ridiculous”. one of my sites is just basically plain html, 2 php includes for header/footer and at 200k hits it crashes and dies. that’s pathetic. I hope you can get your stuff together this last year has made dreamhost look really amateurish.
@inhislikeness.com
[now to rectify that lost $30,000 during the downtime. I mean geez, my webcomic is a multi-gabillion dollar business… the loss of productivity during this time has been STAGGERING.]
Then why the hell are you on $8/month shared hosting?! Pry open your wallet a bit and get a dedicated server.
Hopefully you were just being sarcastic, but if not, my comment stands.
Im verry happy to have our website back online. In the future I hope these things dont happen.
I really want to be happy with DH, but I can’t. I love the features, But what about the downtime? I also host with two other hosts, one smaller one hasn’t even had 1 hour of downtime _the entire year_ and the other one, which also hosts of 350.000 website on over 2000 server only had 2 hours downtime this year (due two a DDOS attack on the DNS servers). Why can those hosts manage their their networks (and not be much more expensive) and DH can’t? Just add the power failure (one day), today and some other downtimes this year and I’m at about 3-4 DAYS of downtime already. This is very disappointing. Every time DH runs into a problem like this they do stuff “against this situation happening again” but then a few days later something else blows. They just might be sitting there on our money and as soon as something happens try to fix it, but only then. There doesn’t seem to be much effort put into prevention. Really, this sucks, And then all these smart-ass blog posts when another host goes down. Are you guys in first grade or in the hosting business?
Hey guys - it’s all baaack!
Excellent work guys - I can imagine the stress you’ve been under these last few hours. Hopefully when the dust has settled you’ll be putting system in place to ensure that was the only time all DNS is lost! 
I still can’t access my status reports.
@Tom Yes, he was being sarcastic. Making fun of most of the people who post about how much money they’re losing on their $8/month hosting plans…
Your comment works great for all those idiots though
It’s great that you all are “nice and honest on the telephone” (c.f. “Allces Restaurant” - A. Guthrie) by telling us just what went wrong. But the reason honesty results in forbearance is from the hope that things will *improve* in future as a result. Well, “we are living in the future” (c.f. “Living in the Future” - J. Prine) and things have gotten worse, not better.
This last outage is my last straw. Tell you what. If you post your usual honest description of what went wrong, *and* the titles of those responsible and when they were fired, then I’ll stick around. (Fortuitously, I got my account renewal message, just *before* all this started.) Otherwise, it’s buh-bye - losers.
@adam - Have fun at MT, been there, done that, back here at DH and happy to not be at MT. Mostly b/c of mail issues (that are still not fixed). Feel free to contact me through my website if you want more info, but basically, I was VERY unhappy with their service.
For all of DH’s problems, they HAVE been quite good for the past 6 to 12 months and for the amount I’m paying ($16/month), I’m quite happy. No, I wouldn’t host a business with them, but anybody who hosts a business on ANY shared host is just asking for trouble. Even a semi-dedicated box isn’t quite good enough, but at least is closer…
And no, DreamHost PS isn’t a good alternative to a semi/full dedicated box. Anyway, I’ve been with DH for years and will probably continue to be with them for years because truly, the service is quite good for what I’m paying for.
== Quote ==
Howard Owen Says:
August 17th, 2007 at 3:12 pm
It’s great that you all are “nice and honest on the telephone” (c.f. “Allces Restaurant” - A. Guthrie) by telling us just what went wrong. But the reason honesty results in forbearance is from the hope that things will *improve* in future as a result. Well, “we are living in the future” (c.f. “Living in the Future” - J. Prine) and things have gotten worse, not better.
This last outage is my last straw. Tell you what. If you post your usual honest description of what went wrong, *and* the titles of those responsible and when they were fired, then I’ll stick around. (Fortuitously, I got my account renewal message, just *before* all this started.) Otherwise, it’s buh-bye - losers.
== ==
Don’t let the door hit you in the ass.
so would have having an external dns service avoided all of this? I heard this service is great, I may sign up for it to avoid this crap from now on:
http://www.dnsmadeeasy.com/
I applicate the hard work but for some reason i can’t get http://www.dreamhost.com up.
Anthony
@Tom - yeah.. just joking. I’ve found that the DreamHost Status Blog gives me as many laughs a day as a good dose of FARK with the amount of ‘I LOST MONEYS OH NOES!’ types.
Plus, everytime I post up here, a couple more people go check out my comic. 
So it’s all selfserving really.
Cheap promotion is cheap promotion.
Cheers,
Jamie
Hi
I was expecting some important emails today. Many people have told that emails to me bounced and some just, I won’t know. Have I lost these emails indefinitely or will they come back? Is there a record of the ones that bounced? Will the bounced ones come back or are they lost irretrievably?
I would really appreciate your help on that one.
Many thanks
Sameh
Unfortunately it’s not just my site that gets affected. I have recommened Dreamhost to over 30 of my clients over the years, because they actually used to be good. Now I’m having to go dig all my clients out of this mess, move their sites, re-setup all their email. No one cares about paying the $20 a month mediatemple service that works. Yes, it’s twice the money, but it’s still extremely cheap considering what you are getting.
And their system of tracking stats and doing just about anything in the control panel BLOWS dreamhost’s away
F*CK THE STOCK MARKET IS DOWN, AND I JUST LOST $6,000 FROM MY SITE BEING DOWN
THE DOW IS DOWN
THE DOW IS DOWN
PS MediaTemple has lots of downtime too.
You Dreamhost apologists are pathetic. You’re not helping the situation at all.
The complaining customers want to like Dreamhost and recommend them to others like they once did. The point is that they’ve been having a ton of problems lately. It’s understandable that this stuff happens and that given the price we pay it can be expected.
That said, when it happens, they make light of it, which is complete bullshit. It’s not a joking matter. Further, it seems like they don’t take steps to prevent it from happening again. In particular, this blog was supposed to be completely offsite and it wasn’t. If they can’t even secure this site, what are the chances any of our services will ever be?
@D - You make a good point about the redundancy of services. When my email wasn’t working and dreamhoststatus was down, I definitely felt what inspires a lot of the rhetoric on here. They had failed rather spectacularly on the “we’re putting this other site up so you can know when things are going wrong” front. But as someone that has worked in IT for a little too long, I can say that every time you plan around a point of failure/vulnerability, Murphy shows up to remind you that you didn’t think of every possible scenario.
I think the light tone in the post mortems masks what (I hope) are very serious discussions that go on during and after an outage like this down at the dreamhost HQ. I think the tongue-in-cheek spirit that defines their corporate culture is a big part of what makes them successful. In the end, the most we could hope for is a sort of executive summary on the changes implemented after the outage.
@D: Agreed. We need to be told why this happened. And has everyone here gotten all the mail sent to them during the outage? I know I haven’t, and I am assuming I’ll be told why.
I think a lot of us would pay more for Dreamhost’s service. It’s not about the money; we chose Dreamhost (at least I did) because they had the best service and the best support community. This is a serious black mark, and I pay Dhost WAY more than $8/month.
All of my servers (ftp-Control Panel-website) were down this morning- don’t know how long they were down as I didn’t get up until around 5am… but my servers got back up within an hour of that. Thank you, DH.
They oughta host this status site COMPLETELY off of ANY of their stuff - not on another portion of the same network, physical or logical.
As someone who works in the IT industry, DH is not the only company that experience outages. For someone who pays a hoster ~ $20/month to host my domains, I tell you that I am very satisfied.
Although some problems may inconvenience some people, we have to realise one thing DH does that a lot of other providers don’t, and that is tell the truth. At least when things are down, DH is honest and tells us what is going on. That is customer service!
Of course everyone gets upset when their blog or web site goes arse over elbow, and the anguish is compounded by the feeling of helplessness at not being able to do anything about it. But let’s put things in perspective. All systems, be they digital or mechanical, are liable to hitches, glitches and breakdowns - and paradoxically, the more reliable a system is the greater the shock and angst when it does fail.
I live and work in Indonesia, where Things Going Wrong is the expectation rather than the exception. Yesterday, not only was my Dreamhost domain down, but the local Telkom ‘Speedy’ [sic] broadband service went tits up at the same time - as it does every other day. So I just shrugged my shoulders and got on with other things, knowing that everything would be back to normal within a day at most.
I like Dreamhost. It’s a cheap and cheerful service, and whenever I’ve had any problem not related to a system-wide issue I’ve had a quick and friendly response from the support people. If I was running a business I wouldn’t put all my eggs in the Dreamhost basket; I’d use it for my fun stuff, but host my business with a company that specialises in business hosting. It’s a simple case of horses for courses.
I’ll end by saying a big ‘thank you’ to all the Dreamhost people who were obviously busting a gut to get things up and running again, and add that I do echo the widely-expressed sentiment that this event should lead to a re-evaluation of the DNS and backup systems, and beefing up the redundancy.
My email is still down. 6:05 pm pst
i like dreamhost; i think they really try to give me value for money. but my stuff was down since yesterday and didn’t work until this afternoon.
last time, no problem. nothing urgent going on. THIS time, i needed information pretty damn badly. i went through serious stress overnight and this morning it will be hard to forgive. it cost me in other ways too.
should i be mad at my pda for corrupting a key redundancy, or dh, who promises uptime and took 24 hours to make it right? 
What happened exactly? Did an instance of named start to hog memory or something like that?
After DH had some DNS issues a while ago, where everything was working but was just a wee bit hard to get to because of some name resolution issues, I changed my set-up a bit.
For a start, I don’t register my domains through DH. Having had problems with previous hosts not wanting to transfer domains, I use a separate registration service (daily.co.uk in my case, many others are available). From their control panel, I replicated all the DNS records from DH’s panel for each domain, then set the name servers to include both DH’s and Daily’s, which means I’m not completely dependent on either one to provide name resolution.
This won’t solve every problem, but I’d rather visitors get “server not available” messages, and that email gets queued rather than bouncing with “domain?? what domain?? never heard of it” type errors.
We use DNSMadeEasy.com for the DNS on some of our Dreamhost sites. These all stayed up. Two other sites that had their DNS at Dreamhost went down.
DNSMadeEasy claim a 100% uptime history and they are cheap. They also allow you to set your own TTL (default is 30 minutes). This is handy when you want to shorten it to 5 minutes in preparation for moving a site quickly.
The downside is that you have to manually enter the records, and keep them up-to-date with anything changed at Dreamhost. I suppose that these could get out-of-date and result in downtime, so this isn’t the perfect solution, but it works for us.
For added redundancy, you could maybe list BOTH Dreamhost and DNSMadeEasy nameservers? Does anyone know a reason why this might not be a good idea? ( I know my DNS basics, but I’m not an expert.)
BTW. DNSMadeEasy also provide cheap mail backup relays for anyone out there running their own mail server.
Thanks Les. You answered my question while I was still writing it! Your post appeared as I posted mine.
I think I’ll start setting both sets of nameservers for redundancy. We also use a 3rd party to register domains rather than do it at Dreamhost - it gives us much more flexibility.
Well, maybe it’s an engineer mindset versus an IT mindset, but I think with proper planning, this blog wouldn’t have gone down (use a completely outside service, like wordpress.com, Blogspot, anything).
Throwing one’s hands up in surrender doesn’t help anything other than enable Dreamhost to continue fucking up.
I like Dreamhost’s attitude in most cases, but not after they fucked up (fucking me over) and rather than say, refund customers even the tiny amount of money for the time we did pay for (hell, I’d settle for a more detailed explanation of what screwed up and their future mitigation plan), they make jokes about it.
They can start joking when the fuckups become a rare inconvenience instead of business as usual at Downhost.
my site is down still. sob sob
is this why my domain takes 30 seconds to resolved?!!
I am surprised at the sheer number of technical failures my account has had in its year of service through DreamHost.
Problems seem to _just keep coming_.
Has it been a particularly rough year for DH hardware or what?
I don’t mind outages or blips here and there, but the problems have been steady as Sundays and I am losing patience.
I too like DH’s attitude and the customer support is excellent.
DreamHost:
– Great value
– Great service
– Lousy hardware
Most of us paid less than $23 for a year of dreamhost. Why are we complaining about 99.5% uptime?
This is insane. I can’t believe how much DH has collapsed in the past 2 years. “Good customer service”–what?? This site, status.dreamhost.com, was down. We couldn’t contact support without the web panel, which was ALSO down.
In other words, our sites go down, and there is no way to tell why, and no way to contact them to notify them and ask why. Since when is this anything other than non-existent customer service??
It’s interesting to me, that everytime Dreamhost experiences a DNS attack, that the ‘bashers’ come out for the party.
So then. I wonder who’s sending the attacks?
ooo superr
ooo superrsadsd dsfdfdsfkldşkfşf dkf fşkfdsfh hlhd dfkhkfhdfhfşjhfsdh ldhfşjdhsfhdshfnmndöfhukyfjalkfıhgjkislfgfdlajlşfahjlkmfnjbhuhdkjduıjkmikdjfıaisjdkdsjfıluaemidkmfıojdfnüındmklfindfn
Im verry happy to have our website back online. In the future I hope these things dont happen.
my site is down still. sob sob.
very good
me to I am surprised at the sheer number of technical failures my account has had in its year of service through DreamHost.
Thanks for the update - we appreciate all the hard work you’re all having to do!