Planned Power Outage
We were just informed one hour ago from our building management that an emergency maintenance window is scheduled for this Saturday morning, February 24th, at 1AM PST. UPDATE! The building has just changed the window to be 23 hours later.. it is now scheduled for Sunday morning, February 25th, at 12:01AM PST. This maintenance window involves taking our entire building’s power offline for approximately 3 hours. The word from the building:
It was discovered by ABM Engineering during the power monitoring
equipment installation, that a Phase C Conductor Cable on UPS #4
has been compromised to the point that immediate action to
repair it is necessary or a ground fault will occur to the building
systems.
Since only one of our three data centers is losing power, not all servers will be shut down. Unfortunately, our core routers and our upstream providers are in that data center, so there will be no network to any servers during the window. Gaah.
Since we have advanced (albeit limited) warning of this event, we will be onhand to physically power off all of our equipment at 11:15PM PST. Barring any unforeseen issues (on our end or the building engineer’s) we plan on having everything back up by 4AM PST, hopefully much sooner.
This outage will affect everyone’s sites and email. Email sent to our customers during this outage will be deferred on the sending server and will be delivered after service is restored.
We apologize for the inconvenience this no doubt causes, as well as for the short notice. We will be posting updates as possible during the outage.
.
February 22nd, 2007 at 7:51 pm
Um, that sucks.
February 22nd, 2007 at 7:52 pm
Wow that’s like closing the internet, no problems for me, really it’s a good time for Dreamhost customers to have a good nights sleep.
February 22nd, 2007 at 7:53 pm
sucks, but I’m glad we found out in advance!
February 22nd, 2007 at 7:55 pm
Okay, I will admit at the outset that this is a more than slightly annoying.
After the events of last summer, the powers-that-be at your company STILL don’t have a redundant data center? Some place, I dunno, outside an earthquake zone?
Are you too busy boasting to us customers how easy your life is and how much money you make from us, that you still don’t have the temerity to address this glaring deficiency in your infrastructure?
I realize this may not be the most minor of expenses, but you might make more money if you also addresses the glaring deficiency between your top hosting plan and the other three…
February 22nd, 2007 at 7:57 pm
Didn’t you guys buy UPS gear for the core routers/switches?
February 22nd, 2007 at 8:04 pm
So all the servers will go down?
This is understandable and i know its beyond yalls (yes i’m from the south) control.
JK
February 22nd, 2007 at 8:12 pm
I would have hoped that the nameservers would at least be unaffected by this so that there might be time to move to an alternate server temporarily, but I’m assuming that’s not the case?
February 22nd, 2007 at 8:38 pm
Well it’s obviously not your fault and this things happen in life, could be worse : no more beer in the fridge ;-))
Thanks for letting us know, outstanding service as always !
TvD
February 22nd, 2007 at 8:44 pm
Can we maybe find out WHICH servers will be affected so that we can notify our clients?
February 22nd, 2007 at 8:48 pm
Wow… I’m so glad that my DNS propagation hasn’t finished yet. Otherwise I might be upset that the hosting provider I just went with will be down for a large amount of time….
Seriously, though… I really don’t mind THAT much. But it is disappointing to see that Dreamhost is having more than a little trouble in the past few months.
February 22nd, 2007 at 8:49 pm
All of them will be affected
February 22nd, 2007 at 8:51 pm
@Kevin: ALL of the servers will be affected.
February 22nd, 2007 at 8:52 pm
And I was just telling my users today that my site was going to be up and available unless my ISP did something dumb.
I gotta go with Kreig. Z. a mirror data center would be a great idea. If you can’t get too far from LA, at least get on the other side of the San Andreas. Bakersfield, Lancaster/Palmdale, Barstow. Even Riverside, San Bernadino, San Diego would work.
February 22nd, 2007 at 8:57 pm
Let me get this straight..
.. your entire network will be “down” for 3 hours, meaning all customer’s websites will be down for three hours, as you have zero connectivity redundancy so at least those servers which aren’t affected are available? That’s insane.
February 22nd, 2007 at 8:57 pm
shit
February 22nd, 2007 at 9:01 pm
CRAP!!!!!
I have yet to recover from this morning’s disaster. NOW, what do I tell my customer,… who pays the bills?
February 22nd, 2007 at 9:05 pm
I love DreamHost and hate to complain, but I have to admit its a bit embarrassing that a hosting company of your size doesn’t even have redundant nameservers outside of their single data center.
I mean just buy 1 server in the most boring landlocked midwest state and let it at least serve dns and dreamhomestatus.com throughout outages.
Let me know if you’d like help setting it up.
February 22nd, 2007 at 9:05 pm
Will people visiting dreamhosted sites just timeout or whatnot or will there be some sort of indication that they will be able to access the site later on?
That would at least mitigate SOME of the annoyance.
February 22nd, 2007 at 9:09 pm
What you don’t have a mirror site somewhere else ? Come on guys these down times are a joke!
February 22nd, 2007 at 9:31 pm
You have no back-up data center? That is not encouraging.
February 22nd, 2007 at 9:31 pm
@ KriegZ: They’re doing us a favor, taking that eyesore you call a Web site offline.
@Will: Are you illiterate or just stupid? The note from the building manager says they need to replace a cable on the UPS.
@kevin: As the post clearly states, ALL services are going down. Can’t you people read? Or are you in such a hurry to say something stupid, you don’t read through the entire question?
@Bill: I’m sure there are plenty of shared Web hosts with redundant datacenters out there. I’m also sure they cost a hell of a lot more than $8 per month. And again, I actually think DH is doing the world a favor by removing the FrontPage special you call a Web site. Minimalism isn’t avant-garde when it looks like you just didn’t try.
@Brendan: Again, learn how to read and you won’t look like a fool. The problem is with the building’s power equipment, not its telecommunications. They have to replace a high-cycle trunk power line. You’re suggesting they should have two of those?
@Mike: Try this: “I’m such a money-grubbing ass I charged you an arm and a leg for discount hosting. That caught up with me. It’s my fault for not explaining that yes, even — especially! — on the Internet, you get what you pay for.”
@michael schurter: For most customers, that wouldn’t make a difference. For those who have purchased a second hosting plan outside the DH network for emergencies, it would. So there’s some validity to asking the question, “Why do I have to use DH’s nameservers? Why can’t I use my own / my registrar’s nameservers? I think I have the answer, and it’s twofold: It’s a lot easier on DH to manage your services if they control the nameservers. In this case, even if they had their own nameservers outside the network, there’s no way they could change the records for all the Web sites people would want to move elsewhere and still concentrate on all the other tech support requests they get. That’s why they don’t have redundant nameservers outside the network and why they don’t let you use your own nameservers.
@john: I’m assuming you’ve seen a 404 error before. They’ll get the Web browser’s generic 404 error. If you want to see what it looks like, unplug your Internet connection and try visiting your site.
@Chad: See previous answers. If you spent $80 on hosting instead of $8, you wouldn’t have these problems.
February 22nd, 2007 at 9:32 pm
@mark: Again, I’m sure that if you spent $80 per month on hosting instead of $8, this wouldn’t be an issue.
February 22nd, 2007 at 9:55 pm
Chad, just a thought, since you’re being snarky … a 404 error? A 404 error comes from a working server. Since the whole of DreamHost is going out, what’s going to be left to present a 404?
And supposing you pull out the modem cable, now what’s going to generate a 404? After all, there’s no server around to send you one.
I think you meant a time out page. A time out page does not have an HTTP status code … think about it. (The only exception would be if the timeout occurred upstream and beyond a proxy server, and the proxy server might return a status code, but, generally, this will not be the case for anyone. Goodness knows what code you’d get for that… I’d have to go look through the list.)
February 22nd, 2007 at 9:57 pm
I simply cannot believe that, after the severe problems last year, Dreamhost hasn’t set up some kind of contingency plan.
February 22nd, 2007 at 10:02 pm
Doug,
Not all servers have to go down (1 out of 3 data centers). However all connectivity does (that means all 3 data centers are no longer internet connected).
Thus, it’s equivalent to a multiple data center, host wide outage. Why have 3 data centers when 1 going down locks out all three? Each data center should have a redundant link and the lack of redundant DNS (so that DNS updates can still be made) is also highly dubious.
I cannot understand why you’re abusing people for questioning why there is no redundancy (so at least the other two data centers aren’t impacted)?
February 22nd, 2007 at 10:13 pm
hehe….doug’s a dumbass
February 22nd, 2007 at 10:40 pm
Personally, I like the KriegZ and Bill websites. KriegZ’s gallery has some very good stuff. Bill’s wiki has a tonne of content and appears to use the same wiki engine as wikipedia, making it a familiar and easy to navigate site. What’s wrong with either of these websites?
Doug, if your website is so cool, why aren’t you linking to it?
I’m also curious what you think of this guy’s website: http://www.useit.com/
:-p
February 22nd, 2007 at 10:58 pm
ha, wasn’t the whole internet designed on an architecture that avoided this?
poor planning, poor implementation, poor service.
I’ll put money up that the access to DH’s “pissing customer money against the wall” blog remains in tact.
I like the correlation between:
“compromised to the point that immediate action to repair it is necessary”
and
in 3 days
come on ground fault, come on ground fault….
_______________________________________________________________________\
“I simply cannot believe that, after the severe problems last year, Dreamhost hasn’t set up some kind of contingency plan.”
–They have, the DH team have perfected the conversion of your bills into downtime, beer and infantile ninja/pirate jokes.
February 22nd, 2007 at 11:54 pm
I just wanted to say that actually ns2.dreamhost.com and this site are kept on a completely unrelated network, at a fourth data center, in San Francisco.
So, nameservice and this site (though not blog.dreamhost.com or anything else) will still work during the outage. If you’d like to point your dns records, etc, to a temporary outside server of your own for the duration of the outage and still keep your nameservers as ns1.dreamhost.com, ns2.dreamhost.com, and ns3.dreamhost.com, that should be fine. Also, you can update your whois records to point to any other nameserver right now if you’d prefer!
February 22nd, 2007 at 11:59 pm
you guys are killing me. outage after outage after outage…one data center going down takes down all three? who was the genius behind that one?
February 23rd, 2007 at 12:20 am
I guess this would be my first real long outage since I got Dreamhost. I don’t think it’s that bad - shit happens.
Keep on the good work Dreamhost!
February 23rd, 2007 at 12:46 am
bene, i havent a clue where you bank, but yes, if my bank’s computers went down i would take my money out asap. if they can go down, odds are they’re neglecting other things as well and my money isnt safe. thankfully, that’s never happened to me or my bank. that’s just silly
February 23rd, 2007 at 1:01 am
If your website needs 100% uptime, buy hosting from a company that can do just that. It’s not rocket science.
Deal with it.
It’s not something they have control over or can prevent in their current situation.
3 Hours, though? I doubt that. With something on a time deadline such as this I’d expect a nascar pit crew on this piece of cable.
February 23rd, 2007 at 1:03 am
brian: how do they not have control over one data center taking down all three??? that’s *ENTIRELY* in their control and they messed up on network design.
February 23rd, 2007 at 1:22 am
And it’s entirely in your control to switch to a host who can give you that 100% uptime you want. :>
Honestly… this isn’t that bad.
I’d rather have them announce it and fix it in 3 hours, than not doing anything and coming to the conclusion *something* broke *somewhere* and they need about a week to figure out what it is. ;P [they = datacenter people, not DH]
February 23rd, 2007 at 1:28 am
it’s like you people have all been brainwashed. i am moving to a different host. i dont need 100% uptime and i dont expect it. my problem is that dreamhost seems to have an emergency every other day. sure, certain things they cant control, like having to reground the building, but having the other data centers go down for that is just crap. i’m getting out asap
February 23rd, 2007 at 1:28 am
If you don’t like the services provided by dreamhost then why don’t you just switch hosts? Seriously, if your mechanic doesn’t do a good job on you car, you switch mechanics right? It isn’t like anyone is forcing you to have dreamhost be your host. And as far as your ‘clients’, you should have them on a 100% uptime guarantee server if you are so worried about their sites.
February 23rd, 2007 at 2:49 am
Keep up the good work DH!
As for all the whiners, no one is saying you HAVE to stay with dreamhost. I am sure that at 1am your customers will be tucked up in bed and fast asleep. I would understand if they decided to make the change at 13:00 hrs - Also at least they have located the problem, informed you and are taking action…. or would you prefer that it wasnt detected, and then all of a sudden your server goes offline for 4 hrs whilst they fix it… At least this way you have advanced warning….
Geez….. and those that whine about having payed money for hosting…… so what….. Everyone pays for hosting with DH, but a majority of us work within IT and/or for an ISP and understand these things happen….
GL DH for the change……..
February 23rd, 2007 at 2:59 am
We just transferred to DH from Lunar Pages a few weeks ago. I am in shock…in my 3+ years with LP, not once did the site go down for multiple hours (or even an hour)…and they offer hosting at $7 a month…
I don’t believe in complaining…here is some constructive feedback for the DH folks:
Learn from this and ensure that it does not happen again. Regardless of cost or effort, do not neglect your duty to your customers…this means taking stock of what you have and what you may lose. Lastly, remember that you can do 100 things right and just one thing wrong, and it is what you do wrong that your customers and potential customers will remember.
This is strike one…
February 23rd, 2007 at 4:00 am
That really sucks. We’ve just transfered hosting this week, and it has been a nightmare. Our email has been down for 3 days, and when I emailed the dreamhost support it bounced on their side.
What do I do? Anyone got suggestions of a better hosting service?
February 23rd, 2007 at 4:09 am
Hopefully, the world will go, and we will all be alive after this. We might even have a good time. This is the time to tell someone you love them!
February 23rd, 2007 at 4:59 am
PLEASE PLEASE PLEASE use something like this so your customers from all over the world will know when the downtime will take place THEIR TIME
http://timeanddate.com/worldclock/fixedtime.html?month=2&day=25&year=2007&hour=0&min=0&sec=0&p1=256
February 23rd, 2007 at 6:17 am
Hanan: that seems a bit over the top - surely all we need to know is that PST is eight hours adrift of GMT? It seems reasonable to assume that we (the customers) know what time zone we are in.
It is a bit annoying that these outages take place during the day for those of us in Europe, but that’s just the inevitable price we pay for signing up for cheap US hosting …
February 23rd, 2007 at 6:22 am
I have to agree with Will,
Having UPS in rack for that gear is a necessary thing - Even if you’re in a Tier4 DC (which is sounds like you’re not, if they’re having these kinds of issues), the core should always have its own backup.
If you guys are looking for a DC in the Mid-west - look me up. I work for some really good Tier 2 and 4 DCs.
February 23rd, 2007 at 6:25 am
Please if anyone can honestly say that this is going to cost them more then … say, $100USD; maybe you should reconsider how much you spend on your hosting plan.
If a 3 hour outage, that starts for me at 3AM; is such a traumatic event for your site/business, maybe you should consider finding a new host that will charge you $169USD a month with an iron-clad guarantee of uptime. losing you as a neighbor on my shared server would be no significant loss to me.
Thanks for the update DH, you’ve always given me good value!
Peter
February 23rd, 2007 at 6:40 am
I first would like to thank DH team for their overall good service and in particular for their excellent support and straightforward communication.
But in concern of reliability and up time, I think this should be quite easily improved (my personal web site survey reports an uptime of about 99%).
First, I don’t understand why it isn’t possible to rent an electric generator for this planned power down at they do in some other place (e.g. http://www.hawaii.edu/infobits/f2000/outage.html ).
Secondly, since it seems the data center is located in area where power failure happens time to time, wouldn’t it wise to add UPS (Uninterruptible Power Systems) protection (e.g. http://www.powerware.com/UPS/Products.asp#large ).
Since reliability is a main concern for any professional service, I implore DH team to put it in the first priority, for its own sake…
February 23rd, 2007 at 6:54 am
i love everyone who is giving these bright ideas, 1 thing you forgot whos paying for it
keep that in mind next time before you try to make smart ass comments to dreamhost, dont you think they have thought of that already?
February 23rd, 2007 at 6:56 am
giles i have no problem with their hosting and i just transfered not to long ago, its probably problems on your end don’t blame dreamhost unless you fully know. and maybe your e-mail you are sending from has been black listed there are millions of variables.
Plus dont go asking for another host on a dreamhost uptime blog…
February 23rd, 2007 at 6:59 am
Marcos Pinto, lol if the power center goes off line, what else do you think will go off line… duh
February 23rd, 2007 at 8:04 am
This downtime is unfortunate, indeed - however if there are 100eds of servers (as I understood it) being down for ~3 hours, Dreamhost either don’t have a diesel powered generator running the entire thing, or they just take down the entire system due to safety issues (if you have worked with 3-phase equipment and with this level of amperage, you know what it means).