Planned network outage July 11
On July 11, 2008 starting at 10pm we will have a brief network outage to restructure our routing tables. We are planning two hours to deal with any quirks that may come up, however our Cisco network engineer estimates the total downtime to be under 30 minutes.
During this downtime DreamHost will be completely inaccessible, however no data will be lost. Email will be deferred on remote servers and delivered once the network comes back online.
This restructuring is to put the final fixes on various packet loss issues we have applied “band-aid” fixes to in the past, including installation of an extra core router at one of our facilities. Cisco TAC believes they have isolated the problem to duplication of routing information that will require changes to router interfaces to repair. Unfortunately this also requires that those interfaces be reconfigured from scratch, so there will be some downtime while this is done.
Sorry about no updates here, we were updating our “reminder” post. Here are the reminders, just copied and pasted:
Update 7/12/08 12:32am PDT:
The network was back up briefly before being taken down by an unexpected hardware fault on one of the central routers. It’s currently coming back up and with any luck, things will be returning to normal shortly.
Update 7/12/08 2:58am PDT:
Everything’s been back up for a while now and we’ve been monitoring everything very closely. From what we can tell, the previous issues are all gone and everything’s running smoothly. We’ll continue to monitor the network very closely over the next 24 hours to be absolutely certain no strangeness pops up.
June 18th, 2008 at 8:52 am
You’re just taking time off to play with your new 3G iPhone. You’re as bad as a cheating boyfriend! “Oh sorry baby, routing tables and packet loss, you know, I don’t know when I’ll be back.”
June 18th, 2008 at 8:56 am
Datacenter-wide network outage up to half an hour?
Not cool.
June 18th, 2008 at 9:00 am
I’m assuming that’s PDT?
June 18th, 2008 at 9:04 am
It’s not 100% clear if you mean “dreamhost.com” or the whole dreamhost hosting network (I assume it’s the latter, but it could be worded clearer)
June 18th, 2008 at 9:06 am
Why can’t they just get everything working once and for all???
June 18th, 2008 at 9:34 am
Thank you very much for the forward warning.
June 18th, 2008 at 9:55 am
@Thomas, Because that’s unreasonable and not at all feasible, everyone has upgrades/maintenance that has to be done, It’s really annoying to read comments from people with unrealistic expectations who enjoy jumping on the bandwagon.
ThanksDH for the advanced warning of this, it’s appreciated as I can now notify my users not to freak out at that time.
June 18th, 2008 at 10:17 am
“OMG!!! Not another outage! This is NOT low severity! Thats it, I am finding a new host that will be cheaper, more reliable, and give me a free puppy! Goodbye Dreamhost! Blah blah blah.”
There, all that is out of the way…..can people not fill this thread with empty threats of leaving.
Just my rant for the day.
Thanks DH for the FYI
June 18th, 2008 at 10:18 am
Oh sure….as soon as I post its gets changed to HIGH severity…..’DOH! there goes part of my rant.
June 18th, 2008 at 10:47 am
Seems our custom severity thing isn’t saved when I hit “Save Draft”! I made a bug for our development team!
June 18th, 2008 at 11:35 am
@Sean:
And remember: “I want my 5 nines for $7.95 a month!!!1!11″
Yes, so many mission-critical, enterprise grade sites run on shared hosting, I’m sure.
However, if a hoster were giving away a free baby Tasmanian Devil, I might have to switch.
@ the other commenters: I doff my hat to your calm and measured reaction to this planned outage.
@Thomas: Because apparently, Cisco put out a version of IOS that had some serious bugs in it and it has taken Cisco it’s usual long time to figure this out. And no, DH’s Cisco enterprise-class router is NOTHING like the Cisco/Linksys router you might have. You might direct your ire at Cisco. That said, others have tried and it nary makes a dent in the armor of their complacency and glacial-movement-like IOS fixes.
June 18th, 2008 at 12:44 pm
Is this why I get the “# It doesn’t seem like brokenfloppy.com has been set up for phpbb.dreamhost.com?” error?
June 18th, 2008 at 1:08 pm
Hmmm… No backup of the config to upload?
Geez. You got some stupid Cisco people working for you. Just saying. I’m done and so are you.
June 18th, 2008 at 2:34 pm
@IceWulf: Since they are changing the entire config what good would the backup of the prior config do? Are you really that dumb?
June 18th, 2008 at 2:45 pm
Wahhhhhhh, waaaaaaaaahhhhhh, wwwwwaaaahhh.
June 18th, 2008 at 9:20 pm
All these people rant and rave that other hosts don’t “screw” them up…. At least dreamhost tell you what they are doing and just not go down like 99.999999% of other hosts. If you say other hosts don’t go down then please get off my tubes your wasting my bandwidth. The only reason people see all this so called bad stuff about dreamhost is because they tell people when there are problems. In my books that is honesty not like some other overpriced retarted hosting company.
And for your information I have never had any problem with dreamhost for the time I have been with them. I have had a total of about 20-30minutes downtime my entire time here which is nearly 2 years now.
So if you want to complain first cancel your account then go sign up for an overpriced hosting company and make a “I hate dreamhost” website… ok? then complain all you want there.
Cyaz
p.s thanks again for the heads up DH
June 18th, 2008 at 11:35 pm
Hi,
In Italy will be Saturday early morning when you will apply the fix.
We do have a lot of time to inform our customers about this.
Good luck DH team
June 19th, 2008 at 12:18 am
Good luck!
June 19th, 2008 at 12:34 am
Hi, iam sahith.
Friends if u r in dreamhost get out of there,they are blood and money suckers,yesterday disabled my account without any warning.And they are not so best.,their server always down.
If you are in dreamhost,,get out of there and choose another best host.
June 19th, 2008 at 5:25 pm
Siri,
What did you do to get your account “disabled without warning”? Maybe you are not so best?
June 19th, 2008 at 6:14 pm
@siri: Take a hike you troll. I’ve been with DH for a while and I’ve hardly ever had a problem. This post seems to be pretty calm, which is a nice change. To those who are pissing and moaning about an outage of this scale because of routing tables, I doubt a single one of you understand any of the technical information actually associated with 1) running a hosting company of any size 2) the infrastructure of what makes the internet and your hosting possible (hint, it ain’t “tubes”). So, just be greatful and thankful that DH has the system in place to communicate this kind of stuff, the technical know-how (and tech support contracts with vendors) and the sheer awesomeness to be what they are.
June 19th, 2008 at 10:54 pm
Is this issue the reason why I’m getting “Disk Quota Exceeded” messages when trying to upload files over FTP/SCP? Or is this a seperate issue i should contact support about?
Cheers
June 20th, 2008 at 3:33 am
Kelly wrote:
Seems our custom severity thing isn’t saved when I hit “Save Draft”! I made a bug for our development team!
—
How to turn this into an hilarious practical joke, “custom severity thing”? I say you cookie all of the ranters, and then invert the security warnings for them, or perhaps give them new levels, instead of low to high, maybe Kittens -> Extinction Level Event!
One better?
Remember their entered names, and provide several custom severity templates like: “Don’t worry #{name}! You’ll be dancing with kittens” all the way up to “OMGBBQTURNIP #{name}, #{name.upcase}! All ur Sites R deleted, it’s the ENd of all things, quick, SacrIFice a ChickEn to BaFomet, NOW.
Oh…because I know someone will point this out, I misspelled some words intentionally.
June 20th, 2008 at 7:14 am
July 11th?? All my sites running DB’s are currently down now…did you guys start this early?
June 20th, 2008 at 7:57 am
http://www.timeanddate.com/worldclock will help you figure out what time this is in your part of the world (July 11th 2008 10PM PDT)
June 20th, 2008 at 8:02 am
Jason: An excellent idea! I think I’ll just consolidate it to “Kitten Extinction Level Event.” I’ll let everyone determine for themselves if they like kittens or not!
June 20th, 2008 at 8:05 am
“is this why my site’s down”?
No. Unless you really are ahead of everyone else (and your server is too, No.
June 20th, 2008 at 10:21 am
FUCK I JUST LOST TWELVE MILLION DOLLARS BECAUSE OF YOU JOSH.
June 20th, 2008 at 10:48 am
Good luck on restructuring your router tables and whatnot. Excellent choice on the time slot by the way, 6AM - 8AM Saturday morning in the UK.
Try not to get too stressed out during it all.
I vote for a plugin to be created to allow toggling between normal and lolcat severity warnings. Lolcat warnings should also come with proportionally freaked out looking cats to match the severity.
June 20th, 2008 at 11:28 am
@lavalamp:
Excellent idea. Would do a lot to raise the dignity around here, after the ranters and hyper-active kiddies get through. Could we also have a lolcat rating system for posts?
You could do a lot with the phrases “I kan haz load in my nappeeZ” or “Kan I haz adderall now?”.
June 20th, 2008 at 8:09 pm
Are you F*ING kidding me?
1- Why can’t this be handled in batches with failover ?
2- 10pm PST scheduled for a downtime ? I work in Technology and SaaS - whenever I’ve ever had a scheduled downtime by a vendor, or to a client, it’s been in the middle of the night to ensure low impact. 10pm PST is not a low impact timeframe.
June 21st, 2008 at 2:39 am
@Jonathan - remember young grasshopper, not all hosted clients are in the same timeframe. If one strives to satisfy all, one succeeds only in frustrating one’s self.
June 21st, 2008 at 2:53 am
Jonathan: No, they can’t do it “in batches”. No they can’t do it with failover. They’re not doing what you think they’re doing. This is open heart surgery, not a manicure. Also, 10pm PDT is pretty close to the middle of the night. it’s 1am eastern, so most US clients (the bulk of their users) will be either out doing something or asleep (or doing something /else/ that doesn’t require a computer), being late friday night, not surfing Dreamhost.It’s as good a time as any, since it’s very early in Europe on a saturday, and least likely to impact them much. Sadly, this means Asia gets screwed. Obviously Dreamhost hates Asia.
June 21st, 2008 at 2:56 am
Will this outage affect all three dns servers (nsX.dreamhost.com) as well?
If so even externally hosted sites and services will be affected as long as the dns zone is hosted at dreamhost.com.
I wish dreamhost would allow AXFR dns requests making it possible to add additional external dns slave servers.
June 22nd, 2008 at 12:07 am
Saturday 3pm, Nice. Lets hope 30 mins is close to the mark.
June 22nd, 2008 at 5:05 am
Thank you, Dreamhost, for not censoring the comments on this blog . . . it provides many laughs to start my day in a happy way. Thanks for being up front and truthful when there are problems. The only company that never has problems is the one that has no clients.
June 22nd, 2008 at 11:27 am
Hostin 100% uptime only 35$ year
http://vexxhost.com/hosting.htm
Summer
June 22nd, 2008 at 12:52 pm
Just as an FYI … I *hate* people who come on here spewing how “All” companies/hosting providers have problems/outages.
No. They Don’t.
They just don’t cost $10 (or less) per month.
Pay $150/mo (Or say, $350/mo at rackspace) for a dedicated server and redundant network/services and you will *not* have these constant problems and outages. DH is a bargain bin hosting company - you *do* get what you pay for when dealing with internet hosting.
DH is excellent in terms of reliability, uptime, and features for a cheap/bargain hosting company @ $10 a month. If you want better than this, you are going to have to pony up and go to a higher tier hosting company. Anyone else charging $10/month is going to be very, very similar.
- Roach
June 22nd, 2008 at 2:35 pm
Yes. They do.
If they didn’t, you wouldn’t have needed the redundancy example for reliability. The reason for redundancy is because they all have problems.
June 22nd, 2008 at 6:49 pm
I must say… this thread is hilarious… I think DH could make a ton of money just by offering counseling to some of these people.
June 23rd, 2008 at 7:34 am
is dreamhost down?
June 23rd, 2008 at 4:08 pm
Hello, I am new at DH, I have faith in you guys!!, and I hope it could be fixed without any problems. Good LUck!
June 23rd, 2008 at 10:23 pm
“Cisco TAC believes they have isolated the problem…”
June 23rd, 2008 at 11:31 pm
Seriously, anyone who understands networks will totally understand this and just go with it. and seriously, while I have doubts that this will resolve its self in 30 minutes (unless they’re using static routing with preplanned configs) this really isn’t that big of a deal, most of you are probably hosting your own blog which your mother and your two best friends read every day, instead of just talking to you. Oh my god, you might have to be inconvenienced enough to actually talk to real people. holy fuck. Everybody has problems, nobody’s perfect, that’s why they call it five 9s. there’s no such thing as perfect up time. so each and every one of you that’s complaining, quickly quit your bitching and move on with life. Nobody’s going to care if everyone in China can’t read your blog for one day during lunch.
If you seriously have nothing better to do than bitch about the up time of your personal website, then you should probably head out to the beach right now and attempt to get a friggin girlfriend. So ladies, you’ve got something like 2 weeks to plan what youre going to do while you cant update your blog for 30 minutes. Get to it. You obviously have nothing better to do.
June 24th, 2008 at 6:09 am
oh c’mon there was a shitty download a little while ago it was about over 10 hours ii dont want any more, can you please stop the shiet?? you always giving lame excuses like your employ suddenly switched off the server while he was over drunk and any other person were even more drunk to take that up asap??
June 24th, 2008 at 8:33 am
Nice time choice (for UK), and nice that we all know beforehand. Chances are this won’t affect me at all as I’ll be sleeping.
June 24th, 2008 at 8:47 am
Great timing as always–the space shuttle is docking with the international space station at that time (we rely on the shared MySQL databases)…we really shouldn’t have to upgrade to a dedicated server! Thanks,
-Commander X
If this happens again we will move to hostmastermonsterdragonfrog.com where they have 115% uptime!
June 24th, 2008 at 11:02 am
I agree with Jimbo from 6/22. These posts are way better than The Onion. Good luck DreamHost!
June 24th, 2008 at 3:09 pm
Message from you (Jun 23rd, 2008 - 04:28:27 / #2544496)
Subject: site down
Reply from DreamHost (Jun 23rd, 2008 - 06:09:32 / #18800251)
Subject: Re: site down
Message from you (Jun 24th, 2008 - 14:26:34 / #2547392)
Subject: site down
man these guys suck is anyone know a reliable host i don’t care of specs or price i just need to find a new place to move to
June 25th, 2008 at 6:42 am
@dreamhostsucksass
There’s this cool new thing on the intertubes called GOOGLE. Use it.
Or you can just read the comments to this post and see the company names trolling for customers.
June 25th, 2008 at 8:29 pm
Yes, an adsense block will be a great source of income for DH. I wonder why they aren’t using it. Probably the ads too will be just complains?? haha..
June 25th, 2008 at 8:36 pm
@dreamhostsucksass
I am monitoring a test account at dreamhost, just to keep track of uptime and speed. Let me tell you, Dreamhost’s uptime is better than many other webhosts. Maybe a few servers or clusters have some issue.
June 25th, 2008 at 9:55 pm
“is this why my site’s down”?
Thanks for the heads up. I really appreciate it.
June 26th, 2008 at 6:43 am
Thanks for the advanced notice guys, really appreciate it….
I’ll most certainly not notice this tho… it’s going to be 2am on a saturday here, I’ll be either out or sleeping but by all means away from a computer (and most of my audience will be too)
June 26th, 2008 at 4:35 pm
La verdad es que este servicio es una mierda!!!
June 27th, 2008 at 7:19 am
@Jonas: no, it will not affect all 3 dns servers. ns2 is hosted off-site at our San Francisco colocation (the same rack that holds the server for this very site)
June 27th, 2008 at 7:19 pm
hi guys, what is the point of this type of annouce ? It seems like there are plenty of other issues to worry about other than this.
I think my site has been down most of the day and I came to the status page expecting to see an update but I see this towards the top and no explanation that is obvious as to why my site is down.
I think you need to look at you priorities to figure out what is a priority and what is not..
al
June 30th, 2008 at 6:03 am
Seriously, shut the fuck up about your small ass sites that get no traffic that you made and uploaded with Frontpage Extensions. Nobody wants to hear it, find another host. Funny thing is, you’ll wind up back to square one because you realize what a really shity host is like. Until you people are Cisco certified and understand how to even turn on these kinds of routers, once again [STFU] NO ONE CARES~!~~
July 6th, 2008 at 5:16 pm
Cool. Glad you told us. I have a busy mirror but this advanced notice is welcomed so I will let them know. This will also probably get the mirror list maintainer notified too.
July 7th, 2008 at 12:08 pm
Thanks for the update.
July 8th, 2008 at 8:31 pm
lol at you losers staying with dreamhost.
I moved my hosting 7 months ago and have had 100% uptime since then!
looking down at you people makes me happy that i left this crappy host behind!
July 8th, 2008 at 9:38 pm
Thank you for the notice, DH!
I really can’t believe some of the people here. I came to DreamHost after having a bout with a really, really bad host that was more expensive.
Thanks again for the heads up.
July 10th, 2008 at 4:15 pm
I wish you a successful maintenance night… and hopefully everything will be fine. Good luck guys!
July 10th, 2008 at 10:08 pm
This seriously brightens my day whenever I read this. Yeah I almost never check the blog but I somehow manage to check it before a outage. I have yet to have a complaint with DH other than a few minor issues, though to me they weren’t minor at the time. DH has been amazing to me. I appreciate the advance notice and wish you the best of luck. Also to the others that think this is simple…it really isnt this is the sort of problem that a team of network engineers could take hrs on to get it right, if everything goes wrong.
July 11th, 2008 at 1:15 am
slice down. and the most inconvenient time too. >_>
July 11th, 2008 at 6:24 pm
i am just glad its on a saturday morning at 1am (EST)
things happen, most ignorant people on here don’t realize that….who knows maybe the sites will go faster when the routing tables get reconstructed….as long as they arent down for 5+ hours we are good!
July 11th, 2008 at 10:22 pm
Hey! maybe next time you guys could use a wall shell script to alert users logged in via ssh that they are about to get kicked off. Good thing I use vim which creates backup swap files in case something like this happens..
July 11th, 2008 at 10:48 pm
Hello people. Anyone have twister?
July 11th, 2008 at 10:55 pm
It was funny because I was just trying to read a friend’s blog and it didn’t work. I then realized they probably use DreamHost and it turns out I’m right.
July 11th, 2008 at 11:09 pm
Hah Me too- I went to check up on Fluidmac (makers of Chopper for iPhone, which you should buy at the low price of $8- it’s rad) and they were down. So I whois’d it and sure enough it’s my own lovely dreamhost.
Checked my sites and they’re down too. At least it’s scheduled
I missed the RSS feed cause I was out in Yosemite backcountry for 9 days. Yay.
July 11th, 2008 at 11:18 pm
WHY didn’t I get an email about this??????
It’s not in my newsletter, it’s not in my inbox.
What, I have to check out you crappy blog to find out I’ll be wasting my ad dollars while you’re down?
Poor communication.
You should send us an email with the subject:
IMPORTANT: SITE WILL BE DOWN
Now, how do I know when you’re back up?
I guess I’ll have to check your blog.
ugh…
I want a .2% refund (so far) for this month.
July 11th, 2008 at 11:25 pm
well i dont care what anyone else says… i think dreamhost f’n rocks… i rarely have issues and the services are outstanding.
i knew about this outage a a while back.. and i made the proper preperations.
thank you dreamhost and keep up the good work!
July 11th, 2008 at 11:29 pm
Xkeeper @ 35
So they posted something here 3 weeks, 2 days ago. Like I said, I am not in the habit of needing to come here and check to see if there is gong to be a planned outage or not. As far as an RSS feed I wasn’t aware I needed to have one running. The usual method IS via email. My other webhost, SPRY sent me an email yesterday telling me that THEY were going to do some scheduled maintenance this evening too.
Pretty efficient, and I was prepared for it.
Being told 3 weeks, 2 days in advance is kinda silly. Even if I had gotten the word back then, I would be sure to forget it 3 weeks, 2 days later.
Bucko @ 36:
Dammit, it is always fun to part a fool from his money. lol
Maybe you misunderstood me. I don’t really care that Dreamhost is doing some scheduled maintenance and my sites are currently down. It is no big deal. My only complaint here is that they could have easily sent out an email today or last night stating that they were going to shut things down for a few hours to fix some problems. Not all of us are clued into needing to have a RSS feed in case they need to do something. I also don’t need to have an RSS feed to tell me about outages or problems with other servers or hardware that has nothing to do with MY particular server.
dreamhostsucksass @ 50
Wow, they took an hour and a half to get back to you? My gawd, what did you do?
Trust me, Dreamhost is one of the better hosts out there. I have used most of the major ones. They ALL have their moments.
If you want more reliability, you are going to have to pay for it. You could go to SPRY and get a VPS if you want, but you will spend a minimum of $100 per month and still wind up with some down time. It is a computer network, not a toaster.
July 11th, 2008 at 11:41 pm
I am also done. I made the mistake of using Dreamhost for mission critical file hosting for a business and since I joined 6 months ago after hearing nothing but good things about it, Blingy has caused me days of downtime and now they have taken the entire network down on purpose. Even with redundancy hosting, it is still down. I can’t believe that people are defending this kind of unreliability. Perhaps they don’t care because they only host blogs on it or something but some of us use cheap, high bandwidth file hosting for business.
Being told that they are an unreliable host like this is no substitute for reliable hosting.
Does anybody know a good alternative that has a lot of bandwidth (other than globat, that’s where I came from)? I am now looking myself and when I find it, will be attempting to get a complete refund from dreamhost but assuming others are in the situation, I would like to know what they can find too. Please don’t suggest dedicated hosting as that has its purpose and massive monthly bandwidth file hosting is not it (that would cost me thousands in dedicated hosting).
Where did you go to 7 months ago Daniel?
Globat was pretty bad but at least I never had any downtime in the 2 years I was with them!
July 11th, 2008 at 11:50 pm
Thanks for the warning before hand. Everything is working fine now.
July 12th, 2008 at 12:53 am
@George.. “Blingy has caused me days of downtime”
The “Blingy Clusterf*ck” took place about 2 months after I switched from another host that cost me nearly 10 times per month, not to mention all the time and energy I had to sink into maintaining the damn thing myself because support wasn’t supportive at all, unlike DH who has answered all my emails in less than 4-6 hours. I was down for about a week total, with intermittent issues totaling probably a month… Since then, there hasn’t been a single hiccup in my service, and even my *gag* Joomla sites are running super fast.
Asking for advice on where to find alternate hosting here is just stupid. Get off your lazy ass and search it yourself. Either that or put on your big girl panties, suck up the 30 minute outage and shut that gaping piehole.
July 12th, 2008 at 3:12 am
when the problem can be solved?
my web is still not up?
It is 3am now, one hour than DH proposed.
Anybody can provide the update?
July 12th, 2008 at 3:28 am
PMSL @ George Helyer
“mission critical file hosting for a business”
You would trust a hosting company that costs less than $10 a month for ANY business related venture… even I find that naive.
I’ve come to live with the ups and downs of hosting with DH, yeah sure when they are up they are great, I can’t fault them. But even I realised very quickly that if you want reliable (>99.5% uptime) hosting you need to look at VPS or dedicated server. I’ve got a VPS with another host and even that is only 99.7% uptime over time.
If your “mission critical” work really is mission critical, go speak to Rackspace because that is what they do, and leave DH for those of us who need cheap and cheerful.
On a lighter note, whats going on DH? I need my email now… please?
July 12th, 2008 at 3:38 am
OH PLEASE !!!!!!!
at least an error page! something like “working in progress” but not just NOTHING !!!
all my website still down! unbelievable!
July 12th, 2008 at 6:55 am
Dreamhost: Next time this happens, you need a landing page with the number for the Wahhhmbulance on it.
July 14th, 2008 at 5:02 am
Obligatory:
Posting in a epic thread.