DONE! Emergency router upgrade
Posted (August 17th, 2007 at 7:39 pm PST) by KellyTonight, hopefully starting at 10:30pm, we will be performing an emergency IOS upgrade to one of our core routers. This is to hopefully quench the problems we have been experiencing once and for all. The downtime is estimated to be no longer than 20 minutes, however if anything goes wrong, it could take much longer.
The impact of this upgrade is that it will bring many of our internet uplinks offline, and make a good portion of our network completely unreachable until the router comes back online.
Update 23:27 PM: We reloaded the router with the latest revision of firmware per Cisco’s (And our CCIE consultant’s) suggestion. All seems to be well, the router is passing traffic and the servers appear to have recovered gracefully.
48 Responses to “DONE! Emergency router upgrade”
Quit whining. Yeah, they did it on purpose to screw you.
The only people that can plan elaborate plans like that are the Illuminati, Microsoft and the US Government.
Everyone else just fights their way through day by day.
BTW - I hope you’re happy elsewhere, but please don’t waste any more bits here.
Actually, its cheap.
I’ve been with DH since 2001 and this is the first major downtime any of my sites have ever experienced. For every story like yours I’m sure there are many more like mine. Yes it is very frustrating to have a downed site (one of mine is still down!) but it is not nearly as common as some of you claim. Unless you want to fork out more money you probably won’t find a more reliable company.
More bits? Ok so dreamhost might not be as bad as RegisterFly, but as of late they have not been keeping paid users happy. Maybe I tried to make a joke about it too much. Sorry if you got mad at my response. But dreamhost does have its issues but maybe at its price it is worth it.
Most of us know you try all you can to bring the service 100% back !
Good luck and thanks’ for your long working hours guy’s !
Tycho
@JR
Ok so maybe you are right and I’m wrong. Whats your traffic like every month?
Moe - shouldn’t you be busy signing up somewhere else and bitching about them?
lol. yea i guess you’re right.
I found this article on the worst 13 hosting companies (no dreamhost is not on it). If anyone is thinking about switching over don’t go to any of these guys.
http://www.webhostingunleashed.com/blog/worst-web-hosting-companies-may07.html
lol dreamhost noobs. Im rollin on dedicated 100mbs @ theplanet.com
Thank you for keeping us informed, DH, and for continuing to identify and resolve the various problems that have been cropping up. While outages do frustrate us all at times I just wanted to let you know that your efforts are appreciated.
While I’m not thrilled that there’ll be more downtime, I must say — kudos to Dreamhost for being 100% transparent. Admitting that one of your core routers died, and that a large portion of your sites will be out for a while takes serious integrity.
Yes, it’s unfortunate when something dies, and sites become unavailable. Yes, there has been a rash of downtime of late. Still, I think that, considering that I’m hosting 15 sites (albeit well-optimized ones) on a $7.95 shared hosting account, Dreamhost does a damn fine job most of the time. In all honesty, I can’t think of anywhere else that gives you as many usable (keyword being “usable”) resources for as little cash.
Cheap, 100% uptime, generous resources — pick any two.
I’ve done some admin work part time before, and I realize that equipment fails. When you have a network as large as Dreamhost’s, chances are that you’ll deal with equipment failures all the time — it’s simple statistics. The MTBF for a single drive may be 50,000 hours or so — but when you’ve got several thousand disks, that MTBF means that you’ll be replacing quite a few drives in a given month. Sure, downtime sucks, but the alternatives (either leasing a dedicated server, or using a less-transparent, less-generous host) suck more.
Best of luck to the admins, and I hope that the issues are resolved ASAP.
That said, if this downtime persists, I might have a hard time using my 3 TB of transfer this month. 
Then again, I don’t know any other shared host that I _could_ use 3 TB of transfer with…
I regged a new domain 2 days ago, not realizing that this stuff was going on at first. Will my new domain show up once the system is back up fully?
Shit happens. I’m not very concerned and am certainly not going to waste my (and theirs) time badgering all the hard working folks at DreamHost about something I’m sure they had no control over. Maybe I’m so forgiving because my websites don’t earn me $50,000 a day like some of you make it seem but I’m willing to bet these sort of problems are rare. At most other hosts they wouldn’t announce anything when downtime occurs and try to have you believe the problem is on your end. 
Thanks for all the overtime I’m sure you guys have been putting in. Some people tend to forget that since you guys have a responsibility to fix these things your days are affected much more than your customers. You are humans after all! (Though all the emails I get from robots are beginning to worry me. They’re taking over!)
good luck getting stuff fixed up, DH - you guys have had a rough run this week - hopefully you’ll get it sorted out and get some sleep
I have many sites on dream host. I’ve been with them for a few years…
Yes, downtime sucks. This is the first “major” outage I’ve seen.
I’m going to give them the benefit of the doubt. They found the problem - they are communicating it out the best they can and they have a planned fix for it.
It sucks… it really does… I’m getting digged, stumbled and reddit-ed at the moment… losing tons of traffic 
- but what can I do? Hopefully they get this resolved; I haven’t lost faith in them yet.
It always cracks me up to see the smug posters that shit on the poor folks who are tired of their sites being down with little explanation and no ETA. Y’know what? Their whining is annoying, but you’re not helping. In fact, you’re more obnoxious than the people whining about losing money or their sites being down.
Do us all a favor and knock it off.
@Vee
Shutup nig.
Actually, V - I disagree.
This is their blog. Their little anybody can sign and write blog. I generally believe 10% of what I read in the comments section of these. I never believe anyone that says ‘I JUST SIGNED UP WITH NEWSERVER.COM!’ because they are probably shilling for their own site, knowing there are a few people in a state of weakness right now.
Why publicly bitch and moan when it’s well stated at the top of this blog that WRITING HERE IS NOT AN OFFICIAL WAY TO CONTACT DREAMHOST. If you have a complaint, talk to Dreamhost, I’m sure they will be HAPPY to write you back and hell, maybe even throw you a bone for your lost business. I don’t know for sure, I’ve never been so upset as to complain that much.
People here that are posting to bitch and piss and moan are doing it in the same way a child has a tantrum. They want to be HEARD while they yell. If you were in a restaurant and you heard a man publicly berating the waitress, you would think to yourself ‘What an asshole’. Same deal here. A smart business person goes to the manager and explains the problem, they don’t make sure every person hears their complaint so they can make snap judgments. It’s bad business and horrible etiquette.
If you don’t LIKE the service you are getting, COMPLAIN about the service you are getting to the people that can FIX it. They’ve made a convenient little button on your webpanel called ‘CONTACT SUPPORT’. If it is so bad that you are losing money that you can otherwise save elsewhere, go elsewhere.
The answers, opinions, and statements of the people on this blog are unimportant in the scheme of things. Dreamhost can’t verify the people writing here are even their clients and not just 14 year old annoyances who find it fun to rile people up. Hell, given that they had a DDoS attack on their blog means that half the comments made are probably FROM the aforementioned twerps.
So in all seriousness, if you REALLY feel (and I mean the collective you, just not you personally Vee) REALLY REALLY believe that you will get better hosting for a better rate with a better rate of communications with a nicer staff that isn’t down at all. You go find them and enjoy them and feel smug in your superiority of finding a better deal. Why be the asshole screaming in the restaurant?
Jamie
Some of you people are just complete douchebags. Go ahead and switch. Perhaps when your new host experiences their own issues (like the ones I’ve had in the past with the Gods at Media Temple) then you’ll realize that no hosting service wins 100% of the time. At least Dreamhost has the stones to stand up, say what they have to say, and let a bunch of assholes fire at will in an unmoderated comment section.
Hello people, i have a joke for the ocation:
DREAMHOST DON´T GOES DOWN. IT COMES UP.
:).
HAVE A NICE OUTAGE.
Your joke’s not very funny. Leave the comedy to the professionals.
Hi,
Sorry for not understanding the post, but timezones get me confused a lot.
The post says it was posted ” This entry was posted 3 hours, 25 minutes ago on Friday, August 17th, 2007 at 7:39 pm” and mentions downtime for “Tonight, hopefully starting at 10:30pm”.
Current Time Is 23:12:30 PDT (08:12 AM on Saturday, August 18th where I am).
So, was the router update already performed? But, then why is the status for resolved still NO.
If not, does this mean I can expect sites to go down any minute now for this update?
Or is this an advance notice for Saturday evening 10:30pm PDT?
A lot of times you do post both date and times when posting updates etc, and it really is appreciated if you do. Thanks!
Hey a Good Karma to you DH
that my website and it stays up all the time!!!
Dh has been belly belly good to me.
Good karma to you all!!!
Ah, status has been changed to DONE. Nice, an update that has gone by unnoticed.
I’m glad things are up and running again 
(Indeed, it’s been a rough week for the Dreamhost techs)
I am switching host now
siteground.com is the best
I wouldn’t visit any sites that people post in here. Chances are you’ll find tubgirl, pedophillia, bestiality or pop-up hell at the other end of the link. I’ll pass.
It’s nice to see the problem resolved. Now if only the websites would reappear…
Glad to see you tracked down the problem
So far my experience with Dreamhost (signed up a few months back): good support, cheap, good resources, marginal reliability. My uptime has been very poor; I’ve had more like 5 major outages (nearly full day down) in a couple months. Some have been runaway server processes and some have been infrastructure problems. Maybe I got unlucky and got a problem server.. Anyway just another data point from a new-ish user. Unfortunately, my idea was to have my sites (hosted at home) available while I was moving.. and these latest outages hit at the worst time.
One thing I’ve noticed. Response to my emails usually get my site back up quickly… but I have to send them. For example I was down today from 3AM to almost 9AM EST, but within 10 minutes of my email the site was back up. Seems like an automated monitoring/alert system could be more efficient than waiting for me to wake up and check in the morning.
So now I need to figure out how to configure my DNS (not dreamhost dns) to have a failover server or something.
Bill brings up a great idea, and hopefully someone from dreamhost will read this and implement a quick php script+ pearl etc else. hell, I’ll gladly help, I have the same issues.
for all and all though, i was pretty pissed yesterday, but ppl are right, I’m really glad you’re transparent instead of just sweeping issues under the carpet.
what would be REALLY!!!! cool, would be a small graphic of the 100, 1000, 10,000? servers, with green/yellow etc to show what’s working and not. maybe I shoudl build one as open source?
then people could get a real ide aof just how much equipemnt you guys have and it puts little problems into perspective once you can see the thousands of bits of hardware.
plus it’s just weird, we know you’re a big company, but there’s a disconnect, we don’t see anything but a website frontend. plz atleast post photos of the server room, for soem reason I love that sort of stuff…
I like Adam’s idea. If we can somehow know the status of all the servers and hardware that Dreamhost maintains, the solution would be simple. Now, how do we get the status?
For what it’s worth, I’ve been with Dreamhost for about 4 years now (I think) hosting one corporate web site and a few other ‘vanity domains’ and email service for friends, family etc.
A bunch of my ‘users’ are relative novices. “The internet is down.” “Can you get to google?” “yes” “then the internet isn’t down.” etc. Once I got their email up and running, I only hear from them when there’s a problem, so that’s a good sense of how often there’s a problem.
I don’t hear from them often. That’s really my point.
Yes, Dreamhost has had a few annoying outages in the last 6 months, I think there were two (including this week’s) that resulted in email or web problems for a significant portion of a business day. The DNS issue this week was pretty substantial. But when my corporate client asked about a credit or refund for the downtime, my response was “Look at how much you’re paying. If you want a credit, I’ll give you a dollar back, because that’s about double what you pay for a day’s hosting.” I was more tactful than that, but I didn’t have to be. I hadn’t quite realized what an amazing deal this place is. And I’m always trying to figure out ways to better use the allocation of disk and network that I’m paying for! I haven’t come up with anything significant yet… Anyone need a mirror site?
Anyway… The biggest lesson learned (to me) is that status.dreamhost.com MUST be a static site when there’s a network problem; having the “off-site status web page” offline because of the network problems I/we were coming to find out about, is just unreasonable… A simple Red/Yellow/Green status indicator and statically-published page would be great… Not that I don’t enjoy reading the bitching and moaning here on the blog.
Best,
-=Alan
PS: hey Jamie, long time no read. I need to go check out IHL again. I didn’t know we hosted at the same place…
an automated alert system is good, but most of those systems are just pinging the servers. Most of the time your sites down, the server is on and fine, but there might be an apache/php/mysql error preventing it from doing what you really want it to do. Plus you have to think of the sheer amount of servers that need to be monitored…
If you want dreamhost to see this, there is a little tab in the panel called suggestions. If you put it in there, people will see and vote on it, and if its popular enough dreamhost will look at it and possibly implement it. Suggesting it here, wont do any good what so ever…….
@the1337g33k: “an automated alert system is good, but most of those systems are just pinging the servers. Most of the time your sites down, the server is on and fine, but there might be an apache/php/mysql error preventing it from doing what you really want it to do. Plus you have to think of the sheer amount of servers that need to be monitored…”
Er, no. Look at “Big Brother” for example (open source monitoring package). It’s not just “pinging servers”, you can configure it to watch logs, do http requests, etc, etc. It’s what folks who actually want to monitor machines do. Trust me, DH’s network isn’t that big compared to a lot of other places (not talking about just hosting companies) .
What it boils down to is that DH is a bargain basement hosting service. They weren’t always, but now they are. Their business plan simply doesn’t include providing that level of monitoring to their customers - it requires resources and staffing which cost money.
The bigger problem is that there isn’t much out there anymore in the mid-range for web hosting. Over the last few years there’s been a huge “race to the bottom” that’s driven shared hosting to what you see here - cheap yet unreliable. It’s somewhat like the Walmart effect on commodity goods. Anyone who leaves DH for another $8 - $20 per month shared hosting company is eventually going to run into the same problems they have here.
If I could find reliable shared hosting for say … $40 a month, I’d jump on it. In fact, 4 years ago that’s what I signed up for here at dreamhost - “Code Monster” was $40/mo and for the first year … not a single problem. Then the prices started to fall, and the reliability went with it. The problem is … that product simply doesn’t exist anymore. You have this … and you have dedicated hosting, which even unmanaged is going to run you somewhere in the neighborhood of $150 min per month.
- Roach
Is this why the temp drives are full on mug? I guess I’ll submit a ticket.
So is there any sane explanation for why an emergency upgrade was necessary? It’s the “change management” equivalent of a :Hail Mary”. Trying to keep things stable usually means you don’t upgrade anything unless you’ve tested it for a long time.
Amazing how many of the sycophants that have been at DH “for years” managed to have their servers stay powered up during the power outages in 2005, 2006 & 2007 (which took the entire facility down) and also managed to be unaffected by the multiple DNS failures in the last couple of years.
Clean living I guess.
@number-six:
Yeah, I was wondering that myself, Not to mention the two major core router failures in … 2005 I think it was? Anyone who says they’ve been here for years and have never had an outage … is lying. Their (DH’s) entire network has failed multiple times per year for the last 3 years. Email has been out about as often.
- Roach
Or, some of us have been with Dreamhost so long now, that whatever minor outages we do experience — on the older, more stable machines and network — demonstrate to us why it is we love Dreamhost and remain with relatively little complaint.
Not to mention that thing about the cumulative enhancements we get — and that whole Files Forever thing. It’s kinda sweet what Dreamhost has done for us over the years.
I’d love to see Media Temple come close to matching what Dreamhost offers its customers [at the price!] — and without the same outage record. Please. Been there once — never again.
@Daniel:
I agree, in a way. I’ve been here a long time. Once DH started doing downhill and I realized you can’t actually use DH for anything professional, I moved my business to a dedicated server and just downgraded my DH account to the $10/month one and keep it for vanity domains, goofing around, and what not. And for $10 a month, you do get a lot of features - no one has ever denied that. Unfortunately, those features are useless when they aren’t available for days at a time.
I’ve sold my business, so DH is now my only host again … and I don’t mind so much now when they have these huge disasters because nothing I have on DH is critical. But not having my personal email for days at a time (which is not uncommon) is still annoying. Or not being able to send email to AOL or Earthlink because DH got themselves blacklisted … also annoying.
Anyone paying more than $10 for DH (any plan other than the base one) really is getting screwed. They (DH) sell a “strictly business” plan for $80 a month … this is not $80 a month reliability. And anyone running an actual business here is just … asking to lose money.
As I said - all the bargain basement hosting companies are going to have reliability problems - people here paying $10 are getting a good deal in terms of price:performance and moving somewhere else that charges $10/mo isn’t going solve their problems. I just wish there was a hosting company like the DH of old where you paid a bit more for a shared host (but less than a dedicated server) and got some reliability along with a decent feature set. I can’t find one of those any more and it’s a shame.
- Roach
Just an update to a post I left earlier. I’m using smokeping (using http requests and also curl to pull a page) to monitor my site, and I’ve had about 4 outages of .5 to 1.5 hours since yesterday (see gaps in graph after page load time spikes):
http://billcomisky.com/tmp/recent_downtime.jpg
I am in contact with support, it just hasn’t helped so far.
Thanks for the update guys and for only marking stuff resloved when it is resolved:)
Thanks for keeping us in the loop.
You know what? Instead of you DH’ers pretending to be customers and writing fake comments here about how you’ve “been with nightmare host for x years and this is your FIRST outage” (man, am I getting sick of reading overused line): maybe you could use that time to set up a PHONE SUPPORT system and you could actually allow your CUSTOMERS to call you.
We’re not giving you our money because we’re nice. We’re paying for a service. You don’t do a good job at providing it.
Is anyone else experiencing any issues with their hosting? My server appears to be down?
Whew. Moe’s gone… Time to turn everything back on. Thanks DH!
thank you…
Are you serious? So yesterday you guys screw us over but at least you did it today and told us. Dreamhost goes down more then an over aged porn star trying to stay in the business. I”m switching hosting ASAP, i recommend you guys do too.