Core router upgrades
Posted (October 17th, 2007 at 2:41 pm PST) by KellyThis Friday, October 19th, 2007 at 10:00PM Pacific time we will be performing emergency maintenance on our core routers. We are going to be upgrading the operating system on one of our core routers in our main facility, and one primary distribution router at our LAX facility. The total downtime shouldn’t be much longer than it takes these to reboot into the newer operating system, and reestablish the links to the outside world. The maintenance window is two hours, though we hope everything from a network standpoint will be completed in 30 minutes.
These upgrades will address many of the issues we have been having over the past months. Cisco has recommended this version of IOS to potentially solve a network flooding issue we have seen intermittently, and if it does not, will allow us to escalate the case even further. The upgrade will also give us support for new hardware we have purchased for the routers themselves, increased diagnostic gathering capability, and bring online our redundant supervisor line card.
We apologize for the short notice and overall inconvenience of this upgrade, however this will hopefully fix various outstanding bugs that crop up during the day. We are attempting to do as much as possible with this reboot to minimize further downtime.
All systems will be affected by this upgrade.
86 Responses to “Core router upgrades”
Is the date on that correct? Today is already October 17 by my calendar…
when?
I don’t understand when?
well, it really doesn’t matter for me since my site hasn’t functioned properly for the last 2 going on 3 days i don’t really give a rats rear what kind of upgrades they are doing.
wait. i’m supposed to care that my site will be affected for two hours on friday night. by then, my site won’t have worked for over 72 hours and i’ll be used to it!!!!!!
and tech support told me my site looked fine. tell that to my clients.
do i sound upset????
Bonnie,
I don’t know if you will take this as “good news” or “bad news”, but your site also looks fine to me (though the flash is a little slow loading - meh, after all, it *is* flash). How is is not “functioning properly”?
Maybe your client/you are experiencing some connectivity problems some of the rest of us aren’t? Seriously, I’d keep investigating before assuming that the site isn’t being properly served, as I can see it fine.
If you wish to know what time will it be at YOUR part of the world, you can use my Dreamhost Downtime Calculator
http://info.org.il/english/dreamhost_downtime_calculator.php
This is where it will take you to for this downtime notice
geyser’s slow… but my friends website on the other servers are ok
I empathize with DH. From an IT and administrator point of view, they’re doing a good job. If you guys are looking for reliability, you’ll need to pay more. I can’t imagine you’re paying all that much money for the bottemless pit of bandwidth/storage on web hosting. Stop complaining. They already know what problems they’re having and trust me, IT’s don’t go any faster just because you cry about your problems. They get fixed when they do.
After 2,5 years I closed yesterday my account with dreamhost . first year was great and I was proud to be with dreamhost. to the end of 2006 , the problems began, but I remained loyal. 2007 was ridiculous. I was using DH primarily for mail hosting. I cannot explain to my customers any more why my mail is unreacheable any more (DNS failures,….).
DH has added itself to the numerous black sheep in the ISP Business. But that is not my problem anymore.
And to everyone here, the money I paid for all domains and all plans permits me one last sarcastic remark:
Have fun with Dreamhost and the “not-reading-this-blog-because-it-is-useless” Dreamhost staff!
Come on, We were coving a major event through photography albums, in India… and now i see even after a full 12 hours later still the issue is very much alive…
I mean I am running my site only for few of these events and when I need the site to run, I see my host asks for a 30 mins down time and presents me a full 12 hours and still running down time..
Thank you DreamHost for such a unreliable service !!! Really !!!
Regards
Saurav Basu…
rlparker - my site (until this past week) zipped right open even through the flash. now it takes 30 seconds for a page to load. so YES, there is a problem and NO it is not my connectivity unless all of north carolina is having a connection issue which i don’t think is the case. and dreamhost SAID it there was a problem with my server all week and that it would be fixed soon. that never happened. only the good news is that yesterday they finally said the same thing that you did - my site was working just fine. ONLY IT WASN’T AND IT ISN’T. so that’s when i knew it was time to leave dreamhost. i was stupid to think they would help me.
I’m siding with those who have issues with Dreamhost. Seriously guys, your webhosting service quality is not as good as it use to be.
I’ve an account with Dreamhost since 2005, and the first two years with dreamhost was awesome, I even recommend some of my friends to host with DH.
In 2007 dreamhost service quality are heading downwards, they started to hide the fact that they squeez thousand of users inside a single machine. Then they started to silently limit php.cgi process. ie. how many of you encounter “500 Internal Server Error” in your own website? my website all have
I’m siding with those who have issues with Dreamhost. Seriously guys, your webhosting service quality is not as good as it use to be.
I’ve an account with Dreamhost since 2005, and the first two years with dreamhost was awesome, I even recommend some of my friends to host with DH.
In 2007 dreamhost service quality are heading downwards, they started to hide the fact that they squeez thousand of users inside a single machine. Then they started to silently limit php.cgi process. ie. how many of you encounter “500 Internal Server Error” in your own website? my website all have less than 3k/day visit for god sake. I dont expect it to withstand a digg-effect, but php.cgi killed when I’m only running ordinary blog is purely pathethic.
I know my money isn’t worth that much, but at least stop being flamboyant, dont try and steal other other web hosting customers (like LunarPage) if you can’t even handle your own users.
Seems Bonnie is right.. Very slow and I see why:
tracert bonberphotographics.com
Tracing route to bonberphotographics.com [208.97.185.46]
over a maximum of 30 hops:
1 1 ms 1 ms 1 ms 192.168.100.254
2 30 ms 30 ms 29 ms [205.200.28.27]
3 30 ms 30 ms 29 ms [142.161.133.33]
4 29 ms 30 ms 30 ms [142.161.132.2]
5 30 ms 29 ms 29 ms [216.13.7.85]
6 63 ms 62 ms 62 ms [199.212.161.142]
7 63 ms 63 ms 62 ms cr1-six-peer.sea002.internap.net [198.32.180.41]
8 91 ms 90 ms 91 ms cr2.lax009.inappnet.cr1.sea002.internap.net [66.79.146.226]
9 91 ms 93 ms 90 ms cr1-cr2.lax009.internap.net [66.79.146.205]
10 273 ms 207 ms 204 ms core1.lax.inappnet-12.cr1.lax009.internap.net [66.79.149.130]
11 91 ms 91 ms 90 ms border1.po2-bbnet2.ext1a.lax.pnap.net [216.52.255.95]
12 103 ms 102 ms 102 ms newdream-1.border1.ext1a.lax.pnap.net [216.52.220.78]
13 390 ms * 400 ms apache2-argon.geyser.dreamhost.com [208.97.185.46]
Trace complete.
@saurav basu: the updates will be done on “This Friday, October 19th, 2007″ so no wonder it’s not resolved yet
Greg - I agree, Bonnie is right. But - did you read the announcement that these comments are attached to? DH knows they have a problem, and they are announcing changes to a core router to help fix it.
Wow, lovely. More downtime, just great. My domains finally came online again after a load issue on the server cause about 72 hours of downtime that DH were VERY slow to fix, and now my domains will be be going down again? This is bullshit, I’ve been looking at a different hosting company as of late, the problem is I have about 50 dollars tied up in my DH account (Credit, due to one of the DH staff’s errors), that they can’t refund to me in any way. Anyone have any suggestions as to a good RELIABLE host? I’ve been looking at hostmonster.com lately, they don’t sound too bad.
Meh still a bit slow IMO, at least slower than it was, I was getting about 40ms response times before all this crap happened…
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=29 ttl=56 time=80.8 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=30 ttl=56 time=80.1 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=31 ttl=56 time=79.7 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=32 ttl=56 time=78.0 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=33 ttl=56 time=80.1 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=34 ttl=56 time=79.5 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=35 ttl=56 time=79.7 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=36 ttl=56 time=78.0 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=37 ttl=56 time=79.4 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=38 ttl=56 time=77.6 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=39 ttl=56 time=79.8 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=40 ttl=56 time=77.4 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=41 ttl=56 time=79.6 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=42 ttl=56 time=77.8 ms
64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=43 ttl=56 time=80.0 ms
^X64 bytes from apache2-hok.geyser.dreamhost.com (208.113.176.42): icmp_seq=44 ttl=56 time=80.4 ms
n0yd, i’m looking into hosting with hostmonster.com too, though i’m beginning to think to use VPS after this. can anybody suggest alternatives?
I will also be moving to a new host due to this latest Uptime problem.
my site is completely down !
completelydown.com
Same here, all sites are down.
Hmm.. I’m not getting any email right now…
Everything down here too.
Hate I You. It’s a puzzle. Work it out.
The end of the month (and our transition off your hosting) can’t come fast enough…
yup. same here.. all my sites on holt are down.
*sigh*
another day with dreamhost.
Ironic that Bonnie’s site appears to be the only one on dreamhost’s entire network that’s up right now.
And redondo fell as well. Are there ANY survivors? They really get to the homeland now. 
(but you know what I mean)
I wasn’t getting anything (mail or web servers) except the login for webmail (that couldn’t pull the custom graphic hosted on the web server)… but now it appears it’s back up.
No slowness on my sites recently.
That’s because they can only host one at a time, I suppose.
UGH!!! THIS FUCKING SUCKS!!!!!!!!!!!!
Yup, my sites are also down. Jabber and mail too.
Hope it will be done soon.
everything is down. and back again. hmmm….
why can’t they just let us know what is going on. I have been using dreamhost for many years now, and I must say, aside from service going way downhill in the last 1 year, their customer service has gone to complete shit. I pay for callbacks, and don’t get them. Status tells me nothing.
We are all just wondering, is it possible to inform your users when something is going on? That’s all we want. Even just a simple note, like “we don’t know what the hell is wrong, but we’re trying!”
grrrr….
My sites on Gobstopper are down as well. I’m losing ZERO DOLLARZ EVERY MINUTE BY BEING DOWN!! heh heh
couldnt have picked a worse time ….
THATS IT! IM CHANGING HOST! DREAMHOST SUCKS§!
This is getting to be ridiculous….
Whoa, dude. Now everything screeeeeams! THANK YOU DREAMHOST!
My site is down just when i need it the most.. again.
anyone knows a cheap and *reliable* shared hosting?
i seek uptime over price here..
Back up! A WHOLE 5 MINUTES!
WAAAAAAAAAAAAAAAAAGGGGGGH EVERYTHING’S DOWN!!!!!!!!!!!!!!!!!!!!!!!!!! NOT EVEN THE COMPANY BLOG IS UP!!!!!!!!!!!!!!!!!!!!
Ok, calm down.
I’m not in PDT. Is this the planned downtime or not?
Again up. Quite fast.
Thanks god, it was fixed quickly.
All the sites on POUND are down.
All is brokendown!!!!!!!!!!!!!!!!!!!!!!!
HEEEEEEELPPPPPPPP
Everything’s back up for me.
To all the whiners out there (and you are plentiful): hosts go down. The fact that dreamhost went down and came back up in 10 minutes with a noticeable speed increase is pretty awesome, if you ask me. Remember… 99% uptime still adds up to hours of downtime a year.
comes and goes … now nothing again
Just started to come back to life for me: how long was it all b0rked for, do we think?
yahu bıktım bu problemlerden, problemsiz bir gün olmayacak mı ? 
Damn, that’s the second time this week that all my sites on all accounts are dead…
I hope it’s done soon.
Meh.. email is working again now.. must have been a hiccup in the matrix
Rahrah, they’re obviously flipping a couple switches. Be patient!!
Geez man — 5 minutes or two while they’re flipping switches to bring new equipment that shows a dramatic efficiency increase? What the heck are people complaining about?!!?
//oski Says:
October 18th, 2007 at 1:55 pm
yahu bıktım bu problemlerden, problemsiz bir gün olmayacak mı ?
Como?
Mike, it was ‘borked’ for about 5 minutes. Woooooo, the end is near!
At least DH is finally calling the situation what what it is… an “emergency”. People need to stop defending DH and help them understand how serious this is. I truly hope DH solve these problems. But even if the technical issues are solved, the damage to Dreamhost’s reputation will take years to repair.
Eric, give it a rest — or go try out the Grid servers that don’t scale over at MT. lol
Daniel… even Dreamhost is calling it an emergency. Pull your head out of DH’s ass.
Well the e-mail and website are working slow and sporadicly here again. Kind of hard to make trouble ticket when it goes up and down. Every time it seems like things are ok, or work for several days in a row, then something pops up. I keep hoping things will become stable around here. Our subscription is almost up and we have been here for 2 years. It’s just a pain in the ass to have to switch hosts. I’m just glad it wasn’t me who signed us up here
My head is with the hosting provider that I’ve had the fewest problems with, Eric — and continually makes improvements. Where’s yours?
Dreamhost —
I might be crazy here, but why would you not get a new router, with the new software on it, and just switch…. to the already booted router by moving what should hopefully be a handful of cables.
Or perhaps the better question would be, why isn’t this a redundant system?
Daniel… since 1999 I have used Dreamhost with very few bandwidth or uptime issues. This year has been very different.
I will be migrating to a virtual dedicated space at Media Temple. I will not consider an upgrade with DH because it seems insane to continue with a host in light of the fact that the quality of the service has diminished considerably. Also, with regard to competitors, even GoDaddy is kicking DH in the ass right now. I know because my clients come to me with Godaddy accounts and I am surprised how much more reliable they are than DH.
My, how times have changed.
Well it’s all right ma, I can make it!
This is insane… 19th October? No services working right now!!
What the hell is going on??
Email down again =(
LOUD NOISES!
Would it be more responsible to defer the planned maintenance until the current outage were fully understood? Adding additional variable doesn’t usually make a better result unless the baseline is well known. It doesn’t appear the current situation is clearly understood and easily corrected.
Dreamhost needs to be more responsible to it’s customers.
Everytime the whole system goes down all they do is say “wee are sorry”.
That is not good enough. They should have a redundant system to keep client’s websites up and running, if the whole thing goes down. Or maybe the should give us free hosing for a week.
Man you think that Popular Hosting Compagny could just buy a new one
and swap the damm main server, but I guess it to the point where getting that cheap of service, If i would know I wouldn’t of got the PS, Meh This seem like the never ending problem…
Come on This down time is ridiculous.
In the mean time I’m shopping around for more reliable Hosting, Something with a Back Bone.
That won’t Crush my Site Dreams…
The following service might be useful for everyone here. Or, others like it. I just happened to google this and found it and it is working for me now. http://uptime.openacs.org/uptime
An Uptime service will send you email notifications when your site goes down. This way you can track your downtime at least.
Site are back up and running again… thanks guys
Is it just me, or are all of these rants and praises of “My sites down!!!!!” and “THANK YOU DREAMHOST!!!!” a little premature since the maintenance hasn’t even happened yet?
Well, redondo is up again, working as it supposed to be. A little downtime yesterday, but no probs this day. (would be bad as it was, as we really need the svn mid-weeks. We’re really lost without svn. I’m really happy to have svn at dreamhost. At top as one-click. Great.)
@Belthesar
I think that the “My sites down!!!” rants are premature but the Thank You’s, Most of us have had DreamHost go way above and beyond to assist us with issues that were not their fault/problem…
EX: I had issues with my primary host blocking mysql traffic, I started with Dreamhost thinking maybe i setup my allowed hosts incorrectly… They helped me to the point were i was able to nail down the issue on my primary box & fix it.
Every next day hosting service is worser then last day. Simply, larger snack want eat then mouth can open dear Dreamhost….
We all want to see what will be happen after 19.10.2007. and after hardware upgrade (again +2 weeks)
Summary:
4.11.2007. all will be again 100% ok ?
We will see….i hope you are right….Last days/months say something else.
ALRIGHT!!!!! Tonight is the BIG night! Perhaps I will pop up some corn and get ready for the router upgrade party that’s SURE to be here!
3 minutes… let’s make with the downtime!!!
I’m scurred
Well, my sites down.. lets see how long this goes.
# Belthesar Says:
October 19th, 2007 at 9:18 am
Is it just me, or are all of these rants and praises of “My sites down!!!!!” and “THANK YOU DREAMHOST!!!!” a little premature since the maintenance hasn’t even happened yet?
I think Dreamhost should have less websites on each server and maybe stop increasing disk/bandwidth weekly - but do it yearly. And have more servers. And use professional names instead i.e. DHSS1, DHSS2… Etc. (DreamHostSharedServer#) Lol. It would be better - Dreamhost is the Worlds No1 Host. 
Thanks for the updates Dreamhost team.
I have several different uptime services, and they all report the same things. Up/Down/Up/Down. Support says ‘everything should be fine now’ yet later on I get another email from the uptime service that my sites are down… These are simple sites, nothing fancy, basically 1-click-installs, everything SHOULD work fine!
Events
(most recent first)
* 2007-10-28 09:31:40 : back_up
* 2007-10-28 09:08:22 : down
* 2007-10-27 23:11:03 : back_up
* 2007-10-27 22:40:12 : down
* 2007-10-27 22:24:55 : back_up
* 2007-10-27 20:47:52 : down
* 2007-10-27 12:11:52 : back_up
* 2007-10-27 08:13:29 : down
* 2007-10-27 07:42:03 : back_up
* 2007-10-27 06:18:34 : down
* 2007-10-27 05:29:23 : back_up
* 2007-10-27 03:22:56 : down
* 2007-10-26 01:03:41 : back_up
* 2007-10-26 00:36:13 : down
* 2007-10-24 00:16:22 : back_up
* 2007-10-23 23:51:49 : down
* 2007-10-23 22:41:22 : back_up
* 2007-10-23 22:06:52 : down
* 2007-10-23 03:30:15 : back_up
* 2007-10-23 03:09:25 : down
* 2007-10-22 00:31:42 : back_up
* 2007-10-22 00:07:43 : down
* 2007-10-21 15:18:05 : back_up
* 2007-10-21 14:59:22 : down
* 2007-10-20 13:44:28 : back_up
* 2007-10-20 12:32:40 : down
* 2007-10-20 03:19:14 : back_up
* 2007-10-20 02:27:48 : down
* 2007-10-19 19:00:36 : back_up
* 2007-10-19 18:37:58 : down
* 2007-10-19 18:16:35 : back_up
* 2007-10-19 17:51:36 : down
* 2007-10-19 14:39:29 : back_up
* 2007-10-19 13:48:57 : down
Cool, the post.
Thanks for the information.
good
goo
lucky
there is no friday october 15th