Core Router Maintenance
As planned, we are beginning our emergency core router maintenance right now.
Our planned downtime window is 2 hours, during which all services including email, webmail, websites, ftp, ssh and the web panel will be offline. We will post updates here as things progress, so please keep an eye out and we’ll let you know when things are back online!
We are going to try to take care of as many things as we can during this maintenance window, so to minimize required outages in the future.
We greatly apologize for the inconvenience and thank all of you for your patience!
– UPDATE 12:18 AM Sat Oct 20
The router is back up, again, after the redundancy test we performed on the upgrade failed miserably. We are now running the new version of IOS, fortunately, so all is not lost. We are currently seeing if the network and the servers are back up and running.
– UPDATE 1:34 AM Sat Oct 20
We believe that all systems are Go! Please write in to support if you are still having any problems. We will continue to monitor this throughout the night. We apologize for the extended outage, we will be contacting Cisco regarding that bug in the redundant failover system.
.
October 19th, 2007 at 10:26 pm
I guess that’s what explains my downed sites…
October 19th, 2007 at 10:29 pm
Dammit >_
October 19th, 2007 at 10:29 pm
Why minimize outages? We love’em! That’s why we stick to DH (or do we…)?
October 19th, 2007 at 10:29 pm
Hopefully this is the answer to this bad month for DH.
I’m tired of having downtime, Please Fix it for good.
October 19th, 2007 at 10:29 pm
I guess mine too.
LOL
October 19th, 2007 at 10:30 pm
Jesus, I didnt even read the post but is EVERYONE down?
How can a hosting company get away with this?
October 19th, 2007 at 10:31 pm
I guess Dreamhost is telling me to get a life and stop working by blocking my SVN access. Oh well, time for Halo I guess…
October 19th, 2007 at 10:31 pm
An email would have been nice. What, it’s up to us to check just in case a “planned maintenance window” is scheduled? Fuck that. Email your users and give them a heads up!
October 19th, 2007 at 10:31 pm
Would you rather they waited until Monday morning to do this?
I think we can wait for a couple hours on a weekend for them to resolve the core router maintenance that we were already informed would be needed.
October 19th, 2007 at 10:32 pm
After 16 hours (at least) of downtime last night, it is good to follow up with another 2-5 tonight.
That really sets the barb.
Good luck!
October 19th, 2007 at 10:32 pm
As much as I would like to complain, I realize that I am paying a very low price for a tremendous amount of disk space and bandwidth. My websites aren’t serious either, if they were, I think I would be with a more reliable host. I do
October 19th, 2007 at 10:33 pm
Perfect timing
The only time in which i actually needed my e-mail
Oh well, ill live
hehe it asks for my e-mail
when e-mail is down :O
October 19th, 2007 at 10:33 pm
question, how do I get to the web panel if it’s down too?
October 19th, 2007 at 10:33 pm
I really hope this clears up the major instability I’ve been seeing on my spanky new (and not so cheap) Dreamhost PS over the last two days. Counting this planned outage, I will have had uptime of only ~85% over the past two days. Definitely not what I signed up for.
October 19th, 2007 at 10:33 pm
id complain but they have been awesome about me paying my bill late, so meh
October 19th, 2007 at 10:34 pm
Big Fuss has a good point. I don’t mind the downtime necessarily but an e-mail would REALLY be nice.
October 19th, 2007 at 10:34 pm
ALL my site downed,and www.dreamhost.com downed also..
October 19th, 2007 at 10:34 pm
my interners! they are all down!!
October 19th, 2007 at 10:35 pm
I hope this network downtime will mean solid stable and fast service for some time afterwords. In case you all haven’t noticed, all of dreamhost is down. I like that Dreamhost puts status updates on it’s off site status page so folks can no what’s going on even when they have to take down their entire network, not even my dedicated server provider does that!
October 19th, 2007 at 10:35 pm
oops its gone!!! got up at 6 to see its not working…..bye going back to sleep…enjoy dreamhost…will get up after 1hr
October 19th, 2007 at 10:36 pm
Mine works fine.
October 19th, 2007 at 10:36 pm
Core Router Maintenance means, the only thing running is this blog, which isn’t hosted by DH for this very reason.
October 19th, 2007 at 10:37 pm
I think we can wait for a couple hours on a weekend for them to resolve the core router maintenance that we were already informed would be needed.
October 19th, 2007 at 10:37 pm
Ooohh…. my sites just came back up!
October 19th, 2007 at 10:38 pm
Well, at least we’re all in this together
October 19th, 2007 at 10:39 pm
…
If I lose money on this I’ll be totally ticked. I’m right in the middle of doing a PPP post.
I’m really surprised you guys think this is acceptable. I know you’re all humans (at least I think you are…), but you should at least hire ONE robot, so that it can correct these human errors that cause massive problems for your (HOW MANY?) customers.
Though, at least this blog is here so we can all complain.
October 19th, 2007 at 10:39 pm
I think that if we are able to get the Dreamhost Newsletter each month, could you not use the same mailing list for planned outages.
Down time happens, but if we could be made aware of it, then we can all inform our users…
Simple communication cost you nothing but 10 minutes of your time.
Agree with others above, we shouldnt have to be checking your site in case you post us something there… gesh, we’re busy enough coding our sites and supporting our own users…
Not a bitch, but a suggestion, EMAIL US!!
Cheers
October 19th, 2007 at 10:40 pm
Oops… my last post was cut short.
Already informed? How… did you get an email? Posting major stuff like this on a blog that your customer has to happen to check in order to get the message is lazy, careless and shoddy practice, imo.. This sort of thing needs to be delivered to my inbox as soon as they plan / schedule it.
October 19th, 2007 at 10:41 pm
QUOTE:
“# Big Fuss Says:
October 19th, 2007 at 10:37 pm
I think we can wait for a couple hours on a weekend for them to resolve the core router maintenance that we were already informed would be needed.”
How were you informed? is there a mailing list I should be aware of?
October 19th, 2007 at 10:42 pm
I am wondering why my sites are back up already. A 2 hour planned down time is a 2 hour planned down time. I’d rather the time be taken to get things right than to assume everything is all set if the upgrade went *that* fast (which I doubt it did…)
October 19th, 2007 at 10:45 pm
@Themicles
That is curious. Extremely curious. Was that small outage just a taste of what is really to come?
October 19th, 2007 at 10:45 pm
I don’t want to add to the heap of complaints, but this really is rather poor. Reading through the past causes of downtime, most seem avoidable:
- NetApp filers can be clustered to allow for zero-downtime hardware replacement.
- Servers shouldn’t be booting off individual local hard disks — use thin flash storage or PXE to bootstrap an iSCSI or NFS root…or at least have multiple mirrored OS disks.
- Networks can be redundant and multipath.
- Virtualization can allow you to move “servers” between physical nodes dynamically without downtime. You already have filers — leverage them!
You have the infrastructure to make vast leaps in reliability if you just do a bit of refactoring. I help keep a similar setup running at work with far less downtime.
October 19th, 2007 at 10:49 pm
In the interest of keeping myself from being blindsided again by DreamHost shutting down over 100,000 domains without an e-mail, I subscribed to the RSS feed in the panel. The trouble is, it DOESN’T show only issues that affect me, and it says EVERYTHING is resolved, even when the core router maintenance wasn’t as I write this.
Do status feeds work for anybody else?
October 19th, 2007 at 10:52 pm
BigFuss,
Ummm. Not an email. If you can read, you’d probably have seen this post here on the main page — which is still marked High red.:
http://www.dreamhoststatus.com/2007/10/17/core-router-upgrades/
Wow, some people here amaze me!
October 19th, 2007 at 10:55 pm
Not sure if this is a bug, but I’m seeing the last person’s “will not be published” email address in the Leave a Reply section.
That’s not normal, right?
October 19th, 2007 at 10:56 pm
@Jesse
No, status feeds don’t work for me either. It shows everything rather than what’s relevant to me, same as your problem.
October 19th, 2007 at 10:56 pm
Oop.. and down she goes again.
Have to agree with Tyler, particularly as an extra-paying PS customer…
October 19th, 2007 at 10:59 pm
Odd that my stuff didn’t go down until about 10 minutes ago o_o I hope all the work I’ve been doing isn’t gone =/
Sucks that this has to happen on one of the only times I’ll have to work on my site this weekend…damn school work.
October 19th, 2007 at 10:59 pm
Again.
We were informed of this!
http://www.dreamhoststatus.com/2007/10/17/core-router-upgrades/
It was PLANNED!
October 19th, 2007 at 10:59 pm
@Daniel (10:52pm)
What amazes me is the free and easy method of contacting everyone via email for a pre-known total outage does not take place..
And you amaze me if you believe that all dreamhosters should be checking dreamhost for this sort of stuff… if a newsletter which in general is 50% crap content, can be churned out once a month, then surely a few lines of text for a pre-known/planned outage is not too far out of the question…
October 19th, 2007 at 11:00 pm
What amazes me is you have time to check the dreamhost status page on the off chance there’s going to be something major going on that affects you. What’s more amazing is that this seems perfectly normal to you to the point where you want to be a condescending prick when others have that sort of “customer service” pegged as worthless at best and negligent at worst. Seriously, how can you possibly defend the lack of email notification on this with “Well, they blogged it! Didn’t you see?”
Give me some of your free time, shithead. Then i’ll camp out “on the main page” to monitor when stuff’s going down with the host.
October 19th, 2007 at 11:01 pm
COOL!!
This downtime gives me a perfect, guilt-free reason to go play World of Warcraft for a couple of hours.
Take your time guys!!!
October 19th, 2007 at 11:01 pm
If you came here to check out what happened, I would have to assume you hit the main page — and would have seen the RED marked post:
http://www.dreamhoststatus.com/2007/10/17/core-router-upgrades/
Shithead? Nice one, for someone who apparently cannot read.
October 19th, 2007 at 11:03 pm
What would be useful is notice put in the MOTD on SSH logins of this. I just started to upgrade wordpress and pop, dreamhost disappears halfway through the upgrade. Fun.
October 19th, 2007 at 11:04 pm
This really pisses me off. Why not do this more towards 1am PST or so? That way it minimizes effects to both coasts. As it is, I’ve been completely down since about 10pm. LAME! That’s prime-time for me!
Also, if they’re gonna do this, and it’s planned, then why not email everyone? That would have been appreciated.
October 19th, 2007 at 11:05 pm
Wow, great job guys. Thanks for the heads up.
I still remember when dreamhost was a quality host, but it’s getting harder by the month.
October 19th, 2007 at 11:05 pm
Daniel,
The idea behind a planned outage is that it takes place at a predetermined time. It is a “proactive” step that customers can be warned about ahead of time.
What you are describing is a “reactive” step — using the blog to find out WHY things are down AFTER they are down.
The complaint is that despite being planned, customers we’re not PROACTIVELY warned beforehand. This eliminates the virtue of the outage being planned, which is pretty silly when you think about it.
It’s like the start of the Hitchhiker’s Guide — the fact that there was a warning floating off where nobody was looking at it means there was basically no warning at all.
October 19th, 2007 at 11:06 pm
Is the site still down? I’m having issues accessing. Thanks!
October 19th, 2007 at 11:07 pm
My DH account is due in January. I’ll be shopping for a new host.
October 19th, 2007 at 11:07 pm
We havent had a good uptime since we started here, do most my work in the evenings and it seems like I’m spending more time waiting to the ftp or the site to return service than I do coding… Hope this will fix some of the issues on the model-t.