Core Router Maintenance
As planned, we are beginning our emergency core router maintenance right now.
Our planned downtime window is 2 hours, during which all services including email, webmail, websites, ftp, ssh and the web panel will be offline. We will post updates here as things progress, so please keep an eye out and we’ll let you know when things are back online!
We are going to try to take care of as many things as we can during this maintenance window, so to minimize required outages in the future.
We greatly apologize for the inconvenience and thank all of you for your patience!
– UPDATE 12:18 AM Sat Oct 20
The router is back up, again, after the redundancy test we performed on the upgrade failed miserably. We are now running the new version of IOS, fortunately, so all is not lost. We are currently seeing if the network and the servers are back up and running.
– UPDATE 1:34 AM Sat Oct 20
We believe that all systems are Go! Please write in to support if you are still having any problems. We will continue to monitor this throughout the night. We apologize for the extended outage, we will be contacting Cisco regarding that bug in the redundant failover system.
.
October 20th, 2007 at 12:15 am
@Mari: Great idea about the suggestion. I’m off to do that right now. Thanks.
October 20th, 2007 at 12:15 am
Though I will say I agree on one point, they should have sent an email and said this was going to happen so we knew and the url to the status page here (I just remembered it by chance) cause if they actually DID send that email, I know *I* sure didn’t get it.
October 20th, 2007 at 12:16 am
This is why my article on RSS feeds being only for geeks was so popular. RSS Feeds suck at informing people. Try this. A RSS feed is like having to call Bob everyday and ask him, “Is there anything that you are doing that will make my life miserable?”. An email is like recieving a phone call from Bob, “Hi, I am just informing you that I am going to make your life a living hell at 10pm pacific standard time on friday…”. Not only that, the mail box acts like an answering machine so if no one is home the message stilll gets delivered.
Use email. RSS feeds SUCK!!!
October 20th, 2007 at 12:19 am
Seems everthing is back and up now!
October 20th, 2007 at 12:20 am
Isn’t two hour downtime over??????
October 20th, 2007 at 12:20 am
Status is still unresolved. Does this mean we’re still to expect down time?
This site is the place to check after all.. eh??
October 20th, 2007 at 12:20 am
OMG! It says 2 hours and 2 minutes and it’s not resolve! OMG! OH NOES!
October 20th, 2007 at 12:21 am
wohoo, my site is back up!
October 20th, 2007 at 12:22 am
“I think it is a waste of time posting here, So why post so much..”
..boredom.
October 20th, 2007 at 12:23 am
My sites & webmail are working
October 20th, 2007 at 12:23 am
aw, it’s down again =(
October 20th, 2007 at 12:25 am
Good luck implementing Web 2.0 philosophy then, Carl, if you plan to shrug off the power that RSS provides to the entire Web and data-sharing and taxonomy concepts… Just sayin’.
October 20th, 2007 at 12:25 am
Kevin, mine continue working….
October 20th, 2007 at 12:26 am
Bringing down servers so early in the evening is not good.
I just started to program an ecommerce site for someone who already had this hosting. Their control panel sucks … bad logic. They should have cpanel.
October 20th, 2007 at 12:29 am
I think this place needs a hug…..
:hug:
October 20th, 2007 at 12:30 am
Are their servers near a fault line? I see that they are in CA.
October 20th, 2007 at 12:34 am
Bob,
for me is quite too late in the morning, depending on where you are located.
I had a downtime from 7AM to 9AM that is quite good, a delay of 2 or 3 hours for people Europe means a downtime in the middle of the day.
In my opinion 22PM Pacific time is good.
By the way, I really appreciate DreamHost, I am a happy customer since 2005.
October 20th, 2007 at 12:35 am
tatertots: The router thanks you for the hug.
October 20th, 2007 at 12:36 am
My sites be FLYIN now… THANKS A SHITTONLOAD DREAMHOST!!!
October 20th, 2007 at 12:40 am
meh - RSS is the wave of the future for mass communications. The spammers will completely kill all serious usage of email for communications long before a way to kill all spammers is implemented.
If you are actually still relying on email for *anything* you deem to be important, you are only going to be more disappointed with each passing year. Whitelists, and Blacklists, and spamtards, OH MY!.
Get with the future and decide *what* you want to receive, and then use RSS to monitor that stuff, get it , and serve it up to you. Email as we now know it is going the same place gopher went.
“RSS rules, email drools”
October 20th, 2007 at 12:44 am
I was drinking a tall glass of water filled with ice.
As I tilted the glass to let the last few drops of water fall into my mouth, I wondered if the jumble of ice cubes at the bottom would fall from the bottom of the glass. When would they overcome their inertia? At what point would they decide to escape the confines of their glass prison and come tumbling down onto my face? These thoughts quickly became obsession. I did not blink, and I did not breathe. The room became silent, and within those few seconds, I grew to detest those miserable frozen chunks. Staring into the bottom of the glass, I felt a great sting in my heart, and a biting fear the likes of which I have never felt before. Everything I hated was in those ice cubes. I saw the face of every bully from middle school. Every embarrassment. Every shortcoming. Laughing children, mocking me with their condescending eyes. I could hear their callous taunts, no longer a dull pain in the depths of memory, but a roaring flame, salient and searing in my conciousness. My eyes became dry. I glared at the icey demons, and felt their algid breath against my skin. Tickling me. Daring me to make one move, daring me to challenge them just as the gunman had in that alleyway so many years ago.Once again I felt the horrid blend of a chill rain on my skin, the dank humidity of an unforgiving night, and the warm gush of her blood as it oozed into my hands.
I tightened my grip on the cold glass as I delved into her, immersing my hands into her very essence, pushing with all my might. Could I save her? An empty cry escaped my lips. A loud, hollow ringing in my ears answered, no. I wanted to salvage her. I wanted to take her into me. Her being. I wanted to become one with the only beauty I knew in this world that was now slowly slipping. AND FOR WHAT? …The rain did not respond….And as the blood washed from my face and lips, I looked skyward and spat. Was…the gun really mine? The ice cubes cackled at me. They laughed uproariously at the pitiable mess. My ears could do nothing against their cacophonous jeering. The frigid jackals were inside me. Once again, I bled.
As the trickle of water hinged on the lip of the glass, it balanced for an eternity and I wondered: could this have been prevented? At some point in my forgotten childhood, could I have stopped this?? Was this out of control? Why hadn’t you paid attention to me?? This may have never happened had you only paid attention to me! HAD YOU ONLY PAID ATTENTION TO M–
The ice cubes jolted forward.
October 20th, 2007 at 12:48 am
Yes, Dreamhost should send out email announcements of planned system-wide downtime. Yes, customers *could* religiously check the DreamhostStatus page or an RSS feed, but the onus shouldn’t be on the customer to actively seek out what DH should be handing to them. Almost all of us here are in agreement about that, except for a vocal few like Daniel. There is no point in arguing with him, he’s completely convinced that the overwhelming majority of us are idiots for thinking it would be nice to get a proactive notice of planned downtime. But it would be nice if Daniel simply realized that since most of us have different preferences than he does, it would be nice if DH would accommodate those preferences.
I’m guessing here, but my guess as to why DH doesn’t send out the notices is because without any notice, most customers will have no idea that there was ever any downtime. The next time they check their site (probably during the day) everything works, and when they check their email everything’s there, and there’s no indication that their sites were ever down. But if they send out the message, then they piss off a huge number of customers who don’t understand why occasional downtime is necessary.
October 20th, 2007 at 1:03 am
Somtimes I sit in the corner of my room rocking myself to sleep. Other times…times like these…..I make comments on this board to try and justify my existance. Lets me honest, the only reason I am defending Dream Host is because I am one of the idiots that works behind the scenes.
October 20th, 2007 at 1:16 am
I check this blog religiously. It seems to be the way they communicate with their customers in these situations….so.
And my PS is still sucking and up and down like a yo-yo. I’m paying way too much for that thing to be down.
October 20th, 2007 at 1:22 am
Daniel how are we supposed to have web 2.0 if all our websites are down? Perhaps we may have web 2.0 here, but in order for web 2.0 to occur across the rest of the web hosts must be as reliable as possible. Do you purposely design down time in order to facilitate increased contribution towards web 2.0 on this blog alone, while letting our blogs sit in limbo?
October 20th, 2007 at 1:23 am
http://tofu.he-hosting.de/uptime.html
Uptimes of three different Hosting companies I’m using, the only one with downtime is Dreamhost.
October 20th, 2007 at 1:37 am
my site has been backup for the last hour or so - long reign DH
http://www.starcrossedtattoo.com
October 20th, 2007 at 1:39 am
Y’know, it’s possible. On the other hand, it’s equally possible that dreamhost actually realized, hey wait a sec, sending email from mail servers that aren’t actually up, or on a network that’s not actually online, to users who’s emails are experiencing the problems they’re posting about, is just downright moronic in the extreme. Which… is actually why they switched to the status blog. Yeah, so calling up the status blog every time your website isn’t available is a pain. Is it more of a pain than getting the same notice via email, which is also not available? My guess is probably not. You can feel free to correct me. Just provide proof of concept if you do. Because I’ll be countering with it.
October 20th, 2007 at 1:43 am
I thought you said it sucked now? CPanel is *not* an improvement. Unless you’re sadistic.
October 20th, 2007 at 1:51 am
Something really strange going on!
Really!
How come all of DH was down and all my sites were up????
WHILE HOSTED ON DH???
I am the webmaster, I should know.
Maybe there is a secret server that only few customers get hosted on? Maybe I am one of them? What did I do to deserve it?
I am contacting support to see what’s going on.
October 20th, 2007 at 2:02 am
I am not sure but i think as soon as my website stabilize a little, i am planning to move it to some other host. may be MT.
so what DH is cheaper then others, its no excuse. You asked a price for your service and we paid for it and you know what, you are not good at it. MT charges little more then you (~ $70/year) but they are more reliable then you.
When i choose you, i was confused between DH and MT but now all doubts clear.
Thank you DH guys.
October 20th, 2007 at 2:03 am
My site is up. But has been dog slow for the last 2 hours…
October 20th, 2007 at 4:38 am
To all those that are leaving… Good! Don’t let the door hit you on the way out. I for one will benefit when you’re all gone. Less problems, more bandwidth, more CPU, etc. So, it’s been nice sharing a block with you, but I’d rather just buy your house.
Adios.
October 20th, 2007 at 4:49 am
MT?
October 20th, 2007 at 4:54 am
It is not wether or not a one technology is newer, better or web 2.0. It’s a matter of the logic involved in implementing a system to do something that it cannot handle appropriatley. If you sat down and made up a use case for informing customers and improving customer relations. You would see that any system built to satisfy the use case looks nothing like an RSS Feed.
This is what is so irritating about internet businesses. They are so hardcore about doing things because they are bleeding edge or convenient for them. They forget to look at what the customer wants or needs. If I where to abandon Dreamhost for another ISP it would not be because of the down time. It would be because Dreamhost chooses to ignore the most typical complaint. “Send me an email”. No amount of bandwidth, space or versitility can compensate for bad customer service which the lack of email notification sums up to o matter how you try and sugar coat it.
October 20th, 2007 at 5:07 am
Ben, MediaTemple
October 20th, 2007 at 5:20 am
I think the comments about MT being more reliable than DH are rather amusing. Go right ahead and switch! I’ll enjoy watching the massive downtime you get with basically no notification in any form from MT.
October 20th, 2007 at 6:13 am
Ignoring the whole confusing your web host with your ISP thing, you’re just a little off. If the outage being reported has to do with dreamhost’s network, or my mail cluster, or anything of that nature, I’d really rather it not come from an email sent from the network, or mail cluster, or what have you, that is currently having the outage–therefore more than likely preventing its being sent, and definitely preventing its being received. A status site such as this one, at least, means that when my email connection *does* time out, I’m not wasting my time sending support an email which they may not get due to problems with the mail servers, or waiting for a reply I may not get for reasons the same. Your ISP doesn’t send you an email when there’s a service outage in your area, I hope to God–if they did, they would officially be the year’s capital morons. Because, much like you won’t see the email until service is restored to DH, you won’t see the email from your ISP until either you get back online, or you get fed up and go elsewhere to check your ISP’s email account. Like I said in another comment, you show me proof of concept where an email notifying you of an outage would actually be useful. Because I guarantee you I’ve got several that will counter that claim almost entirely.
October 20th, 2007 at 6:22 am
All seems fine here on the East Coast, USA. Of course the downtime was sleeptime here, so no problem. And this morning, everything, even Gallery2, seems to be running much faster than usual, or at least recently. Perhaps just that not many are awake yet.
October 20th, 2007 at 7:12 am
Wow, some of you guys just love to find fault. Hardware fails. Would you rather they leave it half-working or fix it for good with a bit of downtime? I use DH and I vote the latter.
October 20th, 2007 at 7:16 am
I just don’t get it… what’s so special about this dreamhoststatus.com that you MUST open it everyday to check DH status? Why not email us? I already have way toooooo many sites to check, why should I add this site on my list?
Don’t you get it Daniel?
October 20th, 2007 at 7:36 am
None of your sites are loading guys…. mine either
October 20th, 2007 at 7:57 am
Other than the fact that it’s more often up when other DH services, like the much complained about email, aren’t? Nothing. However you’d think with the amount of people complaining about email reliability this would be a fair enough compromize.
And then you’d be wining because they emailed you about an issue that’s preventing you from getting your email. If email is supposedly so unreliable that you’re afraid to use it for a business client, why would you use it to get notifications that you’re having problems with it? Am I the only one here who realizes just how little sense that actually makes?
I already have way toooooo many sites to check, why should I add this site on my list?
Better question. Why should I spend 10 minutes trying to get into an unresponsive email system, just to have said email system come back up and have the notification that there was a problem sitting there *after* the fact? And that’s assuming it’s even delivered, if there’s any truth to certain users reporting some emails bouncing to already existing addresses. I get enough email as it is, I don’t need DH status reports getting lost in it where they’re lible to go missing.
I’m curious, by the way. Were any of you pro-email announcement people actually here when DH *had* email announcements?
October 20th, 2007 at 7:59 am
Personally, I hope those of you that are so obviously bent about this site DO leave…it’ll likely ease the burden put on this site (at the very least, the folks that have to read all your complaints) so the rest of us — who understand that nothing is perfect and technology especially comes with its flaws — can be relieved of your pettiness. By the way, all is well here in Florida, Dreamhost Team. Thanks for your diligence and please know that there are still many long-time customers (been with you since 2001) that have no intentions of leaving.
October 20th, 2007 at 8:08 am
Here’s a suggestion for DH that probably won’t take but 20 minutes for a good coder… Add an option to the panel that adds the user to a mailing list, then use RSSFwd (http://www.rssfwd.com/) to send an email into the list, then everyone who wants an email every time something happens can, and the rest of us won’t have to sift through the extra inbox clutter.
October 20th, 2007 at 8:22 am
My site is up and working just fine. Thanks DH!
October 20th, 2007 at 9:01 am
I was using ISP as a demonstration of why this works. Almost all ISP and web hosts I deal with have place in their forms for two email addresses. The one that you use to sign up with and a back up. When I signed up for dreamhost I did not use a DH email address as it would have been impossible. So not having their email servers up is only a half excuse.
Other services that I use also do not use the same equipment and network to send emails and run administration sites on. They either use another system entirely or have redundancy enough that one outage does not take down them offline entirely. One of the providers is just two guys in a basement in the middle of Stockholm. They have triple redundancy on their systems even though they probably only have 3000 accounts.
So lets take these two basement scientists and see what they do. First they send out an email to the addresses that each client used when signing up using the backup systems email servers. They then run backups and switch over to the backup networks. They then make the necessary changes and then switch back to the mainline. Simple.
If something on the mainline unexpectedly breaks (and it does happen) they send out emails from the backup. Again simple.
October 20th, 2007 at 9:16 am
Ok, for everyone who thinks that DH should send them an e-mail everytime something isn’t working right…
http://www.alxconn.com/dhs/
now every time DH posts to their blog, you will get an email. It’s magic!
(note: I don’t guarantee that this will work right the first couple of times around, since I can’t really test it. I think you’ll get 2 copies of the e-mail until I can. I also can’t guarantee that the 3rd party services I used will always be up either. If someone from DH wants to help me test this, I’m sure there are people that would appreciate it.)
October 20th, 2007 at 9:27 am
I am definitely appreciating the quicker wordpress load times I am getting now… good job Dreamhost!
October 20th, 2007 at 9:50 am
My site - http://www.iamboredr.com is still not up…. !!!!!!!