Dreamhost firewall error (Resolving)
Due to a typing error on our primary router while trying to block a denial of service attack, DreamHost is currently offline. This includes all email, web hosting, etc. A technician is about 15 minutes out from our datacenter to undo this mistake.
I apologize for this mistake. I was intending to be editing our non-live access-list, and edited the live one by mistake.
Update 9:54 PM Pacific: We have unblocked the firewall and our routing is coming back online. Sites should start working again as soon as our router finishes loading its tables. I use “resolving” because it is still processing the routes.
.
March 7th, 2008 at 8:54 pm
this is extremely inconvenient (to say the least). i would like to hear how Dreamhost will prevent this kind of mistake from happening in the future.
March 7th, 2008 at 8:54 pm
Well at least dreamhoststatus.com is still up
March 7th, 2008 at 8:54 pm
This is why I love DH.
Seriously, if this doesn’t make you laugh, you’re a sad, sad person.
March 7th, 2008 at 8:54 pm
GB2NT3.51SRVR
March 7th, 2008 at 8:54 pm
You people need to lay the fuck off. Let’s see YOU do any better. I don’t see you hosting thousands of websites, now do I? Downtime is part of this game. Get used to it or get the fuck off the internet. Crying ass bitches.
March 7th, 2008 at 8:54 pm
Half of the internets is down, get this stuff back up!
March 7th, 2008 at 8:54 pm
WTH!!!!!!! Sirs, The Simpsuns blog is not good timing for Simpsuns blogging! I demand formal repayment regarding sirs for hosting
March 7th, 2008 at 8:54 pm
Wow, posted 5 minutes ago and we already have 40+ comments. People are watching this problem like hawks!
My http://ww2db.com is still down, and I’m on a roll with traffic today before this point, too! C’mon Dreamhost, you can do it! Bring our sites back up!
March 7th, 2008 at 8:54 pm
Every time there is downtime due to a Dreamhost staff mistake, they should compensate us somehow. That would give them some incentive to be EXTRA careful.
March 7th, 2008 at 8:54 pm
On the plus side the technician successfully blocked the DoS attack
March 7th, 2008 at 8:54 pm
Very disappointed again. Service has been pretty iffy for the past 6 hours or so. Been checking dreamhoststatus.com and this is the first I’ve seen of any problems. Not the first time similar events have happened. Hosting is really great overall, but when outages occur, your response times really are not so good. I really rely on mail, and mail is pretty much always slow. =(
March 7th, 2008 at 8:55 pm
=)
March 7th, 2008 at 8:55 pm
You need better procedures, show some professionalism.
March 7th, 2008 at 8:55 pm
Well that explains it then
March 7th, 2008 at 8:55 pm
To the gulag with the butterfingered admin!
March 7th, 2008 at 8:55 pm
well…atleast it happened on a friday night when traffic is usually fairly low
March 7th, 2008 at 8:56 pm
Anyone recommend a good alternative to Dreamhost?
March 7th, 2008 at 8:56 pm
One word: OOB.
March 7th, 2008 at 8:56 pm
So, what’s everyone’s favorite band?
March 7th, 2008 at 8:56 pm
this is pathetic, seriously….
March 7th, 2008 at 8:56 pm
Wow, another outage.
At least my lunarpages account is still up:
http://www.lunarpages.com/id/drunkenscholarcom
March 7th, 2008 at 8:56 pm
I can has refund on the internetz?
March 7th, 2008 at 8:56 pm
haha damn fat fingers
keep it up dreamhost, i believe in you
March 7th, 2008 at 8:57 pm
Shit happens to all of us lol
March 7th, 2008 at 8:57 pm
denial of serice? hmmmm… just one more step before skynet takes over
March 7th, 2008 at 8:57 pm
@In Pain: Compensation? Serious? You want compensation for your site going down approximately 30 minutes?? That’s just greedy….
March 7th, 2008 at 8:57 pm
heh. hehe. hehehehe. that’s actually pretty funny cuz i’ve done silly things like that before…….. :O
go lemurs!
http://news.nationalgeographic.com/news/bigphotos/images/060626-lemurs-africa_big.jpg
March 7th, 2008 at 8:57 pm
this is really pathetic. if you’re going to run a hosting company then you should get your act together.
March 7th, 2008 at 8:57 pm
@Robbie: Yes, I’m serious. Other hosting companies compensate clients for downtime. Why not dreamhost?
March 7th, 2008 at 8:58 pm
Really … I mean, to be forced to get someone to drive to the building to fix this type of issue is really really coming down to bad design.
A simple administrative vpn hosted by different IP space should be a must for you guys at this point, with remote rebooting capabilities and everything else.
15 minutes to drive to the site to fix an acl is a bit tough to swallow … I mean, you were configuring it remotely?? But cant fix it remotely??
BAD BAD BAD planning IMO
March 7th, 2008 at 8:58 pm
The Simpsuns blog!!!!!!! MY LIFEEEE (is over)!!!!!!!!!!!!!!!
March 7th, 2008 at 8:58 pm
“This should be impossible, yet here it is.”
Tell you what, make something like this ‘impossible’ and I’ll be the first one to buy your product.
Honestly, every network related outage I’ve seen over the last couple years has been perfectly normal. Where DH falls down is on the redundancy side of things. But let’s be honest, I’m not paying for redundancy. I’m paying for inexpensive web hosting with a company that has a good geeky culture. If they put the $2 million in a proper highly redundant architecture, trust me, your costs will go up accordingly.
March 7th, 2008 at 8:58 pm
haha, nice one
I cant say I have shut down anything the size of Dreamhost, but I have sure had my share as a net tech
Good Luck and thank you for the info!
March 7th, 2008 at 8:58 pm
I wondered if I was imagining things…. ha
http://www.NicheStoreScript.com
March 7th, 2008 at 8:58 pm
Time to go out, people!!!
March 7th, 2008 at 8:58 pm
> Tim
Yes Typos can happen to anyone. But they should’nt be the cause of a general crash. There should be some security system to avoid this.
This is not funny.
March 7th, 2008 at 8:58 pm
Hey, we can turn this into a little chat room. I wish I would have bought more tequila
March 7th, 2008 at 8:59 pm
We just need to host our sites where they host dreamhoststatus.com, it never seems to go down
March 7th, 2008 at 8:59 pm
I’m glad that at least the dreamhost.com site is down too. When my site goes down their’s is always up, so this will give them a little extra motivation to fix the problem sooner.
March 7th, 2008 at 8:59 pm
“poop pie Says:
March 7th, 2008 at 8:52 pm
oh shut up its not like anyone looks at your stupid little sites anyway”
Hahah, best comment.
March 7th, 2008 at 8:59 pm
The good part:
- you admitted it and posted it quickly
The bad parts:
- the technician should have been on his way in when the attack happened, so he would be there immediately.
- there should be enough redundancy so that one mistake like this doesn’t knock 500,000 web sites out of business
- what changes will you make to your policies to prevent this from happening tin the future? If this happened during the day, it could be catastrophic to some businesses
- 500,000 web sites are down. This should be used to change the way you do things.
March 7th, 2008 at 8:59 pm
Bluehost. http://www.bluehost.com/
March 7th, 2008 at 8:59 pm
Okay, it’s near midnight on the East coast. Stop whining and Read a book. Watch TV. Or, go to bed early!
March 7th, 2008 at 8:59 pm
Well mistakes happens and most often we learn from mistakes. I sincerly hope that this will not have repercussions on who mafe the error.
March 7th, 2008 at 8:59 pm
Heh, it’s actually pretty easy to accedentally enter the wrong entry into a firewall access list and block EVERYONE (which includes yourself, so you can’t terminal back in to fix the problem.) Cut her a little slack… it’ll be fixed in a few minutes.. all the tech has to do is arrive and edit one little access list and everything will be accessable again.
Simmer folks.
Matt
March 7th, 2008 at 9:00 pm
More comments! 15 minutes of comments. Comment-o-thon!
March 7th, 2008 at 9:00 pm
Please fired some one
March 7th, 2008 at 9:00 pm
Classic. Glad all of my shizzle is off of the DH nizzle…on a different host. All I have left to do is copy over email and then I can close the account…fo shizzle. My advice is make a clean break as quickly as you can…otherwise it will be death by a thousand cuts.
March 7th, 2008 at 9:00 pm
http://www.top10hostinglist.com/
March 7th, 2008 at 9:00 pm
A lot of cash are lost because of this downtime. Please kindly do the best you can do immediately~