Dreamhost firewall error (Resolving)
Due to a typing error on our primary router while trying to block a denial of service attack, DreamHost is currently offline. This includes all email, web hosting, etc. A technician is about 15 minutes out from our datacenter to undo this mistake.
I apologize for this mistake. I was intending to be editing our non-live access-list, and edited the live one by mistake.
Update 9:54 PM Pacific: We have unblocked the firewall and our routing is coming back online. Sites should start working again as soon as our router finishes loading its tables. I use “resolving” because it is still processing the routes.
.
March 7th, 2008 at 10:24 pm
it happens..
it happened again..
something is bound to happen, yes, EVEN AGAIN, eventually..
Frustrating, can be.. thats for sure..
human? (as in error)..
most definatley.
I would place bets on ‘people (not)being ‘happier’ with ‘other’ hosting companies,
… if ONLY THOSE people could be as HONEST as Dreamhost has (always) been…
instead of ‘blaming’ DH, blame the phuking script kiddies that start the DDOS’s….
oh, and as Always,
MAKE *YOUR *OWN *BACKUPS, AND PLAN *YOUR OWN* SURVIVAL.
THANKS DH! UROCK.
March 7th, 2008 at 10:29 pm
Haiku: 5 syllables, 7 syllables, 5 syllables.
Yay dreamhost! I hope no one gets fired. I would be so sad.
March 7th, 2008 at 10:35 pm
Phew. That was close.
What I liked was the instant post on dreamhoststatus.com about what went wrong, the apology that accompanied it - and most of all - knowing that so many crazies like me were up tinkering with their websites right at this moment.
Welcome back Dreamhost. We missed you like hell.
March 7th, 2008 at 10:40 pm
Anyone who pays that little for a host and expects it to be good is an idiot.
March 7th, 2008 at 10:44 pm
The fingers you are using…are too fat.
To obtain a special dialing wand, please mash the keypad with the palm of your hand….now.
March 7th, 2008 at 11:17 pm
If you were smart enough to make money online, you’d be smart enough to know how to pause campaigns.
March 7th, 2008 at 11:19 pm
WOW YOU GUYS OFFICIALLY BLOW!!!!!!!!!!! I THOUGHT NETFIRMS WAS BAD I WAS WRONG!!!!!!!!!!!!!!! GET YOUR F***ING ACT TOGETHER AND STOP CLOWNING AROUND!!!!!!!!!!! I HOPE ALL THE SEARCH ENGINES FIND THIS, DO NOT BUY DREAMHOST HOSTING!!!!!!!!!!
March 7th, 2008 at 11:20 pm
Someone call Ryan’s mommy so she can come change his diaper.
March 7th, 2008 at 11:21 pm
DREAMHOST IS THE WORST HOSTING ON THE INTERNET!!!!!!!!!!!!!!!!!!!!!!!!!!
March 7th, 2008 at 11:36 pm
i dont think i will renew with DH this year…
March 7th, 2008 at 11:41 pm
blog still not working. Says database error
March 7th, 2008 at 11:42 pm
Dreamhost is hardly the worst hosting on the Internet. I don’t know of any other hosting that is as transparent about what goes wrong when it goes wrong. Granted, I wish things didn’t go wrong. On the other hand, Dreamhost *is* budget shared hosting. I don’t run a business on it and would not put anything mission critical on it. For me, its for experiments, some development, and my personal website.
If you choose to run a business on budget shared hosting, you have to weigh the sometimes higher risk of downtime vs a dedicated managed solution. The latter costs more but probably gives much better uptime. In my experience, Dreamhost has about 1 really really huge bad thing that happens a year like this. Their blog *is* really funny, though. Its just a balance.
Dreamhost, try not to do this particular dumb thing again. Thank you.
March 7th, 2008 at 11:56 pm
Glory Glory be to he his name on high. For he hath delivered the people of Raptah! Jeezuhs! to thy promisssed
LAAAAAAAAAAAAAND of bootylicious and cream.
GET. MAH…CAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANE!! BIASCH!!!!
March 8th, 2008 at 12:12 am
i {heart} dreamhost.
awesome!
March 8th, 2008 at 12:33 am
Thanks for working hard to get our sites up and running. All is forgiven!
March 8th, 2008 at 12:59 am
DH get your hands together!
March 8th, 2008 at 1:01 am
I will leave dreamhost,looking for other web hosting
March 8th, 2008 at 1:08 am
Many of my websites can’t work.what’s wrong?
March 8th, 2008 at 2:10 am
STUPIDS !!!
March 8th, 2008 at 2:46 am
lmao Matt
I’ve always found Dreamhost good, and I like they are at least honest about things going down.
But really there seems to be more f’k ups lately, at least ones that I notice.
Ive made stupid mistakes like this myself, and its great its all fixed up.
but it just makes me realize that dreamhost shouldnt be used for any serious websites.
March 8th, 2008 at 4:15 am
Seriously, someone NEEDS TO BE FIRED ASAP.
Such mistakes can not happen on ANY company not matter how CHEAP it is.
Also, closing the entire network firewall by “mistake” it’s just plain… stupid, really.
First rule of a sys admin… think twice before hitting enter, and before that, think twice again.
March 8th, 2008 at 4:40 am
After being with DH for almost a year, I am no longer surprised. There is at least 1 major outage per month, guaranteed.
March 8th, 2008 at 5:32 am
Severity: High -> Classic!!! it’s like ‘Hmm, everyones down… should we…hmm let me think, no, no..hmm … um no definitely, yer definitely yep, lets set this one to high’
fuk your heart must have hit the fuckin floor like a steam train, you poor bastard, I think I heard it here in London.
I hope you recorded that first phone call for help, I can hear it now ‘FARAAARRRRRKK DUUUDDDEEEEEE FFAAAARRRRRRRKKK’…
Was thinking it would be good if you could just redirect our sites to this blog on downtime, bet we wouldn’t have one unhappy customer
It’s a crying shame to have a site go down, but so worth it for this blog
Glad I slept thru it
March 8th, 2008 at 6:12 am
No le veo la gracia e estos tipos. Nosotros no tenemos nada que ver con sus problemas y entiendo que los que hacen este tipo de ataques no son bien vistos entre sus colegas, ya que es una de las formas más fáciles de ataque!!!!!!
March 8th, 2008 at 6:51 am
I’ve been with Dreamhost since 1999. I love their service. I’ll stay with them. They are a friendly bunch of folks who worked hard to build a local business (I am obviously a California person). Every blue moon, they make a mistake — as do all humans. The real downside to their growth is the fact that they now have to host lots of very evil people’s stuff. Some of the comments posted here are ridiculous.
March 8th, 2008 at 6:57 am
“I’m a CCNP so I do have a clue about configuring routers. The first thing you make sure of (especially with a border router or firewall) is that your out of band access is working before making any emergency changes to a production router. Yes, it is very easy to lock yourself out when you goof up an access list. That’s why you always make sure you have a way to back out the changes before you start making them.”
I’m a CCNA and if you’re a CCNP, you should know that shit happens. If the Firewall was borked due to the access-list having an error, it’s possible he wouldn’t have noticed the error until later. Especially if he thought he was editing a non-live list as was claimed.
The very fact that there -is- a non-live list indicated that there were multiple lists saved within the device. the -fix- once he got there was probably just a simple copy-over and waiting til’ the routing tables did their thing.
It sounded as if he did the edit, drove off home, got a cellphone call once the first alerts starting coming in to the monitoring center(which of course would have been delayed), turned his vehicle around, drove back the 15 mins he was away, uttered a number of profanities, copied over the backup list and waited for propagation while people freaked out claiming damages in this one hour that isn’t seen on Fortune 500 companies (one of which I’ve done work for).
No doubt trying to extort money from a simple unfortunate glitch that could have happened to anyone and cost 1 measly non-peak-hour’s downtime.
March 8th, 2008 at 7:03 am
@RighteousIndignation: ROFL
March 8th, 2008 at 8:37 am
Is there still an issue - everything seems to be down (except dreamhoststatus)?
March 8th, 2008 at 8:55 am
Yes, everything is down again. Let’s see…
March 8th, 2008 at 9:16 am
All my sites are still down (9:15 PST). How much time does the router need to load all it’s tables?
March 8th, 2008 at 9:18 am
everything down, including the panel
March 8th, 2008 at 9:39 am
Hahahaha. Seeing these posts is pretty hilarious now that I’ve moved all my sites and services to companies that actually manage to properly do the job of maintaining an available network.
For the dude who got a VPS at Dreamhost, heh, bad decision pal. If you can stand CentOS, check out ServInt — they are excellent.
March 8th, 2008 at 10:08 am
I think a lot of other hosts would have invented a less embarrassing reason for the outage. At least DreamHost is honest about it.
March 8th, 2008 at 10:12 am
“MOTHER FUCKERS!!!! FUCKING LOOSERS!!!!!!!!!!!!!!!!!!!!!!!! GO to school before admin a Server”
LOL, are you kidding? Go to school and learn how to spell and speak with proper grammar to convey a thought. It’s spelled “losers”, not “loosers”. Dumbass.
Also, why did you capitalize the word “Server” at the end of your sentence? Usually you capitalize the first letter of the first word of the sentence.
You truly are a moron.
March 8th, 2008 at 10:37 am
Yes, it was bad that Dreamhost was down last night for a while. But s**t happens. However, I really appreciate their honesty in posting news of the problem so quickly.
Ironically Google’s blogspot has been down for 20 minutes today as I write this and all you get is an error message. I The only news about it is in a forum in India.
So, Dreamhost is better than Google at keeping users in the loop.
March 8th, 2008 at 10:55 am
Caca Pasa! Stuff happens.
I recall the old saying, “Any sailor that hasn’t run aground isn’t a sailor.” That sort of equates to, “Any programmer that hasn’t broken the build isn’t a programmer.”, OR “Any network admin that hasn’t broken the network isn’t a network admin.”. It’s all the same.
To all you folks ragging on Dreamhost, back off until you’ve done the job yerself.
I’d also tell you that if you want five-nines of reliability dig deep and come up with the bux to pay for it. You get damned near three-nines of reliability for your $10/month. If you think changin hosting providers will solve the reliability problems, more power to you. I can tell you from personal experiences in rescuing amateurs from other hosting companies that Dreamhost is head and shoulders above the lot.
disclaimer: Reasonably happy Dreamhost customer since 2002. Ecstatic Dreamhost customer since my experience rescuing domains from Verio and 1-on-1.
March 8th, 2008 at 11:05 am
OMG!!! You run one of the top 10 hosting companies in the US and you block your entire hosting service??? DUUUH!!!!
March 8th, 2008 at 11:21 am
next up:
sorry everyone, I was carrying a tray of muffins and my foot got tangled in this cord, so most of the sites we host were dark until after breakfast.
or…
oops, I fell asleep admining the servers and my head fell on the keyboard. but I’m awake now!
cue the keystone cops!
March 8th, 2008 at 12:22 pm
Down again?
March 8th, 2008 at 12:31 pm
DH posts a lot of “a technician is on their way to our datacenter” msgs. Why doesn’t this company, which is obviously doing a large business, simply staff the datacenter 24/7??????
March 8th, 2008 at 12:38 pm
Oh Sweet Jesus, can this get fixed now, please?
I was directing business contacts to my website yesterday and if my site is down for any length of time, the assumption will be that I’m off the ball for not having fixed it.
People aren’t patient when taking a first impression of a potential business partner.
I already lost money on bank overdraft fee and the conversion rate when my “misappropriated” renewal payment was taken and paid back… now I’m potentially losing face as well as business.
Really not cool guys.
March 8th, 2008 at 12:40 pm
… and it’s back! Phew.
Best of luck to everyone else waiting/confused/other.
March 8th, 2008 at 12:41 pm
First major outage for me. I’m having fun though reading through all of your posts. Of course, my clients aren’t too happy that their sites are down, but I can’t really do anything about that now….
Things like this happen. Things will be back on top soon enough!
March 8th, 2008 at 12:42 pm
Haha, just finished hitting the “Submit Comment” button, and my site loaded. Good deal!
March 8th, 2008 at 1:09 pm
I’m not surprised you’re spamming for them, since they are spam-friendly. I’d never give those scumbags a penny of my money.
They were home to the famous “Mail Gravity” opt-out spammers for years, even got busted for being sneaky about moving them around on different IPs. They kept very silent in public about that while getting called out on it.
Even when they’d try to keep a spam-free image by booting the little guys, the big spenders were safe there.
March 8th, 2008 at 4:25 pm
Still, my other hosts are never down and also have >100.000 and >300.000 customers, strange huh? Proof: http://tofu.he-hosting.de/uptime.html - I still wish my total DH uptime will reache 99.5% at some point.
March 8th, 2008 at 4:34 pm
would this have anything to do with why my Mail program won’t connect, either POP or IMAP????
since yesterday, unable to connect either way.
bumming..
staying with DH under these conditions is like flogging oneself.
I mean… I’m not a million dollar customer.. but I want to read my freaking email… I’ve done NOTHING to my Mail program settings.. all of a sudden,,, nada, neinte, nunca.. nothing…. won’t connect… bsh*t.
March 8th, 2008 at 5:12 pm
ALL hosts have downtime.
March 8th, 2008 at 5:19 pm
I love reading all the people who hate on DH when things go down… I could easily spend hours here laughing at them.
March 8th, 2008 at 5:19 pm
I love reading all the people who hate on DH when things go down… I could easily spend hours here laughing at them.