Dreamhost firewall error (Resolving)
Due to a typing error on our primary router while trying to block a denial of service attack, DreamHost is currently offline. This includes all email, web hosting, etc. A technician is about 15 minutes out from our datacenter to undo this mistake.
I apologize for this mistake. I was intending to be editing our non-live access-list, and edited the live one by mistake.
Update 9:54 PM Pacific: We have unblocked the firewall and our routing is coming back online. Sites should start working again as soon as our router finishes loading its tables. I use “resolving” because it is still processing the routes.
.
March 7th, 2008 at 9:49 pm
“Steve” says, “As a software engineer for a large web company, I am required to say that this kind of mistake should NEVER happen.”
As a leader of a team of sysadmins for a large IT company, I wish that software engineers would stick to their code monkey work, and not pretend they understand how infrastructure works. Go write some buggy code please, and leave the sysadmin work to the big boys.
People screw up. Shit happens. Get over it. As others have pointed out, if your stuff is so damn important, why are you hosting for 10 bucks a month? I’ve been using DH for a bit over a year and this is the first actual outage I’ve seen that touched my stuff. And you know what? Tomorrow the sun will rise, birds will fly, and life will go on. Your precious site will be back up just fine. GET OVER IT. Dude screwed something up. These things happen.
Ever write a bug, Steve?
March 7th, 2008 at 9:49 pm
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!! 1 HOUR OUCH !!!!!!!
March 7th, 2008 at 9:49 pm
500 GET
March 7th, 2008 at 9:49 pm
GG DH.
March 7th, 2008 at 9:49 pm
I’m losing almost a half cent a minute! Hurry up and fix it already!!!
March 7th, 2008 at 9:49 pm
my fault for not migrating off already, this is the third or fourth major downtime in 2 years. dreamhost has gotten terrible, just to state a fact known to its long-time (7+ year) customers.
March 7th, 2008 at 9:49 pm
http://www.vandertramp.com/INTERNETDOWN/
March 7th, 2008 at 9:50 pm
@charm we are in the middle of enrollment season at my school, I could care less about the internet, this is costing us enrollment and by direct association MY JOB. which not coincidentally is directly tied to keeping our website running.
March 7th, 2008 at 9:50 pm
@Kirk
Hey Kirk, I do know what it’s like to lock yourself out of a system, and I’ve pulled a similar trick by restarting sshd with a malformed configuration on a remote server.
That being said, for one hour in, I don’t think it’s out of line to ask for at update if they’ve got a tech on the scene, reconfiguring the firewall.
March 7th, 2008 at 9:50 pm
Come on Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it Fix it
Juzt Fix it!
Sincerely Fix it!
Christ!
March 7th, 2008 at 9:50 pm
heh time to reroute email.
March 7th, 2008 at 9:50 pm
To the people flipping out at the top of the comments:
I think it’s time for evening meds.
To those keeping their sanity wrapped around themselves:
Kudos.
To Kelly:
C’mere and get a hug.
March 7th, 2008 at 9:50 pm
it’s back up
March 7th, 2008 at 9:50 pm
very rapid
March 7th, 2008 at 9:50 pm
Working again?
March 7th, 2008 at 9:50 pm
IT’s up now.
SEE y next down time!
March 7th, 2008 at 9:50 pm
hey! it’s taking too long now!
March 7th, 2008 at 9:50 pm
Phew for a second there i thought DH was running off my laptop i reboot from updates and come back to see DH doesn’t work!!
March 7th, 2008 at 9:50 pm
Like I SAID, it’s up.
March 7th, 2008 at 9:50 pm
Ouch… 1 hr and counting
March 7th, 2008 at 9:50 pm
I think it’s great you guys are honest about this stuff. I feel for you. At my first (real) job I built a web system for tracking customer data for a large cellular company. The web was new back then and ops didn’t do backups on this server and the production server was the same as the development server. Anyway I did an rm -f on what I thought was just some development stuff, but it was taking a while. You can guess what happened. My boss was fairly cool and was able deflect at least some of the heat by blaming ops for not having a backup.
March 7th, 2008 at 9:50 pm
IT’S BACK UP FOR REAL! MY SITE JUST LOADED!
March 7th, 2008 at 9:51 pm
They are probably jamming screwdrivers into and jerking out cards from their cisco 7500. Bang on it with a rock, and maybe that loud fan won’t buzz anymore. If I didn’t get a Terabyte of oversold space for $9.00 a month, I would be outta here. Thanks Dreamhost for teh awesome.
March 7th, 2008 at 9:51 pm
It is back!! my websites are loading up now.
March 7th, 2008 at 9:51 pm
http://www.youtube.com/watch?v=lebhrb9Iofo
March 7th, 2008 at 9:51 pm
Thak god it back up !!!!!
Good luck everyone else!
Lol
March 7th, 2008 at 9:51 pm
All better
March 7th, 2008 at 9:51 pm
Everything seems to be back to normal now guys. Go check your sites and if they aren’t back up just keep clicking refresh a few times.
March 7th, 2008 at 9:51 pm
Well, it’s late and I’m sleepy.
Good night to all and to all a good night.
-Yo’ Daddy
March 7th, 2008 at 9:51 pm
Has anyone ever thought of “social networking hosting”? Hosting companies that are run by the very community that they serve! HAH! That would be something else… Because the community always seems to know how to run a business better than these “dumbasses” that do “nothing but fuck up” and still manage to take our $10/mo!
March 7th, 2008 at 9:51 pm
My mail’s working again, and my site is back up.
Good thing.
March 7th, 2008 at 9:51 pm
Sites back up. Email too.
Do not ASK me what I am doing on the internets at 12:51am. Maybe it was because _Jumpers_ kinda sucked…
March 7th, 2008 at 9:52 pm
aaaaaaaaaaaaaaand we’re back!
March 7th, 2008 at 9:52 pm
it’s over. all is well with the world and we can go back to making our millions
March 7th, 2008 at 9:52 pm
*shrugs* My sites up now. Seems to be fixed??
March 7th, 2008 at 9:52 pm
Its back now, faggots.
March 7th, 2008 at 9:52 pm
If the problem is what they’re describing, an incorrectly entered firewall rule, it should take a minute to fix. (Actually, it should take 5 seconds or so, but we’ll give them a bit.) Just remove the damned rule. Done. Fixed.
Then you can re-add the correct rule to work around the DDOS.
It should not take this long to remove a broken firewall rule.
March 7th, 2008 at 9:52 pm
Yup, it’s up now.
March 7th, 2008 at 9:52 pm
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
REMIND ME AGAIN WHAT I’M PAYING YOU GUYS FOR?!?!?!?!?!?!?!?!?!?!
March 7th, 2008 at 9:52 pm
Well, it’s late and I’m sleepy.
Good night to all and to all a good night.
-Yo’ Daddy
Just checked and things are up. Someone could’ve made 10 bucks!
March 7th, 2008 at 9:52 pm
http://www.youtube.com/watch?v=dRc4njfVyAE
March 7th, 2008 at 9:53 pm
Wooohoo! Back to normal! Thanks for the rapid response to a colossal oops!
March 7th, 2008 at 9:53 pm
It’s up!
Someone found the emergency bottle of Viagra!
Now remember, if it stays up for more than 4 hours, call your doctor!
March 7th, 2008 at 9:53 pm
Ehh, again?!
Dreamhost has crap uptime…everyday its down atleast once…its starting to really piss me off
March 7th, 2008 at 9:53 pm
373
Nic Says:
March 7th, 2008 at 9:32 pm
@ #326 Cal: You’re a moron. Ever think some people might have hundreds of dollars worth of PPC ads running to these sites.
Just because you haven’t figured out how to make real money online, doesn’t mean other people haven’t.
Hello Dumb Ass - i was talking about the people who are saying they are making millions, they are full of shit and you should learn how to read and understand before you respond and post. The world if full of jack asses like you - talk shit
March 7th, 2008 at 9:53 pm
hahahaha i love watching you little pussies cry
March 7th, 2008 at 9:54 pm
WE ARE ONLINE AGAIN!!!!
Yeaaaaaaaaaaaaaaaaaaaaaaaaaaaaaah!
March 7th, 2008 at 9:54 pm
Mine is still broke
March 7th, 2008 at 9:54 pm
I really don’t understand this anger. We had often problems and the Dreamhost support team helped us each time
Some where our mistakes. Are you more happy if soembody is fired. Try to be a little mere human. It isn’t the end of the world and neither the end of your online business. I think we will go far beyond 1000 posts
March 7th, 2008 at 9:54 pm
Yay it’s back!!