File Server Issue
Posted (February 8th, 2008 at 9:19 am PST) by TerriOne of our file servers is experiencing difficulties that we are currently working on diagnosing. This will cause load and downtime issues for users on these servers:
abbazaba absinthe achilles adams admiralty aero alameda almalibre alondra altoid alvarado archibald arrow atlantic avalon aviation babyruth badillo balu bandini barranca barry baseline bazooka beaudry bellflower beverly bixel blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bubbleyum bundy burton butler butterfinger cahuenga caramello casamento catalina centinela century cerritos cheeto cheezit chex chiclet cienega cognac colbert compton cookie cordial crackerjack crenshaw crunch cupcake dingdong doheny dorito drumstick dumdum edendale euclid fairfax fc23 federal fig fireballs firestone fletcher flower foothill formosa frito frontseat fruitland fudge fundip garland garth garvey gobstopper goober good gower grand granola grappa gumdrop gummi haven heath heaven highland hillhurst hillmont hoho holt hoover hope hops hyperion jalapa jawbreaker jefferson jerky juniormint jurupa kidsgames kiki kitkat krackel lankershim larchmont laurel lemonhead licorice lifesaver limbo-spunky3 limbo-spunky4 locust look lucas malt manchester mariposa mayo melrose mento milliken mills monroe moorpark mounds munchies nash necco nerds nilla nougat nutrageous oakdale ocean olive olympic ord oreo ouzo overland oxnard paramount payday pernod pez pico planters plenty pocky popcorn poprock powerbar pretzel price pringle rampart razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rossmore rum runts sake saltine sanborn sawtelle schnapps scotch selma sepulveda shocktart skittle skor skybar slauson smores snicker snocap soto sourpatch spree starburst sugardaddy sunset taffy tampa temple tequila toblerone toffee tootsie topanga torrance truffle twinkie twix twizzler union vanowen vermouth vine virgil warhead washington western whittier whopper wilmington wilshire york zagnut zuma
If you are on these servers, this will may also cause email services to be unavailable. We apologize for this, and we will post more information as it becomes available. Thank you for your patience.
Update 9:50 a.m. PDT: The file server had hung on a thread and decided to throw a kernel panic. Right now it’s saving crash data and then should boot up. After it boots up I’ll need to soothe the servers that are hurting because of the broken mount. We should be back up and running within a half an hour, but I’ll post again if something else goes wrong.
Update 9:56 a.m. PDT: The file server was still not happy and died again. We are still working on resuscitating it.
Update 10:08 a.m. PDT: We found the problem. A bad drive was completely hosing the file server. We replaced the drive and it booted just fine. I’ll be massaging servers that still hurt now.
Update 12:38 p.m. PDT: The rest of the servers that were affected by this have been fixed. If you continue to have issues, please contact support.
122 Responses to “File Server Issue”
Everything is down, i hope it will be resolved quickly. =O(
Please resolve it quickly, all my sites are down!
And I thought I broke it! Whew! It wasn’t my super long Blog Carnival.. Annoying let’s hope there’s no permanent damage to the files!
Investor, if that’s the case: “In DH backups we trust XD”
not just email - whole site is down.
I am glad you guys are working it, but I must say I object to the fact that all of my sites, email, and jabbers are down gets classified as only “medium” severity. If EVERYTHING is broken and that is only “medium,” what the heck qualifies as severe?
More down time, yippee.
As it happened, I was messing around with my .htaccess file at the time of the outage. I was certain it was all my fault.
damn sites are down
I guess the server can’t handle htaccess, blog carnivals and coffee pots all at the same time:! ;D
Will, your site is now loading SLOWLY!
Kenneth
This is NOT a MEDIUM severity! I am losing business, all my sites, email, etc. are down! This is SEVERE!
Arg! After that whole billing fiasco, now this? I’m seriously loosing my faith on DH…
Yeah I agree this should be marked as SEVERE, since everything (http, email, ftp) is down!
I agree, this is not a mere inconvenience. This is not a medium issue. All of my work is at a total standstill. My clients are going to be so VERY pleased.
To quote DMB:
“….you pay for what you geeeeeeeeet….”
Hurry up and fix! Hurry up and fix! Hurry up and fix! Hurry up and fix! Hurry up and fix!
I think it’s just “Medium” because we may recover from it unscathed (pissed, but able to just resume normal operations). I think it would be severe if there was data loss, or equipment damage, or an outage lasting a long time.
So only 234 or so are down. I guess if dreamhost has about 50,000 servers then this really is a ‘medium severity’ issue >.>
After the whole billing thing recently, other ‘minor’ problems, and now this… with the change in hosting plans we had not so long ago, I think those of us who have been here for awhile and stick around after all this crap should get our space and bandwidth doubled again like they used to do for us at random.
Yes, that is coming from a person who isn’t even using 2% of space and even less of bandwidth, but still paid another 94 cents per month for the upgrade :p
I want to see a picture of Josh with bald spots from ripping his hair out over this!
SEVERE!
ps what is the ugly font on this page?
Can everybody who says “you get what you pay for” just stop already.
I paid for hosting. no more no less. I didn’t pay for it to be down all the time!
I like DH. They are the best alternative out there but the downtimes are getting just a tad annoying now!
You must be a fan of nickel slots…
Do you get this upset when they don’t pay out, too?
yes please hurry, this is really annoying 
My sites are on the rockyroad server. They were down for a bit, but they are now back up. So this probably means that there is a light at the end of the tunnel for the rest of you, too!
Holy crap that’s a lot of servers.
Of course, this will be the first time I’ve had any downtime in a about six months. I feel lucky.
oh man, timing always seems to work out like this. I just posted the announcement for the official Chiditarod 2008, and everyone will be coming to the site. like, now.
please do your best, dreamhost team. i need you now.
ps - redundancy???
some of my sites are coming back….
“Update 10:08 a.m. PDT: We found the problem. A bad drive was completely hosing the file server. We replaced the drive and it booted just fine. I’ll be massaging servers that still hurt now.”
this is exactly why i LOVE dreamhost. downtimes happen to every host. but i’d rather read a “real” explanation like this than crappy business-speak. dreamhost, i love you.
Funny the way it’s hitting me
1/2 my sites are working 1/2 my sites are down.
Anyone on Burton have their sites back yet?
I am on pernod and my sites are up.
uff
Burton is still down for me
it is severe. all sites are down. all my future sites will have to be on another host, sadly.
melrose still down
How the fuck can a single drive take down an entire file server? Unless it was THEE master drive, and in that case, it should have had 2 redundant mirrors. Using a Ti-86 to switch NFS servers? Do you guys know what redundant raid is?
How long is “massaging” supposed to take?
I’m up! way go! good job!
Thanks for getting that going so fast. Keep up the good work.
while admiralty is still down, i am kind of enjoying the phrase “massaging a server”
melrose is up, thanks
maybe you should hire envygeeks for a day of consultation?
My websites are down, but happily my email works 
Jabber is dead too…
Awesome! I was paying someone to test my new site today. Now I’m paying someone to sit here at the office and watch the connection be refused.
İnternetin Sosyallaştığı Site. Bir Pirikette Sen Koy! Site Down
http://www.piriketseverler.tr.gg
Well, archibald seems to have finally waken up!
… I seem to be back
Aviation is still down. Help!
The irony here is that my primary dedicated server with SoftLayer died miserably yesterday so I temporarily switched my largest site over to DreamHost. When it rains it pours.
+1 ender. Real explanations are best.
… waiting patiently - kinda - not - please hurry and bring aviation back up
This should be severe. It’s not moderate when 20 domains on your server are not accessible. I think it’s only severe if you get way overcharged? The DH priority scale is a bit skewed based on frequency of issues, it seems.
It’s been an hour since the last update and my sites are still offline. Can we get another update, at the very least?
my site on licorice is back up
Dreamhost probably has between 1500 and 2000 servers. For those that are mathematically retarded: if a unique server exploded every day, each server would still go years without issue.
Some of you seem to base the frequency of posts on how it would pan out at all the little hosts out there with 1 or 2 servers. That’s not how it works.
And for those that think something should be “severe” because it affects you–that’s also not how things work. If 99,900 out of 100,000 customers are fine, that’s not a catastrophe by any means.
Ohhh.. thanks Neo.
I should probably play the lottery then? I get nailed by each issue.
Thanks for the “probably” statements, you condescending artard.
I’m on burton, which is still not back. Just a blank page at my site.
First major outage I’ve experienced in a long time. I use Hyperspin for daily reporting on site up/down time. They say my site up on average 99.866% over last 30 days. Not including this outage.
That’s a darn good average. So relatively relaxed since reading this post + comments. But, guys, please get things back on asap. Thanks.
235 servers. With a single drive. Ridiculous.
foothill seems fine as of about 10:10
if I get a USB brain implant, can I pretend to be a server and get a massage?
My site was back up at 6.35pm GMT and is still up. So looks like burton is fixed.
Thanks!
Sorry, that’s 7.35pm GMT. Darn time zones
damn whats happening on cerritos?!
most of my sites are up, there are only 2 that can’t establish a connection to the server..
my new & most important site is down, i’ve put lots of efforts in promoting it by PPC among other methods.. and my members gonna loose faith in my service if this continues..
please fix the problems soon.
I’m so lucky at this time. My servers arent in these
Wow… this is really the first time since this blog started that my server is actually mentioned.
@57 - File server. Not hard drive. Idiot.
Hmm, this is surprising. My server’s actually having issues for once. Good things can’t last forever, I suppose~
Tootsie still not working.
This is supposed to be resolved?!
I’m still having troubles accessing my site.
At least update this blog with new downtime info!!!
All my websites on Taffy are down, sometimes they come back up for a few minutes and then they all die
All my sites are down. Ftp works.
blowpop seems to be down still…. 4:00am Eastern Feb 9
Oh, and FTP works. ditto.
How do I know which server I’m on?
Whichever one it is, it’s been down for hours now, and at 3:25 am CST, it’s still down. Awesome.
Laurel is down still too.
And I’d like to pick a fight with everyone who is complaining that “ZOMG MY BUSINESS IS DOWN WAAAHHHH WWWAAAAHHHH!!!”… but I am le tired… so I will take le nap, then I will FIRE ZE MISSILES! when I return to consciousness.
Looks like problems continue, at least with burton - site down again from 30 mins ago.
@5×5 in your account page, you’ll see a tab named Account Status. Click on that to get a drop-down list; among the info will be the name of the server you’re on.
the problem does not seem to be solved. my domains don’t work!
Thanks for the info, Neville!
morning wake up in UK. haven still down. Like others thought it was my fault. First outage since we started our community site. Given warning by InternetSeer. Odd test results last week suggested something wrong - reboots, heavy useage on village site, unlikely. Comfort from DH and now this.
I’m on rosecrans and it’s still down as well. FTP works, though.
Seems blingy mail server is tied too this, all send on e-mails form mail is not working properly…
heath is up again after 2 hours downtime. Fingers crossed.
What’s with the multiple timezones used in the dates.
We not in the US don’t know what they are, please use UTC (was GMT) times.
Any BTW, my sites are still down at 1055 GMT.
All my site in crunch server still down, please do something to fix this!
Krackel looks like it’s still down.
Jalapa is still down - >4 hours now
bixel is still down.
chiclet is down.
Pocky is still down
please bring it up soon!!!!
dingdong is still down…
it appears price went screwy at around 7.00am, EST, feb 9th.
This is not good. Site still down 3-1/2 hours later. And can’t get in to burton via FTP either. No response to support ticket filed at 10:26 GMT. Odd, as always auto-response at least to such reports.
Oxnard has also been down for several hours. I’ve had a few sites pop up as available for short periods of time while the rest stayed unavailable. Oh well. Hope they can get it sorted pretty soon.
Server Rollo Still down.
Is it a good idea to publish “publically” usernames like that?
@93 What usernames? Are you referring to the list of servers?
federal is down
My website is not reachable by now… Neither via webftp I’m able to get to it, please fix this up. It’s hosted on lemonhead…
Server washington is still down
Our server (wilshire) is down!
Just noticed that my site is back up. FTP access works, too. So looks like burton is fixed again. Fingers crossed that things stay up.
Krackel still down…
I find it a little odd that DH can flag “Problem fixed, all servers should be fixed” when a number clearly weren’t.
Surely they have some simple automated tool to probe sample domains on each affected machine to check it?
Crap. Today was the day I was going to move a site from compton to a new dedicated server. WTF this sucks! BAD TIMING!
Broadway and others are still down as well
chex is down
moorkpark is down and the panel too.
moorpark is down and the panel too.
fundip seems to be down, along with FTP
The panel is also not working at this time. I know that this is not the *official* way of reporting this issue, but the official way is not working, so… PLEASE FIX THIS ASAP!
OK, looks like things are back to normal for now. Thanks guys!
not that you read this but my domain on whittier is down….
* * * NEW PHP SENDMAIL ISSUE - DOES THIS AFFECT YOU TOO? * * *
php sendmail isn’t working with ANY of my sites. In other words, my contact forms don’t work, WordPress won’t email me notification about new posts or comments or subscribers, etc. This is for both my sites. Everything was working fine for over a year before this morning.
My client email and webmail are working.
You have indicated that this server issue for hosting is fixed. it is not fixed as of 4:15pm PST. This is affecting my sites which generate leads for businesses. I have been waiting FOUR HOURS at this point for some test leads to be delivered. YOu got big problems there.
So you have THE SAME problem I do, Michelle.
I believe you’re right.
This is terrible. I hardly ever complain to speak of. I’m usually very understanding of DreamHost’s issues including the billing error and nothing in life is perfect.
But to TELL site visitors their contact request has been forwarded to you, or their referral of your site to their friend has gone through, and it hasn’t… is just a marketing and customer service disaster, I’m sure you’d agree. Talk about losing money and opportunities and p’ing people off for no good reason.
My site was down when I went to bed, and up and running smoothly when I got on the computer today. That’s how I like it to go. Simple and painless.
DreamHost is finally delivering my PHP sendmail from earlier today… this is a good thing.
IF you’re going to have an issue where it doesn’t go through right away, at least it’s a plus if it goes through within a day or so. That’s what happened.
Thanks.
Cheezit is still not properly sending PHP or Sendmail requests
Sendmail is busted or severely delayed on Thorin also.
it is very important for internet users.i will write your site .. please write
me back. thank you
moorkpark is down and the panel too.
Siteniz icin aradfiginiz tum html kodlarini burada bulabilirsiniz
wwww.kodservisi.tr.gg
html kodlar ve webmaster araclari icin tikla
thanks
My websites are down, but happily my email works
Oh… Crapp….