File server failure (Resolved)
One of our file servers failed about 30 minutes ago causing high loads and site inaccessibility for some of you. We are currently swapping in new hardware so everything should be online shortly. The disk shelf is fine so there won’t be any lost data.
We’ll post an update as soon as everything is back online.
–
We have swapped out the hardware and the file server appears to be running again. We are working now to restore service on affected servers. No data was lost in this failure.
.
October 17th, 2006 at 3:25 pm
Dude, I’m not one to jump on the “beat up on the sysadmin” bandwagon (being an IT guy myself) - but what’s up with all these hardware failures recently? I will say what many others have said: I’m a loyal Dreamhost customer since 2002, but I can’t remember a time where there were so many problems as the last few months. Having said that, I have websites at 4 other hosting companies, and they are generally the same or worse in terms of problems. But I thought Dreamhost was special!
October 17th, 2006 at 3:34 pm
that was insanely fast hardware swapping. good job!
i’dlike to second chris - what is up, lately?
October 17th, 2006 at 3:36 pm
The downtime I experienced was 90 minutes (for email, web, telnet, ftp, everything). Just coming back up now.
October 17th, 2006 at 3:38 pm
subversion on ryu is dog-slow, has been for about 45 mins. is this slowness related to this failure?
October 17th, 2006 at 3:47 pm
I’ve got two sites still down.
October 17th, 2006 at 3:48 pm
My site has no problems, but my FTP access on the other hand has just decided to “actively refuse” me. I wonder if this downage has anything to do with it?
October 17th, 2006 at 3:50 pm
My sites are down, and my email is down. I come to the emergency status page and find no mention of what’s going on. In fact, the page says issues here have been resolved.
“One of our file servers…” Hello? How are we to know if it’s the server we’re on?
October 17th, 2006 at 3:51 pm
Mark still appears to be down. No sites and no http://FTP...
I’m sticking it out Dreamhost, because you’ve been good to me. But if my clients start getting pissed, I’m going to have no choice. My livelihood comes first…
October 17th, 2006 at 3:53 pm
Rob. Check your Dreamhost control panel and around the Manage Domains little submenu thing. It’ll tell you which servers are on what, like for mail it would be something like Felix or other.
October 17th, 2006 at 3:57 pm
My site is apparently still down, too.
October 17th, 2006 at 4:02 pm
“Running again” would appear to be a relative term…
October 17th, 2006 at 4:03 pm
I’m on hillhurst. All my sites are down. Customers angry….
October 17th, 2006 at 4:05 pm
yeah this is bad. i, like many of you, rely on dreamhost for real work. … not just hosting my family’s photo gallery. this recent outage is literally costing me money - i cannot hit my subversion repository, and our development has come to a halt. no response from dreamhost other than the above.
i’ve been a customer since march 1999, but this may be the last straw.
how about an update guys?
October 17th, 2006 at 4:11 pm
My sites are on glass and everything just went down about ten minutes ago.
October 17th, 2006 at 4:14 pm
I’ve got two sites still down (+48 HOURS!!!!!)
BAD SERVICE:…..
October 17th, 2006 at 4:20 pm
I’m back up. Thanks!
October 17th, 2006 at 4:22 pm
Yeah, same here.
The timing on this stinks. I’ve got several art shows going during the month of October and this is costing me silly money…
DF~>
October 17th, 2006 at 4:25 pm
i too appear to be back up. you guys have to post a status or two. not good.
October 17th, 2006 at 4:27 pm
I’m still down at hillhurst
:( 
October 17th, 2006 at 4:29 pm
Yeah, my site is offline too. As an I.T. sysadmin myself for my day-job, I’m sure they’re doing everything they can to get things back online. All things considered, I’ve been satisfied with their service. At my day-job company we’ve had our network offline longer and more frequently than Dreamhost’s outages.
October 17th, 2006 at 4:31 pm
Also still down… Resolved? wtf…
October 17th, 2006 at 4:31 pm
Load average on server galaga is nuts and has been for a while now. That server is seeing several reboots a day, which totally sucks. I feel like I’ve been really patient through all the recent issues, but come ON already, get it freaking together. I’m tired of unreliable service, no matter how much I’m paying for it.
October 17th, 2006 at 4:36 pm
New deals = new customers. These new customers = more load on servers, sometimes at an unexpected amount. Nothing is perfect, and whining won’t fix anything faster. My FTP went back to normal like a half hour ago, by the way.
October 17th, 2006 at 5:49 pm
hillhurst is down.
my website is down.
i am unhappy.
i feel like a jerk for getting my friends to sign up for dreamhost accounts. it makes me seem stupid to have recommended dreamhost to my friends when the service is so unreliable.
October 17th, 2006 at 5:55 pm
ya hillhurst aint workin for me either. all of my sites are down. dang i hope this is fixed soon.
October 17th, 2006 at 5:59 pm
Down on twix.
October 17th, 2006 at 6:06 pm
hillhurst is down for at least the fourth time in less than 36 hours, and we’re talking outages of an hour, two hours, and longer.
I have referred clients here, too, using hillhurst (and other servers)… each client signs up for their own account, so it’s not like we load up the servers taking undue advantage of the “unlimited” thing.
My hillhurst client is very unhappy right now.
An no apparent response, nothing that appears to cover this on this website really.
The “deal” is great, but meaningless without reliability.
October 17th, 2006 at 6:26 pm
still doesnt work
“resolved” my ass. where the heck is the status of the real problems?
October 17th, 2006 at 6:32 pm
Have you tried pinging and trace the connection routes of your site?
That will tell you for sure the status of your site and server.
October 17th, 2006 at 6:34 pm
I’M GETTING SO FUCKING SICK OF THIS BULLSHIT!!!!
October 17th, 2006 at 6:36 pm
HEY DREAMHOST…HERE IS HOW YOU KEEP YOUR CUSTOMERS HAPPY:
USE THE SAME SERVERS THAT DREAMHOSTSTATUS.COM IS ON…because it never seems to go down!
Also, ever hear the phrase…. “pennywise pound foolish?”
c’mon!!!
October 17th, 2006 at 6:51 pm
I have no access to mysql…is this related to the system-wide problem?
October 17th, 2006 at 6:56 pm
Looks like some SQL servers are still down. Three hours after the update that says everythings up. Good Game.
October 17th, 2006 at 7:36 pm
Dreamhost will never get better. With the cheap amazing packages they offer I bet they get 10 new clients for every pissed off DH customer who leaves. And they’re laughing all the way to the bank as they screw over their longtime clients.
October 17th, 2006 at 8:09 pm
Ouch.
October 17th, 2006 at 8:16 pm
i am getting 503 error constantly
is that what u guys get ?
October 17th, 2006 at 8:53 pm
I can’t get my POP emails, what’s going on?!
October 17th, 2006 at 9:04 pm
My site still down at this time. Update at all? (No new word here for nearly six hours . . .)
October 17th, 2006 at 9:14 pm
My trouble is with ssh. It’s slow as hell. At times i’m waiting minutes between commands.
I’m not losing faith though.
October 17th, 2006 at 9:18 pm
Our site has been down for 7h now and our staff, readers, and paying advertisers are all still waiting to be back up and running. Dreamhost has been more like Nightmare host for the last 6 months.
October 17th, 2006 at 10:06 pm
My mail is not currently working.
October 17th, 2006 at 10:47 pm
Okay, folks. We know you’ve all been inconvenienced here.
To make it all worthwhile, we’re giving every Happy Dreamhost Customer 100 Terabytes of Storage and 45 Petabytes of bandwidth. You can use as much of it as you want during the uptime between our Regular Happy Dreamhost Outages.
Are you all Happy now? You should be.
Okay - heck, I’m going to get in trouble for this - 60 Petabytes for everyone!
Whee!!!
The Happily Overcommitted Dreamhost Network.
please email comments or questions to us at our gmail account - our mail servers, Onan and Liberace, are down for maintenance / regular network failures.
October 17th, 2006 at 10:47 pm
blogto.com is STILL down. Ummmmm. Fix. It.
October 18th, 2006 at 12:49 am
smithers is down. Please someone report the outage too…
October 18th, 2006 at 1:08 am
Outage verified…
October 18th, 2006 at 3:26 am
My site is down too..
October 18th, 2006 at 3:40 am
Completely down here.
AGAIN.
October 18th, 2006 at 4:00 am
My site’s down as well.
October 18th, 2006 at 5:20 am
Still unable to access my webmail…
web access seems ok, but things have just been sluggish in general lately.
October 18th, 2006 at 5:58 am
Down doobie-doo down down… I can’t even get to the web panel to report the outage! Web very slow, webmail nonexistent.
October 18th, 2006 at 6:04 am
Site went down this morning, I have webmail though. I had no problems last night.
October 18th, 2006 at 6:05 am
…and it just came back up.
October 18th, 2006 at 6:14 am
@ RaMSeN, Josh, Brian, Jason, Dorothea, your sites seem fine here…
October 18th, 2006 at 6:35 am
If anyone from DH reads this - note we have been doing business with you for a long time - but things have gotten to the point where we can no longer tolerate this downtime. You need to do something in short order for users that need and are willing to pay extra for reliable service.
October 18th, 2006 at 6:40 am
my site’s down as well…
October 18th, 2006 at 6:57 am
@Najdorf
No. My webmail access is _still_ down. Is this a DNS problem?
October 18th, 2006 at 8:08 am
My website just went down at 11:00 AM EST time, its been doing fine all day until now
October 18th, 2006 at 8:48 am
19 HOURS and counting with no site.
PLEASE, Dreamhost staff, PLEASE give us a update with explanation. This is NOT RESOLVED.
October 18th, 2006 at 8:55 am
Its acting up again. Any time when we should get it totally resolved.
October 18th, 2006 at 9:08 am
beaudry is down this morning…and i can’t access my ftp==says target machine actively refused it.
what’s up??? i’ve got many businesses at stake!
October 18th, 2006 at 9:29 am
Yep…I’m generally a very patient person but this is getting very difficult to handle especially when there isn’t any communication. I have several sites and many of them are still down. A simple explanation would be greatly appreciated.
October 18th, 2006 at 9:57 am
my site is back up. the verified outage on the Current system outages you are tracking section hasn’t been updated yet.
thanks dreamhost
October 18th, 2006 at 10:30 am
Resolved hu? All my sites are still unreachable… have been all day.
October 18th, 2006 at 10:49 am
@David
What does the Current System Outages Tracking system say?
October 18th, 2006 at 11:36 am
Is anyone having a problem with their websites on the server “squid.dreamhost.com”? My sites (including squid itself) are down.
October 18th, 2006 at 11:42 am
Sorry, I meant “squirt.dreamhost.com”
October 18th, 2006 at 12:06 pm
Other companies have problems all the time, Look at valueweb over the last few days:
Oct 18 EST Attention all customers:
9:00 AM The mail servers are now accepting connections. Expect to see some latency and possibly connection errors as our mail servers stabilize. We appreciate your patience.
3:00 AM We are still experiencing issues with our email system. We are working as fast as possible to get this resolved. We apologize for any inconvenience this may cause you.
Oct 17 EST Attention all customers:
9:00 PM Our Administrators have implemented some modifcations to the hardware and software within our mail network. We cannot guarantee that this will resolve the recent mail problems however these modifcations should help to resolve the issue with the mail server load. We appreciate your continued patience.
3:00 PM We are currently making modifcations to the hardware within our mail network. While we cannot guarantee that these changes will correct the current issue, they are being implemented in order to address the current problems concerning mail delivery.
9:00 AM Valueweb is experiencing ongoing issues related to mail server load. These issues may cause you to be unable to send or receive mail. Our administration and engineering teams have been working to have this issue resolved as quickly as possible. We apologize for any inconvenience that this may be causing you in the meantime and thank you for your patience.
3:00 AM Our Administrators continue to work diligently to resolve the mail server load issue. We appreciate your continued patience.
Oct 16 EST Attention all customers:
9:00 PM Our Administrators continue to work diligently to resolve the mail server load issue. We appreciate your continued patience.
3:00 PM Our Administrators continue to work diligently to resolve the mail server load issue. We appreciate your continued patience.
1:55 PM Valueweb is currently experiencing issues with it’s mail servers related to load. Symptoms include being unable to receive mail, being unable to send mail, as well as being unable to log into your e-mail. Our administrators are working diligently to resolve this as quickly as possible.
11:30 AM Our Systems Administrators are working with our storage system vendor in order to diagnose and resolve the load issues on our mail servers.
9:00 AM Currently our mail servers are experiencing extremely high load. This will cause failure to connect, timeouts or slow connections when both sending and receiving mail.
3:00 AM There are currently no system outages.
October 18th, 2006 at 12:15 pm
Down since the last four hours…and counting.
October 18th, 2006 at 1:18 pm
My site is down. May I know why? http://www.myth200.com
Pls assist, as I’m working on several projects on the site. Time is precious.
Thanks!
October 18th, 2006 at 2:04 pm
After a very sloooooooooow day all of my sites are down. I like looking at the pictures of employees smiling, partying and “living it up” on the dreamhost.com home page. It makes me feel good when my sites are down. At least someone is enjoying themselves. I came from godaddy and never had an outage that I noticed in over 2 years. Dreamhost was recommended to me because of the control you have. (godaddy gives you basically none) However dreamhost has failed to realize that control means nothing if your site is down all the time.
October 18th, 2006 at 3:40 pm
my site on lenny has been slow as hell for the last few days.
October 18th, 2006 at 3:46 pm
my database suddenly stops working?
October 18th, 2006 at 4:37 pm
I am on lenny and my sites and databases are down.
October 18th, 2006 at 5:26 pm
Need a new host! Reccos?
October 18th, 2006 at 5:33 pm
Some subdomains working other not. Main website is down with a 502 error … dudes .. there is a virus causing that exact page to show. What’s up?
Any update on when it will be up?
October 18th, 2006 at 5:36 pm
I started getting a database error out of nowhere and its still there. I cant connect to a my other databases either. Its been close to a couple hours, Im not throwing a hissy fit I am simply eager to get my site working.
October 18th, 2006 at 5:48 pm
My site is still down: http://chomypham.com. I would like to know when it work again?
October 18th, 2006 at 5:54 pm
DH, the status page is a great idea, but it’s only as useful as the information on it. That means when you say “a file server” we have no idea which one, and that doesn’t help anyone (unless it is a blanket for multiple issues…). If you name the specific one, not only do people know that you are working on it, they won’t flood you with as many extra support tickets. My site is just blank right now. I can ssh and see the files, but it’s not coming up with anything, but I have no idea whether this is related so I have to submit a ticket anyway. Someone whose site is down might assume that it is related to this and not send a ticket to save the staff some work, but that’s just going to be more downtime for that person. Please be specific!
October 18th, 2006 at 5:57 pm
28 hours straight downtime here. No public update. WEAK.
New host recommendations?
October 18th, 2006 at 6:14 pm
I can see my database with php and see my files via ftp but my site still says database error.
October 19th, 2006 at 3:34 am
No matter how many times Dreamhost’s servers fail they will still be infinitely superior to the terror that is easyinternet solutions.
October 19th, 2006 at 3:54 am
I have to laugh at all of the people that post their websites saying its not working. Whenever I check its always fine, shut your whining and stop inadvertantly spamming, you know thats why you do it
October 19th, 2006 at 6:06 am
Craig, you mean when you check them hours / a day after the the post? You must work for DH support!
October 19th, 2006 at 12:17 pm
46hours later, we’re back online.
October 24th, 2006 at 4:33 am
Boy, I’d hate to see how some of you people would have reacted if you’d been Managed.com customers.
Managed sold its business to another firm, which then physically relocated all the servers from California to NJ… without warning anybody.
Some sites were down for a WHOLE MONTH, even though the company said that all issues were resolved after the second day. To make matters worse, they mixed up the equipment, so some customers found themselves trying to conduct businesses with other people’s data.
A lot of stuff just ended up lost. A few people were able to recover their data when others posted messages saying, “Hey… what’s this? It isn’t mine! Anybody recognise it?” but most have lost their stuff forever…
With that in mind, a few hours downtime doesn’t seem so bad.