Frisky File Server Outage
Posted (August 14th, 2008 at 10:25 am PST) by briansOur Admins had to replace several drives on one of the fileservers that your machines are mounted to. The drives are rebuilding at the moment. Therefor performance isn’t quite at 100% at the moment. This will explain any problems that you are encountering when attempting to access your site, email, etc. Unfortunately, we do not have an ETA as to when this issue will be resolved. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.
UPDATE: RAID reconstruction is at 52% and counting. This does not have any affect on your service, other than the slight delay that you may be experiencing. Meaning, you do not have to worry about data loss and such. We will make another posts once it complete. Again, we apologize for the inconvenience this has caused.
Update: reconstruction is complete and all services should have returned to normal now.
94 Responses to “Frisky File Server Outage”
We are looking forward to the Frisky cluster back online at 100%. While, our site is down/sluggish, I have to say that I really like how Dreamhost built this http://www.dreamhoststatus.com interface. Makes SysAdmin monitoring a little easier.
Again?? Come on this is the last straw it seems that every month there is one issue after another with you guys….I am going to have to find another provider, even if it does cost more….
The important thing is to get Frisky up and running at 100%. Take your time, we will wait. I just hope that the problem is easy to fix.
Great timing Dreamhost! Just when a potential Advertiser is checking the site out everything goes dark. As Borat would say, “ees niiiice.” 
Anything can be break.. no worries I still love your unique services.
I agree as soon as my year is up im going to have to go find somewhere else. and I agree is with Joe, Seems like Every month its something. MySQL is super slow, about every other week something is down that effects my sites. Ive been with Dreamhost for about 1yr 1/2 and Ive been happy until about 4 months ago and its all started going downhill.
Oh and I Just Noticed.. its rated as a medium. To me Websites are down. it should be like top dog priority. dont ya think?
How hard and expensive it will be for DH to implement High Availability solutions for its clusters? I think that this must be one of the main goals for such an experienced hosting provider as DH.
Your File Servers:
slapper
Your Web Server:
elise
Your Email Server:
frisky
Your MySQL Server:
splinter
Would my site be down if this is my config?
Yes, BJ. Your mail server name is the same cluster name (frisky)
Yeah. Not Medium. Severity High my sites are Down. : /
@BJ Elise mounts data from the problematic fileserver, this caused an io backup to the point where it crashed.
Mine is back up, but is slow as snail snot.
@Moidsch - I agree. The “Status” DreamHost reports seems to imply how much of a problem this is for _them_. In this case, I suppose it’s only a fraction of their users affected and they think they can fix it quickly. However, for those of us affected, we’re COMPLETELY OFFLINE. This status is supposed to be for the _customers_.
I’ve suggested before that DreamHost have two indicators:
1) Severity: How bad for those affected, from minor slowdown up to totally offline
2) Effort: How long it will take to fix, from minutes up to hours (and beyond?)
[sarcasm] This was particularly Severe for me, as my potential player (yes, that’s singular) could not get to my game site. [/sarcasm]
Gratsee!
This is such BS. I’ve had it with DreamHost. One problem after another. I NEED TO SEND ANDRECEIVE EMAILS OR MY BUSINESS COLLAPSES. You’re about to be fired…should have done this a year ago.
Go DH, go!!!
@Nyhm & Moidsch
“Severity” has to do with how many customers are affected, not by the seriousness of the problem for those customers. Obviously, it’s serious for anyone affected. Otherwise they wouldn’t post it here at all (at any point in time there are hundreds of open support tickets - not every problem shows up here, just the ones that are serious enough that they want to limit the number of support tickets generated).
@brasscrest - Thanks for the clarification. I’ve never been quite sure what DH meant by their Severity level, except that it doesn’t seem to be useful to us (the customers). Don’t get me wrong - I love this status blog. I just think Severity confuses people (and tends to belittle serious situations from a customer point of view). That’s why I think there should be DH _and_ Customer-centric indicators.
Nice to know what’s going on at least! Thanks for the status update.
Any word on when this will be back up? any sort of time frame?
@Andrew - Are you still down? My sites seem to be back to normal (try my link above).
@Nyhm: I’m not on this cluster (fortunately!), but from the description I also think that it’s rated as “medium” because the fileserver itself is still up. I know from experience that this can cause all sorts of issues, upto completely crashing webservers because of the huge backlog of requests that can pile up.
Anyway, wouldn’t it be faster to shut everything that mounts files from this cluster down completely so the rebuild will go faster? If sites are unaccessible anyway this would seem like a logical step.
@Nyhm - yea i’m still down completely. Your’s seems to be working. But mine… still def down… hmm
My sites have been extremely slow since Saturday and I’ve been patiently going back and forth with support to get it fixed… I have no idea if this file server was the original cause of my problems, but I’m quite anxious to get it fixed… Has anyone else had any http slowness since before today?
Symptoms like: html loads fast, first image loads fast, second image stalls for 5 seconds, third image stalls for 10 seconds, etc. Often pages take over 30 seconds to load, and the dropouts are randomly different on each request.
Extremely annoyed, to say the least.
@Andrew - Thanks for the insight into this type of server configuration. It has always been a bit fuzzy how things are set up here. (I’m tech savvy, but I’m not experienced with this type of mass shared serving environment.)
For the record, my sites seemed to be fine until this afternoon, when all services were gone. This issue is still not marked as resolved, but I’ve been fine since a couple hours after the incident. (Downtime only about 1pm-3pm EDT).
Mine is down now (8:51 PM EDT). I am rather annoyed at this. Just was in the midst of authoring a new blog post.
Umm… Still down… 7:00M…
I left a perfectly reliable webhost to come to Dreamhost. In 5 years my site was down twice, that’s it! If this continues, I am so outta here!!!
haha, signed up yesterday! woohoo, and i have an interview tomorrow, YAY!……………as borat would say, “ees niice!………………..NOT”
Are we still having issues?
1:19 AM EDT
i jus hope all my files havent been lost - i cant go thru uploading 30 gb all over again
If by slight delay you mean completely and unusably slow/broken, sure! Certainly is a medium priority issue. Who NEEDS their website, really.
Mmhhh, hope the server will run again in a hour or so. I also have som clients which could looking today at the page …
äähhm, my blog as well is down …
For people with sites with advertisers this is BS!
..or business sites for that matter
Disappointing to learn that my web site and mail is down once again. I don’t know why this has a “Medium” status either. It’s not the sort of service that a company like Dreamhost can be proud of. Perhaps they should concentrate less on sending out humorous newsletters and more on providing a robust service for their paying, and increasingly disgruntled, customers?
@Andrew I’ve got a feedback from my User’s that my blog some times is realy slow. This started 2-3 weeks ago … So theres is def. a issue about loading …
shit
Ggrr. Any idea how long this will take? Would really appreciate a timeframe to work with! Everything is down!!
Is it just me or is it nearly impossible to access the main dreamhost website or the dreamhost panel? Just how much is out of action right now?
@envee no I don’t have access as well. Also not to the panel or the blog from DreamHost. So it’s a major problem …
I can’t get to the main Dreamhost site from here - this *might* be a routing issue with my ISP[1], as a trace stops before it even leaves their network, but I suppose it could be DH releated…
Everything was fine last night (UK time) but can’t get any access to web, email or SSH this morning.
[1] Virgin Media, UK
A quick check with the very useful http://downforeveryoneorjustme.com/dreamhost.com shows that DH is apparently up. Looks like it’s a network issue between them and some places.
And I think my ISP is guilty - if I use their webcache, my site loads….
@Les - Am in the UK with TalkTalk and can’t get through to the main dreamhost site. I tried pinging the website - no luck here either. I just wish someone would update the post to say that their main website is out! And that they’re doing something about it!!!
Guys - the least you can do is keep us informed about the expected timeframe of the services being down. I can’t do anything till my website comes back alive again!
Iv got 20MB Broadband, And I left my pc to upload to my website overnight, i come down at 6am to find everything timed out - and now i dont know whats uploaded so far, and today is a busy day for me, Please Dh try and fix it soon 
Come on Dreamhost, usually you are good at keeping people updated. I cant connect to panel, email, website from Virgin UK.
@Gorm yeah true usually they make emails or so. It must def. be something bad …
Same, via Virgin, UK.
ääähm, im in Zurich, Switzerland, same here …
If you’re using Virgin, set your browser to use their webcache - webcache.virginmedia.com, port 8080, and you should find you can get to DH sites. This is a problem with parts of Virgin’s network not being able to find a route to DH (and other places). I’ve logged it on their support newsgroup, which is read by actual support people who are more likely to understand “routing issues” than the front line phone people…
DH main site, panel and my own site work just fine with the webcache on, so this time DH are not to blame.
If other ISPs are affected, it may be a common route that they and Virgin use to get to particular sites.
I’m in Israel, Same problem here - can’t get to dreamhost main site, web panel or my sites (on dillinger)
Thanks Les, yep was able to get to my site and email via iphone over 02s 3g network.
I am getting married tomorrow and going on honeymoon on Sunday - really need to access my email before then!!!!!!!!!!!
Sorry, but this is shocking. Cannot access any of my sites, neither can I send or receive emails? Get moving, guys!!!
P.S.: severity “medium”??? You are kidding, right?
yes the webcache works thanks
GAH! just when i had my record on my vbulletin! THIS HAPPENS! Thats $200 lost for me, thanks dreamhost
Now it’s coming up again …
Down again …
Hi. I’m on Spunky, but I’m unable to resolve anything at dreamhost except dreamhoststatus.com. I can’t access my web panel, my sites such as http://www.johnlarge.co.uk or my email via webmail or smtp/imap. Is there a core router issue between the data centers and the UK? I’m with Virgin Media ISP, and all other sites seem to be resolving fine. Sorry i couldn’t post this via the web panel, I simply cannot access it.
Sweet ….seems like all my sites are down (Including DH Panel) 
@John, even I’m in UK and experiencing the same problem as yours.
Am in the UK on TalkTalk and pretty much all websites seem to work except a few like sitemeter and pretty much almost anything on dreamhost. Though my friends in other countries are able to see everything pretty fine. I am accessing my website through a proxy right now.
Dreamhost - would really appreciate an update on the issue.
The Virgin Proxy works for me as well, in accessing my site at least, but its not something I have ever had to use before, and it doesn’t help me at all with SSH, which is still out of commission. This must be a larger issue than just a fileserver! I still can’t access the panel even with the proxy on though
It’s working now??!
Confirmed, Virgin Media UK, and now working. My Cable Modem just restarted on its own, was down for about 2 mins, modem came back online and sites now work.
[UK Virgin Media] — no DH hosted web sites, no acess to DH account or help pages — (DH main page and ftp access ok) — BUT… also no access to GoDady account, help page or webmail for over 2 hours
yep — DH and GoDaddy pages are up again
hey - my websites and mail ist still down!
it’s already noon here in Switzerland… arrrrrggghhhhhhh
Not up yet here. Half a day of work down the pipes.
Are any other file servers affected? I’m on Boinger and my site has been s l o w most the day.
@reto are you able to get to the DreamHost homepage? I can’t get even to that …
I checked my site at 9AM GMT and it was down, 3 hours later it is still down. I can’t SSH to my server, I can’t get to panel.dreamhost.com to raise a support ticket. I can’t see dreamhost.com, THIS IS REALLY BAD. I have customers wanting to use the site and I have no info from dreamhost on the problem.
Unhappy Customer
Who will be looking for a new host
Why is everbody surprised?
This is one of the most notoriously bad hosting companies in the entire world.
@Nate why do you think so? Do you have any experience with other hosting companies?
Hi,
for those that are unable to see http://www.dreamhost.com and panel.dreamhost.com. It seems like this is caused by a problem with routing. Can you guys try to do in a dos box: tracert http://www.dreamhost.com
this is probably caused by http://www.dreamhoststatus.com/2008/08/12/network-maintenance-retry/
they probably think everything is okay.
okay,
this blog is screwing up the command you need to type. So try this one:
tracert panel.dreamhost.com
and put your results.
Hello Mee,
Yes, and Dreamhost are by far the least relliable. I think so much of what Dreamhost do is great. The Cpanel is nice, their support are usualy quick to respond and helpful, but when it comes to reliabilty, they really suck. (I think) they’re getting worse, and there is a certain arrogance about their attitude. They don’t seem to realise how adversely an outage like this effects their customers.
The attitude seems to be ‘Whoops, George dropped a Doughnut into the server again. We’ll get it sorted out when we’ve finished giving him a wedgie’. Which is fine and quite endearing until its your own shit that goes south. Then its really not very helpful.
Hi Nate
Yeah I know what you mean. The Service at Dreamhost is cool, but I always got the feeling if the server is down, no one even realise that …
To fix an error when it is so major is a must for a provider.
I hope they work on that as well …
i think dh is great - i only had server down problems 2nd time this year
and for the amount of disk space and bandwidth provided - thats jus perfect for me
I think everything is back for me.Servers are running good.I always enjoy your service. 
Grrrrrr! My sites have been down for nearly 12 hours now! And I don’t get why there hasn’t been an update here for about 22 hours!?
what’s going on DH? When will it be resolved? I can’t even make a support complaint because I can’t access panel.dreamhost.com
aaaargh!
sunny
All my sites are still down. WTH? — 403 errors, I can’t even SSH in. I’m starting to get a bitter taste of DH’s stability.
4 11 ms 6 ms 5 ms ct1-bb2-fe0-0-0-0.conversent.net [204.17.65.4]
5 9 ms 8 ms 9 ms ny1-bb1-ge-0-0-0-600.conversentnet [209.113.217.113
6 11 ms 10 ms 10 ms ge2-10.br01.nyc02.pccwbtn.net [63.218.109.193]
7 12 ms 12 ms 12 ms ge-2-2-0.1102.ar3.jfk1.gblx.net [208.50.86.193]
8 85 ms 84 ms 84 ms te9-4-10g.ar3.lax2.gblx.net [67.17.72.86]
9 85 ms 85 ms 85 ms ge1-gx.dreamhost.com [67.17.162.162]
0 85 ms 85 ms 85 ms panel.dreamhost.com [208.97.187.201]
Red, you are able to reach the pannel however, the central Database is down due to this the panel is probably unavailable for you. But can you check if http://www.dreamhost.com works for you?
fRISKY FAIL server is still not fixed!? OMGWTFBBQ. My sites are still taking a minute to load a single page!
@Andrew Vit. At least they’re loading. Seriously WTF! this is getting out of hand.
my sites are ssssssssssssssssssssssssssllllllllllllllllllllllllllllloooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooowwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
Dreamhost’s reliability is horrible. Remember way back when when they used to talk about their 99.999999% uptime? I’m guessing they no longer do that.
It’s weird how whenever there are mail server issues, they never list Loonie as being affected, yet it always is. My email has been and still is hooched ever since this latest problem arose and has since been closed as “resolved”.
Thumbs up DH.
I didn’t even notice…. I don’t know why people are saying their reliability is horrible. I have only seen my website down once or twice. And when there is a problem they are very prompt about it and get it going fast. I have yet to see any other web host actually tell you when there is a problem or when they screwed up (see billing mistake)
Thanks
توبيكات
Hope to see Frisky cluster up and running soon. All my websites are down