Problem with central services (panel,webmail, etc..) and spunky cluster
We are currently experiencing problems with one of our internal firewall machines that is causing problems with our central services connecting to our customer database. This problem is expected to be fixed within 20 minutes, we apologize for the trouble and will update this post as soon as it is resolved or more information becomes available as to the cause.
UPDATE 2:40 PM PDT **
Problems with central services have been resolved. Panel , webmail and other central services should be working as usual. There are still a few problems with the spunky clusters that may be causing outages for customers websites/ftp. These problems are also being fixed at this time and this post will be updated when those are fixed.
UPDATE 4:39 PM PDT **
All of the networking problems and spunky cluster problems have been resolved. We apologize again for the outage and if you are still having trouble feel free to contact support with the service you are having problems with.
UPDATE 5:11 PM PDT **
There have been reports of problems with the spunky cluster , we will re-open this status post until we have more information to what caused this problem. If you are on the spunky cluster and are experiencing downtime it is still due to the networking issue.
March 25th, 2008 at 2:38 pm
Looks like a serious filer problems. Check out this “top” snapshot:
top - 14:04:05 up 12 days, 14:43, 9 users, load average: 90.08, 115.03, 88.18
Tasks: 3 total, 1 running, 2 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.5% us, 16.5% sy, 0.7% ni, 0.0% id, 65.8% wa, 0.2% hi, 0.3% si
Mem: 4144552k total, 3817624k used, 326928k free, 49424k buffers
Swap: 6313512k total, 10648k used, 6302864k free, 712924k cached
Load average of 115!!! And the CPU wait states go anywhere between 65% (as seen above) and 99%!!! That’s nuts.
That’s what’s killing performance right now. The CPU is spinning idle while it waits for IO.
I appreciate that I’m getting a pretty good value for money when it comes to disk quota and bandwidth quota but these sort of outages are not really acceptable. I hope that the smart folks at DH pump some more money into their hardware infrastructure.
March 25th, 2008 at 2:38 pm
my sites are still down, anyone else?
March 25th, 2008 at 2:38 pm
site email is up, but still can’t ftp
March 25th, 2008 at 2:39 pm
Spunky sites are still down…
March 25th, 2008 at 2:39 pm
well good to see im not the only one, boy I hate losing money
March 25th, 2008 at 2:40 pm
Hey there, any news on a powerbar status report?
March 25th, 2008 at 2:40 pm
love the fact this problem will be fixed in 20 minutes… Posted 1 hour, 26 minutes ago
March 25th, 2008 at 2:40 pm
My site has been down for 4 hours now. You guys really suck. Thanks a lot.
March 25th, 2008 at 2:40 pm
dreamhost is toast.
March 25th, 2008 at 2:41 pm
i only update/upload 1 time ever 2-4 weeks.. and it seems almost all of the time I try there is a problem!! LOL
March 25th, 2008 at 2:41 pm
No pop mail for last hour outgoing on the east coast….incoming works fine thus far..Virginia
March 25th, 2008 at 2:42 pm
I started having problems with input/output errors on gir on Saturday night. Everything has been very intermittent since then. So you people with a few hours of downtime don’t have a lot to bitch about (comparatively)
March 25th, 2008 at 2:43 pm
Goddammit Dreamhost.
March 25th, 2008 at 2:43 pm
Oh joy.. spunky cluster is having problems. Why would it be on any other cluster.. I am hosted on spunky. :p
Thanks for your hard work DH Team.
March 25th, 2008 at 2:44 pm
Having trouble both sending and receiving mail. When mail does receive it is sending duplicates of old emails back (like 50 at a time) to me along with some new. Anyone else having the same problem?
March 25th, 2008 at 2:44 pm
Nice job Dreamhost. You shall not be recieving anymore money from me.
March 25th, 2008 at 2:45 pm
All my sites are on the same cluster and aren’t up. Don’t think its spunky. Never see that till now.
March 25th, 2008 at 2:46 pm
Now it says
Site Temporarily Unavailable
We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”
Sounds like a bad case of the mushrooms to me.
March 25th, 2008 at 2:47 pm
Dreamhost sucks.
March 25th, 2008 at 2:47 pm
still not alive!! Grr… 1h 35 minutes downtime…..
March 25th, 2008 at 2:47 pm
Can you tell us when this will be resolved?
If you can at leaset give us a time frame until this mess gets fixed-
March 25th, 2008 at 2:47 pm
“webmail and other central services should be working as usual”
Yet my email is still down….
March 25th, 2008 at 2:48 pm
Back up to full strength and speed. Sheesh.
Matt
March 25th, 2008 at 2:48 pm
My sites aren’t working right now 5:48PM EST. Anyone else still having this problem?
March 25th, 2008 at 2:49 pm
Spunky always seems to get shat on.
I’ve been having issues ever since around noon, and now there’s some sort of “error id: “bad_httpd_conf”" error.
March 25th, 2008 at 2:49 pm
Lol, they say that the panel is working well who gives a fuck about the panel, that’s not what my website is you douche bags, my website is the website, not things on your end. Don’t bother updating with ‘Our business is safe, now on to yours, idiots.’ we really don’t care about how you guys are doing we want OUR sites to work.
March 25th, 2008 at 2:49 pm
I was defending DH when they took the spunky cluster down at short notice to move it. Now it’s starting to get a bit annoying. Ah well, I guess I’m getting what I paid for…
March 25th, 2008 at 2:50 pm
She can’t take much more-a this, Captain! She’s gonna blow! Abandon ship!
March 25th, 2008 at 2:50 pm
PAL: ftp now work or sites 4:50 central
March 25th, 2008 at 2:50 pm
well, I invite all who are currently very unhappy with the service over the past WEEK+, to register a complaint with the Los Angeles Better Business Bureau. www.labbb.org
March 25th, 2008 at 2:50 pm
Moe, no kidding. Get the websites back up FIRST and THEN fix the panel.
March 25th, 2008 at 2:52 pm
Yeah, and all my sites are down as well… grrr.
March 25th, 2008 at 2:52 pm
Up now! w00t!
March 25th, 2008 at 2:52 pm
I’m on Shocktart and it’s boned, too.
Network Error (tcp_error)
A communication error occurred: “Connection refused”
The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.
For assistance, contact your network support team.
March 25th, 2008 at 2:53 pm
Poor timing, Trying to migrate a site. I can’t do business this way. This is not the first time that I have noticed problems with the spunky cluster.
March 25th, 2008 at 2:53 pm
All of my sites are still down…
*Sigh* Guess it’s time to look elsewhere.
March 25th, 2008 at 2:53 pm
Get what you pay for. This is what happens when you shop at the dollar store.
March 25th, 2008 at 2:53 pm
this sucks!!! im trying to upload pictures of an expensive item for a client!!!!GRRRR.. oh well its only money
March 25th, 2008 at 2:53 pm
http://www.labbb.org/BBBWeb/Forms/Business/CompanyReportPage_Expository.aspx?CompanyID=13131294
March 25th, 2008 at 2:54 pm
I jumped the gun. Sorry folks. One of my sites is up.. but all of the others still down.
March 25th, 2008 at 2:54 pm
Back up, yay! :).
March 25th, 2008 at 2:54 pm
this is what my website says i don’t know about it maybe it’s a different problem
this morning it worked fine then stopped working at all now i see this
Site Temporarily Unavailable
We apologize for the inconvenience. Please contact the webmaster/ tech support immediately to have them rectify this.
error id: “bad_httpd_conf”
March 25th, 2008 at 2:54 pm
my web is online again http://santos.invazores.org/
March 25th, 2008 at 2:55 pm
andy: I totally agree. You buy a $500 car, dont expect to drive it cross county right?
March 25th, 2008 at 2:56 pm
Someone explain why problems with their central services can affect the spunky cluster?
We had slow sites before the move, the weekend we were down and now this. Like a foll I have something like 70 domains hosted here and yet I still don’t even get replies to tickets after 24 hours!
March 25th, 2008 at 2:56 pm
My account is now closed. Fortunately, I already had all of my files stored elsewhere so I don’t need to wait for Dreamhost to “find” them.
March 25th, 2008 at 2:56 pm
My site is not responding, or if I chose one of the subdomains I get the “bad_httpd_conf” error. Still down for me
March 25th, 2008 at 2:57 pm
Well, I found a lot of interesting info about dreamhost at this site: http://tinyurl.com/yptmlp
March 25th, 2008 at 2:57 pm
I am not sure how many web sites have been down for now 1h40 at least. All I know is that the 8 web sites I manage have been down that long.
A shame really that dreamhost reports the severity of the problem as medium.
Based on 340+ user reactions to the outage in 1h40, I would be curious to know at what point a problem becomes critical to dreamhost.
March 25th, 2008 at 2:57 pm
When this will be finished???? You said that the problem has been resolved but my server still doesn’t works….