DreamHost Status

I am opening a new post for you guys - we did install an additional file server for your cluster but the hardware proved to be faulty and that’s why you haven’t seen the improvement that we had anticipated. We have our storage specialist at the data center today working on this and we are hoping that he’ll have it up and running after the visit. If that is the case we’ll only need a day or two to offload Clunker and have your mail back to full speed. I will update this post once I have more details.

Update Frida, May 3 2008

The new hardware is up and we are balancing the data so that should be done within the next couple of days and the mail issues you have been seeing will be resolved.

On the night of Tuesday May 6th we’ll be replacing one of the network switches that serves the following mysql servers:

toto
rover
dogbert
dino
beethoven
jaws
camille
bootsy
spacey-vsql01
frisky-vsql01

We hope to be able to do this one server at a time with only seconds of downtime for each, however we’re posting this in case that’s not possible and we need to bring all the servers down.

Update: This switch was successfully replaced last night.

Just a notice to let you know tomorrow we’ll be changing the charities we have up at our Give to Charity area, so if you want to donate to Kidd’s Kids or the MIND Research Institute and have it matched by us, step up now!

So far we’ve received 21 donations for a total of $491.11 to Kidd’s Kids and 12 donations for a total of $274.67 to MIND.

MORE!

We are currently experiencing problems with the mysql server “sanspoof” crashing. The issue is being looked into at this time, we apologize for any trouble and will update you as soon as there is more information regrading this problem.

UPDATE 23:22:43 PDT

It appears that the memory in this server is causing it to panic and crash. We are going to be replacing the memory and that should fix the problem. This update should only take about 20 minutes.

UPDATE 12:54:29 PDT

Problem has been resolved by replacing the bad memory. All mysql service on Sanspoof should have returned to normal. If you are still having problems please contact technical support.

If you’re on the spunky cluster (check the name of your mail cluster under Account Status in the upper right corner of the control panel), you may have been experiencing delays in sending and receiving mail for the past couple of hours. The mail database for the cluster is having issues, and it’s being worked on. Sorry about the inconvenience! Please check back for updates.

Update 3:30pm Pacific time — This is now resolved, and we’re continuing looking into the reason for the frequent backup mail database issues that have caused the problem. If you’re still seeing issues with your mail, it’s likely unrelated, so please let support know!

The web server flamenco didn’t come back after a multitude of reboots, so we are migrating it to new hardware. It should be back up within an hour. Sorry about the inconvenience this is causing you, and please check back here for updates.

Update: Some networking issues caused flamenco to be down for longer than expected, but we’ve got it going now. I apologize for the outage.

We are currently experiencing issues with the main file server for the blingy cluster. Our file server admins are on their way to the data center to look into the issue, and we are doing our best to get this fixed up and working again as soon as possible. This issue will cause websites and email slowness or unavailability until it is resolved. When we have further information or it is fixed we will post an update.

– Update Mon Apr 28 10:06:16 PDT 2008 –

The file server is back up and working, we are going around fixing up any web servers which are still having issues or needing a restart after the file server issues. The file server admins are continuing their investigation into the cause of the file server problem.

Update 4/28/08 12pm Pacific — All servers are back up and running (except for flamenco, see post above). Sorry again about the inconvenience this has caused you. If you’re still seeing problems with your web or email, please let support know as it’s likely unrelated.

We’re currently investigating an issue on looney that’s causing web and mail service to become spotty, and more likely, completely down. This effects the following servers:

apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston

We apologize for this, and should have things working soon!

UPDATE: As of 10:15PM PST, this should be fixed now. If your site is still down, please email support and let us know!

The snocap server has been plagued with some intermittent issues. We’re working to resolve this as fast as we can, and will hopefully have more information on this shortly. If you’re on the snocap server you could be affected by slow website performance, website showing as down, slow uploads and downloads, and trouble updating your dynamic websites.

Customers on the Frisky cluster are experiencing a range of issues:

-slow site performance due to high loads
-slow data access times
-server crashes
-slow/unavailable mail

One of the file servers in the Frisky cluster has crashed - the admin team is working on getting it back up but due to the nature of the server in question it is effecting all servers that mount data from it (which includes many of the hosting servers and all of the mail servers in that cluster). We’ll update here once we have it fixed (if we cannot bring it back up within a reasonable amount of time we’ll see about removing the mounts so that it has less impact). While this is a terrible situation it has allowed us to pinpoint the cause of similar performance hiccups that we have been seeing with this cluster and should allow us to resolve those for our customers. We will update this post as soon as we have more information for you.

Note: if you would like to determine if you are on this cluster just go to your panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and click on the ‘DNS’ link for any domain hosted there and scroll to the bottom, the mail MX records will have the name like so:

mail MX 0 mx1.balanced.frisky.mail.dreamhost.com.
mail MX 0 mx2.balanced.frisky.mail.dreamhost.com.

UPDATE 16:23

We are currently moving the data drives from one file server to another. We are still a few hours from resolution on this issue. As soon as we have the drives moved over and the new hardware booting up correctly we will update this status post again.

UPDATE 17:26

This issue is now resolved. We have replaced the faulty hardware and moved your data drives over to the new file server. This should be completely resolved for you.