Please check back here for updates.

Update 5/7/08 11am Pacific — We have narrowed down the problem to only affecting customers using the following ISPs: AT&T/Sbcglobal, Comcast and RoadRunner. We have tested the following solution with the help of one of the affected customers, so this should work for all: You will need to change your ftp-only user to sftp to go around what we suspect to be bandwidth throttling affecting only FTP connections.

Edit your ftp-only user to be sftp in the control panel under Users/Manage Users ( https://panel.dreamhost.com/index.cgi?tree=users.users& ). This should schedule a user update, and as soon as the little clock next to your username in the control panel is gone, log in with that user using the SFTP settings on your upload client, and you should be able to upload the large files. If your current client doesn’t have SFTP connection, you will need to download another one.
Please note, if you’re already using a shell user to connect through ftp, then all you need to do is change to sftp in your client, you don’t need to modify your user.

If the above doesn’t make your problems go away, then it’s most likely unrelated. Please send support a message, so we can look into it for you.

Update 5/8/08 9am Pacific — Apparently Verizon and other ISPs have been added to the list. Even if your ISP is not listed, please try to change to sftp, and if that doesn’t work, let us know with as many details as possible.

Update 5/8/08 10:50am Pacific — We’re continuing to look into this problem to make sure it’s not something on our end that’s causing the connection drops. We have just tested a few more cases were the customer switching to SFTP didn’t make a difference, but so far we couldn’t replicate the problem, but we’re still checking, examining more data.
In the meantime, continue trying to switch to SFTP to try that way. We recommend that over FTP anyway for its security. If you’re using telnet, you should be switching to ssh instead.

Here’s a link to the article in our Wiki regarding the use of SFTP…
(http://wiki.dreamhost.com/SFTP)

We are fixing this as we find them, and trying to catch up with the ones that broke a little while ago. Very sorry about the web downtime this has caused you, and please check back here for updates.

Uptdate 5/6/08 5pm Pacific: This issue is now resolved. Sorry about the delay, we have reconfigured all apache services for which the firs fix didn’t take. If you’re still having problems, then it’s probably unrelated. Please write to support with some details, and we’ll be happy to check into it for you.

Downtime is expected to be less than 30 minutes. It should happen shortly after midnight PDT.

–Update
The maintenance was preformed and the servers were back up within 15 minutes.

toto
rover
dogbert
dino
beethoven
jaws
camille
bootsy
spacey-vsql01
frisky-vsql01

We hope to be able to do this one server at a time with only seconds of downtime for each, however we’re posting this in case that’s not possible and we need to bring all the servers down.

Update: This switch was successfully replaced last night.

apok axl blanka brazil charm curly dhalsim egon guile limbo-looney moe slappy soy tank tofu vega winston

We apologize for this, and should have things working soon!

UPDATE: As of 10:15PM PST, this should be fixed now. If your site is still down, please email support and let us know!

-slow site performance due to high loads
-slow data access times
-server crashes
-slow/unavailable mail

One of the file servers in the Frisky cluster has crashed - the admin team is working on getting it back up but due to the nature of the server in question it is effecting all servers that mount data from it (which includes many of the hosting servers and all of the mail servers in that cluster). We’ll update here once we have it fixed (if we cannot bring it back up within a reasonable amount of time we’ll see about removing the mounts so that it has less impact). While this is a terrible situation it has allowed us to pinpoint the cause of similar performance hiccups that we have been seeing with this cluster and should allow us to resolve those for our customers. We will update this post as soon as we have more information for you.

Note: if you would like to determine if you are on this cluster just go to your panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and click on the ‘DNS’ link for any domain hosted there and scroll to the bottom, the mail MX records will have the name like so:

mail MX 0 mx1.balanced.frisky.mail.dreamhost.com.
mail MX 0 mx2.balanced.frisky.mail.dreamhost.com.

UPDATE 16:23

We are currently moving the data drives from one file server to another. We are still a few hours from resolution on this issue. As soon as we have the drives moved over and the new hardware booting up correctly we will update this status post again.

UPDATE 17:26

This issue is now resolved. We have replaced the faulty hardware and moved your data drives over to the new file server. This should be completely resolved for you.

Sorry for the delay in an update, the networking issue has been corrected and since this the web servers working normally again. We will continue to monitor the situation, but if you are still experiencing issues related to this please contact support with details.

UPDATE 3:01:AM PDT

The filer has finished recovering and all services are back up and running. We are working with the filer vendor to find the source of the crash to prevent any further outages.

Update 24/03/08 10am: We’re working on the file server again to alleviate the load that’s causing problems with web, mail and mysql services. Sorry about that.

Update 27/03/08: We are doing emergency data moves to quell the stem of problems recently caused by your file server. During these moves, your data may be inaccessible. We are moving as we can off as fast as possible. Very sorry about the continued inconvenience!

Update 27/03/08 This series of moves has finished. We are going to keep an eye on things to see how much it helped and may have to do more moves tonight and tomorrow morning to get everything working smoothly again. This post will be updated with more information as soon as possible.

Update 29/03/08

We are continuing to move data off of the problematic file server but it’s a bit of a catch-22 because customers on that machine are continuing to add data at a very high rate. It filled up this morning for a while causing device full errors as well as mail problems and issues serving websites (when these fill up it causes problems across the board). To explain in more detail, when we move data it does not immediately disappear (there is a ’snapshot’ created of the old data that remains in case there was a problem with the move - that ensures that we do not lose customer data but until the admin team can check the move to make sure it went through properly we cannot delete the old data). We just did some of that and have some breathing room again and of course more moves are still in progress but we are asking customer on this cluster to help us by holding up on any non-essential uploads of data for the next couple of days. As soon as we have a significant portion of the data removed the problematic file server will begin to function properly (and additional moves will go much more quickly and smoothly) but right now we’re having trouble moving data more quickly than it’s being added by people. If everyone could please limit uploads to absolutely essential data until we reach the turning point where everything is working this will be resolved much more quickly (in other words if for example you are setting up a repository of large files you’ll actually be better off waiting a couple of days and getting the all clear from us on this issue because you’ll be able to access that data reliably instead of cramming it on there now and slowing the recovery process).

In the meantime we’ll be doing everything we can to safely and quickly move data off and get things back to normal.

Added information: Some of the people recently moved to the new file server are seeing errors because the data did not get set up completely (loading the site will work but just show an empty index). The admin team has been running an rsync that will fully restore all data and should hopefully finish by 9 PM PST - once that is finished all site and email data will be available for those users.

Update 30/03/08

We’re still racing to keep ahead of new data being added so any help we can get on that front is greatly appreciated (we’re still asking for customers to limit uploads as much as possible to speed up the recovery process). Some customers who are being moved are seeing blank directories still but those are due to moves in progress and the data will be fully restored when those complete.

Update April 1, 2008

We seem to be ahead of the curve right now, we are moving data off the primary volume and on to a secondary one faster than new data is being uploaded. The volume hasn’t filled up completely in a few days. We are working closely with the technical support team to see how we can speed up the process further. Thank you for your patience.

Update, April 1, 2008

I apologize for the late update but we’ve been going over our options (while moving data of course). While we’re not seeing any real relief in terms of data uploads we do have some very large moves that are almost complete. Once those finish we can start deleting the data (for example one is around a half TB or around 500 GB which will be 4-5% of the total but it’s going to take until around Friday to delete it all, so we’re dealing with a ton of data). Tomorrow’s update should be earlier in the day and hopefully we’ll have some progress to reports from the large moves being complete.

Update, April 3, 2008

The data moves to other file servers has been running constantly, but last night and this morning some complications happened with the moves, requiring admin attention. To clear up some space there had to be a short interruption in file serving, this is now finished, space is available and the moves are continuing. The admins are fixing up the last of the web servers which were having issues after file serving was restored. Our apologies again for the continued issues.

Update, April 4, 2008

Today has been a pretty good day of progress. We were able to complete even more moves and free up more data from the file server. Moves have been going quicker and stability is dramatically improving. Monitoring of the servers and email in the blingy cluster today have shown a significant decrease in problems. Issues do still exist but the problem is noticeably getting better. We are also pleased to note that we have more storage that will be coming early next week. We believe that this will go a long way in helping us fix this major problem.

Update, April 4, 2008

Things are continuing to improve today - when I got in I was pleased to see that we had held firm and even gained a percent (the effected file server was down to 95% which is as low as I have seen it in the last week and since I have been working it has dropped to 94%). Performance should improve as we gain ground (this will speed up moving data off as well). This progress, along with the added storage space we are expecting early next week should hopefully allow us to restore service for our customers to normal.

5:45 PM PST : The moves we started just a while ago seem to be causing server problems, we’re looking into it and should have it resolved shortly (they were run just like the ones that had completed so we have to determine why these specifically caused an issue). Update: this resolved itself before we could detect the cause but we’re monitoring the situation to ensure that it’s not a recurring issue (we have no indication that it will be).

Update, April 8, 2008

Please see our other posting for details on the work we did on the effected file server:

http://www.dreamhoststatus.com/2008/04/06/30-min-blingy-downtime-tonight/

We are also continuing to offload data and are making good progress (it’s never as fast as we would like it to be of course). There’s excellent detail here in case you missed it:

http://blog.dreamhost.com/2008/04/07/another-anatomy/

which chronicles the situation and fills you in pretty much up to today. We’re seeing the data dip to 90% so we’re hoping to have it down in the 80’s by the end of the week (every percent we gain helps and as performance improves we can speed up the rate of moving but we’re still looking to hit critical mass where you get the proper level of performance).

Update, April 9, 2008

As we had hoped progress is speeding up as we free up more space - while the file server is showing 95% usage, around 11% of that is data that has already been moved and is no longer in use. Due to a software issue we haven’t been able to remove it yet (the admin team is working on the best way to execute that), but once that is gone we should be around 85% usage which is another large step forward.

In terms of effect, I have already seen improvement in site function for many customers as well as greatly increased speed in moving chunks of data off as well as receiving reports that mail is functioning quite a bit better. That said this issue remains at the High severity rating and in unresolved status as we have not reached a normal level of service. I can’t stress enough how sorry I am that our customers have had to put up with this but I thank those of you who have stuck with us (check the newsletter for details on what we’re doing for Blingy customers) and look forward to providing you with the level of service we strive for at DreamHost.

Update, April 9th, 2008 21:59 PDT

Unfortunately, we need to unmount the volume again to kill these snapshots before they leave us with 0 bytes of free space. In 2 hours (midnight) I will be taking the problematic volume offline to delete the phantom snapshots. Total downtime will be between 10 and 30 minutes. Sorry for the short notice and additional outage!

Update, April 10, 2008

Well the snapshot mentioned yesterday is gone and we’re actually at 83% used today which is below where we were hoping to see marked improvement (85%). Of course we’re still moving data off (which increases the usage on the file server) so that won’t fully translate to customer usage improvement but it should be quite a bit better and keep improving until we stop moving data.

Update, April 14, 2008

Okay, we’re finally getting ready to mark this as resolved.. things have seemed pretty much okay for a while now. But, just to be sure, we’re dropping the severity to Medium for now and leaving it as unresolved.

Update, April 17, 2008

We’re still hearing some reports of site slowness - we were able to resolve an issue causing high loads today which should help but we’re not going to consider this resolved until everyone is receiving good service.

Servers that can be affected from this are listed below:

adonis alboz anton anwaltskanzlei apollo aries atlas azcuehnos baachus barefoottours batserver bogus charon charybdis chronos dione dionysus djm electra erebus erida fresh gladys gorgon helios hera hypnos icarus iris jaky janky-vserver01 janky-vserver02 janky-vserver03 janky-vserver04 janky-vserver06 janky-vserver07 janky-vserver08 janky-vserver09 janky-vserver10 janky-vserver11 jove kermit kiru kratos kztixo medusa mf1 mf2 midas morejewel morpheus mytimeofday nemesis ninja odysseus oriza osiris pan pegasus persephone phi privateserver rhea shyfry sisyphus somaticvision statler tartarus tethys thanatos thetis triton tyche vivenet warppipevps zeus

Update: Drooler is back up, and we will be monitoring the situation closely in case any further problems arise.

Update 4/8/08 4pm : We have added a new file server a couple of days ago and have another one on order. We continue spreading user data across the existing file servers, and are looking into possible user issue that are causing the servers to be unstable. Sorry about the inconvenience.