DreamHost Status

Blingy Slowness Issues

mir — Sat, 10 May 2008 18:56:09 +0000

We’ve continued to offload the problematic file server and loads across the cluster have actually looked quite good (almost all servers under a load of 5, many around 2-3 and only one was ever at or above 10) so we had hoped that we could restore snapshot backups for users on that server. Unfortunately the result increased the disk usage again on that file server to the point that is has been displaying the same symptoms as before. The good news is that the fix is simply a matter of disabling the backup snapshots again and then dropping that data. We’ve already done the disabling and are in the process of dropping - this is resulting in some very inflated loads across the cluster but as soon as we have completed this I’ll be issuing soft reboots (easier on the hardware) that will fix the loads you’re seeing as well as adding an update here. This should eliminate the performance issues that have been reported (we’ll just have to get that particular file server even lower before we restore those snapshots again). My apologies for this and for the concerns that it has caused.

Update: The utilization is already down from 93% to 92% but this is multiple TB of data at issue so it may take a few hours to complete the dropping of the data. I’ll be keeping an eye on it and rebooting any servers that appear to need extra attention (changes like this often result in that extra step being needed to stabilize things). There will be another update once we’re satisfied that things are fixed.

Update: We’re now back down to 89% - I am still seeing load issues so we’re not out of the woods yet but once the process completes we’ll know if that did it or not (I will continue to update).

Update: Utilization is at 87% and loads have improved so I am updating the severity to medium but not marking it resolved as we still need to see the loads and performance back to what they were a week ago.

Failing over Milk

Sandon — Sat, 10 May 2008 15:06:21 +0000

The web-server ‘Milk’ is being failed-over due to bad hardware causing it to crash a couple minutes after it boots. It is being moved over to new hardware and the move should be completed within 30 minutes. This should only effect people with ‘milk’ as their webserver. You can find out if milk is your server by clicking “account status” in the panel and it would be listed as “Your web server”

Edit: Turns out this was a bad site on the server causing it to crash in strange ways. This site has been disabled and your sites should be back online!

Emergency OS patch on file server

Kelly — Fri, 09 May 2008 15:27:48 +0000

Per an open ticket with Sun we need to apply two patches to one of our file servers. This is to hopefully fix a degraded zpool which will not finish a parity rebuild. There are exactly 78 users in the frisky cluster on this file server. This will bring your email and web services offline for the time being. The patch should only require about 15 minutes of downtime. I apologize for doing this patch during peak hours, but we really need to get this data back up to full integrity.

Tech nerd details: The raid array is a raidz2 operating with one failed disk. It has been rebuilding off of a hot spare for a day or two and reset the rebuilding process itself after getting to 99%. We contacted Sun and after analyzing troubleshooting information believe a kernel + ZFS patch should resolve the problem. Fortunately this is a raidz2 so it can sustain a disk failure and still be fault tolerant.

Update: Well, one of the two patches was installed. It seems SunSolve is rejecting our service contract to download a specific patch in the dependency tree. We’ve updated our case with Sun. The system is back online and serving files!

DingDong and Pizarro down

glen — Fri, 09 May 2008 02:14:10 +0000

The HTTP servers DingDong and Pizarro are both currently unresponsive to our reboot efforts. We are working on getting manual reboots done in their respective data centers or evaluating whither moving to new hardware is necessary. Estimated downtime as long as 1 hour.

Update 7:44p
Pizarro is back up and seemingly stable on new hardware
DingDong is awaiting a tech to reach it’s data center still.

Update 8:15p

DingDong has been manually rebooted and is up. Tests of sites show them responding. If you have any further issues related to either of these 2 servers, please contact support through our panel.

Central database crash

Kelly — Wed, 07 May 2008 23:45:16 +0000

Our central database server crashed and restarted itself. It is currently replaying transaction logs and should be back in under an hour. This should not affect your websites, email, etc, but the user control panel (https://panel.dreamhost.com) and similar services are down until it comes online.

We are monitoring the situation and will report back where when it comes online!

Edit: And webmail! I forgot those two were tied together. Regular IMAP/POP3 email access should continue to work.

Edit: 5:28PM Pacific It looks like we’re back in business! The user control panel and webmail are working. We will continue to check the rest of our central services and update this if we find anything else still broken! If you are having problems, please contact technical support.

FTP problems (connection drops)

andrea — Wed, 07 May 2008 16:49:09 +0000

Some of our customers are experiencing problems related to their FTP service. This includes error messages while connecting or dropped connections. We’re looking into it, and will post an update as soon as we know more. Sorry about the inconvenience.

Please check back here for updates.

Update 5/7/08 11am Pacific — We have narrowed down the problem to only affecting customers using the following ISPs: AT&T/Sbcglobal, Comcast and RoadRunner. We have tested the following solution with the help of one of the affected customers, so this should work for all: You will need to change your ftp-only user to sftp to go around what we suspect to be bandwidth throttling affecting only FTP connections.

Edit your ftp-only user to be sftp in the control panel under Users/Manage Users ( https://panel.dreamhost.com/index.cgi?tree=users.users& ). This should schedule a user update, and as soon as the little clock next to your username in the control panel is gone, log in with that user using the SFTP settings on your upload client, and you should be able to upload the large files. If your current client doesn’t have SFTP connection, you will need to download another one.
Please note, if you’re already using a shell user to connect through ftp, then all you need to do is change to sftp in your client, you don’t need to modify your user.

If the above doesn’t make your problems go away, then it’s most likely unrelated. Please send support a message, so we can look into it for you.

Update 5/8/08 9am Pacific — Apparently Verizon and other ISPs have been added to the list. Even if your ISP is not listed, please try to change to sftp, and if that doesn’t work, let us know with as many details as possible.

Update 5/8/08 10:50am Pacific — We’re continuing to look into this problem to make sure it’s not something on our end that’s causing the connection drops. We have just tested a few more cases were the customer switching to SFTP didn’t make a difference, but so far we couldn’t replicate the problem, but we’re still checking, examining more data.
In the meantime, continue trying to switch to SFTP to try that way. We recommend that over FTP anyway for its security. If you’re using telnet, you should be switching to ssh instead.

Here’s a link to the article in our Wiki regarding the use of SFTP…
(http://wiki.dreamhost.com/SFTP)

Update 5/09/08 8:15pm Pacific — We’re still looking into this issue, and thought it was resolved for most users, however, that doesn’t seem to be the case. At this point, all tests we’ve run on users mentioning problems come back working 100%. We’re unable to verify the problem at all, other than the support that’s coming in about it. We do have some support personal on some of the effected networks, and will be doing tests.

We apologize for the delay in getting this fixed, however, it’s not reproducible on our end, at all, so that makes it even that much more difficult to track down. We’ll keep updating, the more info we have.

Update 5/10/08 7:45pm Pacific — Ok, this still seems to be happening, however, we may have found a way around things. We’ve tested on a known problem network, and the test went through correctly. Please try FTP again, as some head way in the issue has been made. If not, hold tight, we’re still working on the problem!

Webserver Hermes being moved to new hardware

chih — Wed, 07 May 2008 16:25:40 +0000

Hermes crashed earlier and server isn’t coming back up when rebooted. It’s currently being migrated over to new hardware and should be back up shortly.

Sorry for the inconvenience this has caused.

UPDATE: Migration is complete. Your sites should be back up and running. Contact support if your sites are still down.

Email server emergency maintenance tonight (janky,randy,postal,spunky)

jordan — Tue, 06 May 2008 22:26:05 +0000

About 30 minutes ago we had some major problems with one of our email load balancers that keeps email for janky,randy,spunky and postal clusters. Tonight at approximately 11:30 PM PST we will be doing some maintenance that may affect the performance / uptime of any customers that have email in these clusters. It is only expected to last about 10-15 minutes at most and we apologize for the short notice. This post will be updated as soon as the maintenance has completed.

UPDATE 11:30 PDT
Maintenance has completed all mail services should be up and running again. If you are still having problems please contact support.

Problems caused by apache service updates

andrea — Tue, 06 May 2008 18:18:21 +0000

We have noticed that any apache service updates initiated from the control panel are breaking the service. This includes anything that has to do with the domain’s web service, like adding a new domain, changing FTP users on domain, etc. This doesn’t just break the actual domain that initiated the change, but all domains on that apache service.

We are fixing this as we find them, and trying to catch up with the ones that broke a little while ago. Very sorry about the web downtime this has caused you, and please check back here for updates.

Uptdate 5/6/08 5pm Pacific: This issue is now resolved. Sorry about the delay, we have reconfigured all apache services for which the firs fix didn’t take. If you’re still having problems, then it’s probably unrelated. Please write to support with some details, and we’ll be happy to check into it for you.

Delay in quarantined junk mail delivery

andrea — Sun, 04 May 2008 06:47:39 +0000

We have noticed that messages that were quarantined by our junk filter are not getting delivered to the Junk Folder. We checked on the messages, and they are still on the servers. The reason they weren’t getting delivered was due to a connection problem to the mysql database that controls the Junk Folder. The problem is now fixed, but it will be a while before it all catches up, as mail has been backed up since last night. We apologize for all inconvenience this has caused. We’ll post updates on the progress of junk mail delivery as soon as we have them.

Update 05/04/2008 noon Pacific — Quarantined junk mail is again being delivered to the Junk Folder. Sorry about the delay.