DreamHost Status » Email Issue

Email server emergency maintenance tonight (janky,randy,postal,spunky)

jordan — Tue, 06 May 2008 22:26:05 +0000

About 30 minutes ago we had some major problems with one of our email load balancers that keeps email for janky,randy,spunky and postal clusters. Tonight at approximately 11:30 PM PST we will be doing some maintenance that may affect the performance / uptime of any customers that have email in these clusters. It is only expected to last about 10-15 minutes at most and we apologize for the short notice. This post will be updated as soon as the maintenance has completed.

UPDATE 11:30 PDT
Maintenance has completed all mail services should be up and running again. If you are still having problems please contact support.

Delay in quarantined junk mail delivery

andrea — Sun, 04 May 2008 06:47:39 +0000

We have noticed that messages that were quarantined by our junk filter are not getting delivered to the Junk Folder. We checked on the messages, and they are still on the servers. The reason they weren’t getting delivered was due to a connection problem to the mysql database that controls the Junk Folder. The problem is now fixed, but it will be a while before it all catches up, as mail has been backed up since last night. We apologize for all inconvenience this has caused. We’ll post updates on the progress of junk mail delivery as soon as we have them.

Update 05/04/2008 noon Pacific — Quarantined junk mail is again being delivered to the Junk Folder. Sorry about the delay.

Spunky cluster mail issues

andrea — Mon, 28 Apr 2008 21:49:23 +0000

If you’re on the spunky cluster (check the name of your mail cluster under Account Status in the upper right corner of the control panel), you may have been experiencing delays in sending and receiving mail for the past couple of hours. The mail database for the cluster is having issues, and it’s being worked on. Sorry about the inconvenience! Please check back for updates.

Update 3:30pm Pacific time — This is now resolved, and we’re continuing looking into the reason for the frequent backup mail database issues that have caused the problem. If you’re still seeing issues with your mail, it’s likely unrelated, so please let support know!

Central Frisky Server Down

mir — Fri, 25 Apr 2008 20:55:05 +0000

Customers on the Frisky cluster are experiencing a range of issues:

-slow site performance due to high loads
-slow data access times
-server crashes
-slow/unavailable mail

One of the file servers in the Frisky cluster has crashed - the admin team is working on getting it back up but due to the nature of the server in question it is effecting all servers that mount data from it (which includes many of the hosting servers and all of the mail servers in that cluster). We’ll update here once we have it fixed (if we cannot bring it back up within a reasonable amount of time we’ll see about removing the mounts so that it has less impact). While this is a terrible situation it has allowed us to pinpoint the cause of similar performance hiccups that we have been seeing with this cluster and should allow us to resolve those for our customers. We will update this post as soon as we have more information for you.

Note: if you would like to determine if you are on this cluster just go to your panel here:

https://panel.dreamhost.com/index.cgi?tree=domain.manage&

and click on the ‘DNS’ link for any domain hosted there and scroll to the bottom, the mail MX records will have the name like so:

mail MX 0 mx1.balanced.frisky.mail.dreamhost.com.
mail MX 0 mx2.balanced.frisky.mail.dreamhost.com.

UPDATE 16:23

We are currently moving the data drives from one file server to another. We are still a few hours from resolution on this issue. As soon as we have the drives moved over and the new hardware booting up correctly we will update this status post again.

UPDATE 17:26

This issue is now resolved. We have replaced the faulty hardware and moved your data drives over to the new file server. This should be completely resolved for you.

Looney Mail Issues

mir — Fri, 25 Apr 2008 14:33:04 +0000

I apologize for the late post but I wanted to make sure that we had a resolution for you before posting here. Last week I saw complaints about slowness and timeouts. With some really excellent feedback from customers I was able to pinpoint when the issues were occurring and keep an eye on it and with the help of the admin team we determined that it was due to a file server that needed a bit of offloading. We had the hardware available to add 4 TB of space to the cluster and are installing that today so we just need to remove some data from the troublesome file server and all will be well. This is expected to be finished within the next 7 days so by then mail should be working properly again (it may even be sooner than that depending on the point at which it will return to normal). This should also give the cluster enough space to be fine for the next few years (this is an older cluster which grows slowly as no new customers are being added) so you guys are in great shape for a long time to come!

Spunky / Spacey Mail Downtime

brians — Tue, 08 Apr 2008 23:10:55 +0000

We are currently experiencing problems with the process that handles scheduled updates. This has caused a disruption in mail service. This will account for the password errors, imap connection errors, etc. that are being generated when attempting to access your mailbox. Newly created user aren’t being setup either. Our admins are diligently working to resolved the issue. We sincerely apologize for any inconvenience this may have caused. Please check back here for future updates.

UPDATE::2008-04-09 10:57 AM PDT

Our Admins are still investigating the cause of the issue. We truly appreciate your continued patience.

Update 04/9/08 5pm pacific: The problem is now resolved. If you’re still having problem with your email address, it’s unrelated, and we would be happy to help. Please contact support with details through the control panel.

Spunky Slowness and Downtime

greg — Thu, 03 Apr 2008 21:20:21 +0000

Since the spunky cluster move some servers have been experiencing a higher than average load. This affected website load times, website downtime issues, webmail connections, email connections, and other services that reside on the individual servers in the spunky cluster. The cause has been traced to a cross connection saturation which links two areas of the data center. Capacity was doubled and the load on all spunky servers dropped back to normal levels. If you’re still experiencing problems please open a support ticket since this issue is now resolved.

Janky Cluster Filer Problems

greg — Sat, 22 Mar 2008 00:39:26 +0000

We’ve been working on an issue with the Drooler file server which is in the Janky cluster. We were able to correct some initial problems we saw that was causing problems such as high load, slow email, slow websites, sites timing out, and email connectivity issues. Some of you might have received an email saying the problem was fixed, and it was, however the symptoms have returned. We have our Admin Team currently working to resolve this issue as quickly as possible. Once this issue is corrected we’ll update this post with more information.

Servers that can be affected from this are listed below:

adonis alboz anton anwaltskanzlei apollo aries atlas azcuehnos baachus barefoottours batserver bogus charon charybdis chronos dione dionysus djm electra erebus erida fresh gladys gorgon helios hera hypnos icarus iris jaky janky-vserver01 janky-vserver02 janky-vserver03 janky-vserver04 janky-vserver06 janky-vserver07 janky-vserver08 janky-vserver09 janky-vserver10 janky-vserver11 jove kermit kiru kratos kztixo medusa mf1 mf2 midas morejewel morpheus mytimeofday nemesis ninja odysseus oriza osiris pan pegasus persephone phi privateserver rhea shyfry sisyphus somaticvision statler tartarus tethys thanatos thetis triton tyche vivenet warppipevps zeus

Update: Drooler is back up, and we will be monitoring the situation closely in case any further problems arise.

Update 4/8/08 4pm : We have added a new file server a couple of days ago and have another one on order. We continue spreading user data across the existing file servers, and are looking into possible user issue that are causing the servers to be unstable. Sorry about the inconvenience.

Problems with Janky cluster

andrea — Tue, 26 Feb 2008 00:14:58 +0000

If you’re hosting on our janky cluster (see under Account Status/Your Email Server in the control panel), you may have been experiencing slowness with your site or difficulties checking email this morning. The reason for this was that one of the file servers was having connection issues. We found and already disabled a customer that we believe was causing a huge strain, and are looking for others. The file server is looking much better, please try checking email of check your sites again.

Very sorry about the inconvenience this has been causing you. If you’re still seeing problems, it may be unrelated, so please let us know (with details) by contacting us at https://panel.dreamhost.com/index.cgi?tree=support.msg&

Update: 6:00pm (PST): These issues have been resolved for the last few hours, but we wanted to make sure before posting and update. All of the mail login and site speed tests we’ve run are going through fine and things seem back to normal. If you’re still having problems with any sites or email hosted on the ‘janky’ cluster though, please contact support right away.

Mail delay on junk filtered domains

andrea — Mon, 25 Feb 2008 20:47:52 +0000

Most of our customers using junk filters on their domain(s) are experiencing delay in receiving mail. This is affecting all mail clusters, but only the domains that have junk filtering enabled through the control panel. Mail is slowly going through, but it’s still very backed up, so it will take a while to catch up. We added more machines to alleviate the problem.

Very sorry about the inconvenience! We won’t mark this post resolved until the clog is all gone, so please check back for updates.

If your domain is having delay issues despite it not being filtered, please let us know at https://panel.dreamhost.com/index.cgi?tree=support.msg& . We’ll need details like the address affected, the full headers of the delayed message, or the sender and recipient and when the message was sent, so we can look into it for you.