Update - 9:55am PST: This issue should be resolved now. We’ve restarted the individual cluster mail services and things seem to be running smoothly now. We’ve tested several of the affected mailboxes and have verified that they are working. If you continue to have issues, it was likely not related to this incident and you should contact support for further assistance.

Servers that may be affected by this are:

absinthe achilles adams ak alondra altoid atlantic aviation babyruth badillo bandini barranca baseline bazooka beaudry bellflower betsybird beverly blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bundy butler butterfinger cahuenga centinela cerritos cheese cheeto cheezit chiclet cienega cognac colbert cookie crackerjack cupcake dingdong dorito drumstick dumdum fairfax federal fireballs firestone fletcher flower formosa frito fruitland fudge fundip garth garvey good gower grand granola grappa gumdrop gummi hillhurst hillmont hoho holt hope hyperion jalapa jefferson juniormint jurupa larchmont laurel lemonhead licorice lifesaver limbo-spunky4 look lucas mariposa mayo mazalien melrose milliken mills monroe moorpark nash necco nightlife nilla nothingless nougat nuno nutrageous oakdale obvious ocean ord overland oxnard paramount payday pernod pez piggy plenty popcorn pretzel ps4769 ps5117 ps6496 ps6519 rampart rangerxxx razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rum saltine sanborn schnapps scotch selma sepulveda shoutitlouder skittle skor skybar smores snicker soto sourpatch splashy spree stravinsky suburbview sugardaddy sunset syt taffy tampa tequila toffee tootsie truffle twinkie twix twizzler union vanowen vermouth vine virgil washington wayne western whittier whopper wilmington wilshire wiredgeek wmo york

UPDATE 6:57am Pacific: We are also adding Spunky email to the list of services that are down due to this, as some email accounts are hosted on that file server.

We have also discovered the cause for this downtime.  The shelves that hold this file server are redundantly powered, but it seems that both of the power supplies have taken a dive.  We are currently working to restore power to this shelf and hope to have that completed soon.  Check back here for more updates!

UPDATE 7:33am Pacific: We’re almost there!  We ended up replacing the shelf for this file server, as we discovered that one of the power strips died and the power supply connected to the hot strip was also dead!  So, this normally wouldn’t have caused an outage like this, save for the surprise dead power strip.  The file server is coming back online now and we’ll post the all clear here when we have completed some tests.

UPDATE 10:45am Pacific: The file server issues have been resolved and all services appear to be working correctly now. We are still however dealing with the newest status post about network issues we are dealing with.

http://www.dreamhoststatus.com/2009/06/08/emergency-network-outage/

So if you are noticing problems still it is likely related to that outage now. Sorry about the file server downtime. Typically our backup power supply would have saved us from this outage. This issue is resolved.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.

update: This issue appears to be resolved and only lasted for about 10-15 minutes.  We found that the ‘imapproxy’ service stopped running on a number of mail servers, as well as some issues with the SMTP service.  We re-initialized these services and they came back quickly.  If you are still having any issues, please feel free to contact Support and they will be happy to check!

What Does this mean to my website?
Likely Nothing! 99.8%+ of our customers websites use our sendmail Mail Transfer Agent (MTA) which will continue functioning as is . We are simply blocking connections for any software that uses its own MTA . If your site is using the mail() function via PHP, or any similar scripting language, you will not be affected. If you are using /usr/sbin/sendmail , you will not be affected.

So You are Blocking Port 25?
Sorta! We are blocking OUTBOUND access to port 25 by all scripts that are not our own MTA and are not connecting within our Network . Using an SMTP connection through an e-mail app or script is considered an INBOUND connection. As well if you are using AUTH SMTP to our Mail servers the connection will still go through. Nothing will change in that regard.

Why are you doing this?
Spam! Spam! SPAM! All too large a percentage of spam complaints and blocks we are seeing lately are due to spammers running their own MTAs from hacked accounts. These MTAs circumvent our existing anti-spam measures , which lead to the unfortunate result of blacklisting that affect the flow of HTTP server based mail for all customers that share the server.

I have legitimate reasons for running my own MTA or connecting to port 25 outbound!
We’d encourage you to start using our default sendmail setup. Otherwise we will not be adding this block to Private Servers, so upgrading is always an option if you are going to want to run your own mail service . If you have any questions about your use contact support.

Will my Contact Form, Wordpress, Drupal, PhpBB, ZenCart or other One-Click install be affected?
No. These and most common scripts use the mail() function using our default MTA . They will not be affected.

Update: May 11, 12:30pm: This issue should be resolved now. Unfortunately, we’re not quite sure what the cause was. We rebooted the mail servers to get a clean slate and they came back up running fine. We’ve been testing mailboxes that were having issues according to our customers and weren’t able to duplicate any errors. If you have further issues with this, please contact support, as it is likely something unique to your specific account. Again, we apologize for any inconvenience this caused you.

You can check which cluster you are on from the panel by clicking “Account status” in the upper right in the “Your Email Server” field. Please check here for updates.

– UPDATE Wed May 6 09:14:35 PDT 2009 –

The admins have corrected the problem and email service should be restored. We are running some more tests to verify that all is working normally again. We identified a problem where mail servers were not using all available authentification databases, they are now correctly setup which will improve reliability of email due to more redundancy. Our apologies for the problems this morning with email.

UPDATE: We apologize for the lack of updates on this issue. We have been closely monitoring the junk filter queues to make sure there are no remaining issues. We feel strongly that this issue is behind us, but if you continue to see any delays, please feel free to contact support!

Update 1:30pm PSTAccording to the admin working on this, most email addresses should be able to receive outside mail again without errors. We’re double checking right now and running mass configs on the mail servers to make sure we didn’t miss any. Those should finish by the end of the day. If you’re still having issues tomorrow morning, please do let us know, so we can look into it for you. Please provide details, so we can better assist you. We’ll update this post as soon as all email addresses are reconfigured.

Update 5:40pm PST We believe that all incoming mail is now operational; all updates have been completed. If you’re still having problems with mail, please submit a ticket through the web panel with as many details as possible, so we can look into it for you. Sorry again for the inconvenience this has caused earlier.

Update 8:37pm PST There were some users who may have been missed when the all clear came down the pipe a few hours ago. Just to be safe, our admin team is running the cleanup scripts again. This will get anyone still seeing bounces (of which there should only be a handful at the most) cleared up in a few more hours.

Update 9:54pm PST I’ve been given a proper “all clear” by the admin in charge of this cleanup process. If you’re still seeing issues, please contact technical support.

Please stay tuned for updates.

Update: 16:55 April 7th: First off we would like to apologize for the recent database errors we have been experiencing that cause your panel access to go down. We are getting all our heads together to try to resolve these central database issues once and for all. We have stabilized the central database machines now and we are working on finding the source of these issues. All services are up and running currently but we are watching the database closely. Once again we are really sorry about the problems this has caused you and we aim to have this repeat issue completely resolved as soon as possible.