DreamHost Status

One of our DreamHostPS host machines has been having some hardware difficulty this evening and we’re going to be replacing it. The affected DreamHostPS machines are:

beachhead darklandz ezasamblea mp sixside vahati

We expect to have this resolved within the next 2 hours. We apologize for any inconvenience and thank you for your patience.

Update: 3:31am pacific: As you may have noticed, your sites are still online. We were able to get the old hardware up and running while we prepared the new hardware for failover. We will have to take them down for a few more minutes in a little while, but for the time being they’re up again (until we either take them down for the failover, or the hardware dies again!)

Update: 4:00am pacific: We have taken the old hardware offline and are currently moving things over to the new hardware. The DreamhostPS guests will be coming back up shortly. We will update again when things are all clear!

Update: 4:18am pacific: the hardware failover has been completed and the guests are all up and running just fine! If you have any issues, please contact support!

We are fully aware of the authentication issues that you are experiencing when accessing your Mailbox via WebMail. There appears to be an issues with one of our central databases. I have an admin looking into the issue at the moment. We hope to have this issue resolved shortly. Sorry for any inconvenience this has caused. Please check back here for further updates.

–UPDATE–Fri Aug 1 10:51:59 PDT 2008 (17:51:59 GMT)
We’re still working to resolve the issues with webmail. Currently we recommend all webmail users give roundcube.dreamhost.com a try. Roundcube is our beta alternative to SquirrelMail. The initial login to roundcube just requires you to input your entire email address and then your password. Don’t be confused by the @dreamhost.com portion of the user login. As a matter of fact, pretend it’s not even there! Simply enter:

Username: user@domain.com
Password: yourpassword

Sorry for the inconvenience and we will have SquirrelMail working properly as soon as possible.

UDPATE: The problems with WebMail have been resolved. One of the WebMail machines was deleting sessions. The machine has been fixed. I’ve tested user accounts extensively as well. Again, we sincerely apologize for any inconvenience this may have caused. If you continue to have problems with webmail access, please submit a support ticket.

The web server dalitz crashed a little bit ago, and has been checking its filesystem since. It should be back up inside of an hour. We apologize for this inconvenience, these filesystems take about that long to check.

Update 12:17AM Pacific (Friday): Looks like Dalitz has come back up and is running normally.

A major security update has been released by the Pligg team today and it is suggested any and all customers who utilize our one-click install of Pligg to upgrade immediately.  That can be done from the One-Click Installs sub-section of our web control panel under the Goodies section.  If you happen to see any issues with the upgrade, please contact our support department right away.

We are currently having issues with our ns2.dreamhost.com server. It is not responding to queries and it may cause issues if you’re trying to point a new domain to our nameservers. After a quick check, we believe this may be due to a bad drive in the server. Our admin team is currently working on this situation and hope to have it resolved as soon as possible. We will update this post when we have more information or the issue is resolved. We’re sorry for any inconvenience this causes you.

Update: This issue has been resolved.  Basically, the nameserver had an issue with database replication and needed a bit of TLC from our admin team to get it working again.  It is now accepting queries and any issues caused by this should now be taken care of.

This will explain any problems that you’re encountering when attempting to access your website, ftp server, etc. I have notified an admin and they are currently looking into this matter. We truly apologize for any inconvenience this has caused. Please check back here for future updates.

UPDATE: Capone is back online. Fscking a 4TB volume prolonged the reboot. Again, we apologize for the inconvenience the downtime caused and truly appreciate your patience. If you continue to experience problems, please submit a support request.

Our Admins are diligently working towards getting the machine back on line. Sorry for the inconvenience this may have caused. Please check back here for future updates.

Cheezit is live on the new chassis, contact support if you have any lingering issues

This is directly related to the network turn-up we did late last night referenced here:

http://www.dreamhoststatus.com/2008/07/30/network-maintenance-tonight-0730/

It appears that the IP access lists were not matching with GlobalCrossing and ourselves, causing a subset of our users whom routed in via GlobalCrossing to a specific subset of our IP’s to not be able to reach DreamHost. The GlobalCrossing link has been shut down until we can rectify this with their BGP team. All traffic is flowing over our other internet connection for now. We apologize for not catching this sooner, we simply did not see any support volume related until people began to wake up about 30 minutes ago!

Update (Minutes later!): I just got off the phone with GlobalCrossing and it seems that some interaction between our two Cisco routers was causing some form of CPU spike on both sides. We are both in the process of opening cases with Cisco support to figure out what was causing this problem. The plot thickens! We will be leaving these uplinks offline until this bug can be resolved.

Some customers on the spunky cluster might be seeing some outage issues. One of our network file servers is acting up and members of our admin team are scrambling to get this matter resolved quickly.

Possibly affected servers are as follows: admiralty aero alameda alondra archibald arrow atlantic avalon aviation babyruth barry bazooka beaudry bellflower betsybird blowpop bonbon bottlecap boyd broadway burton butler catalina centinela century cerritos chex cognac compton cookie cordial cupcake danielof dingdong doheny dorito drumstick dumdum edendale euclid fairfax fireballs fletcher foothill fundip garland garth garvey gobstopper goober good gower grand grappa gumdrop haven hillhurst hillmont hoover hope hops hyperion knocker lankershim larchmont lemonhead limbo-spunky4 linq locust look lucas ludlow lydia malt manchester mayo melrose milliken mills monroe munchies nilla ninja nix oakdale ocean olive ord overland pernod pico pikespeak plenty pocky poprock pretzel price pringle razzle ritz rockyroad rodeo roscoe rossmore rum sake saltine schnapps selma skittle skybar slauson snicker soto sourpatch spunky-vserver03 spunky-vserver16 spunky-vserver19 spunky-vserver27 spunky-vserver39 starburst sunset taffy tampa temple tequila topanga twix vanowen warhead washington wayne western whittier whopper wilshire york

We’ll keep you updated as the matter persists.

Update: Everything is happy again! It appears that the file server temporarily lost its routes. Our admin team is cleaning up a few straggling services and sites should be up again for the most part. If you happen to notice any further problems, please contact Technical Support immediately and we’ll gladly help you out.

Update: Unfortunately we have to repair a disk on the same file server which we are doing now - we’re hoping this will take no longer than an hour but we’ll update this post again when it is resolved.

Update: Good news - the repair has already completed so services are coming back up. In addition to that the admin team is offloading this file server so that it will no longer cause problems.

Tonight, around 10pm, we’ll be plugging in 5 new network links to expand our bandwidth by 50%. We don’t expect any downtime to be incurred but we’ll have our Cisco consultant on-site in case anything happens.

Update 11:53pm Pacific: We turned on the links, and we seem to be cooking along. The remote router hiccuped once, but we have been stable for over an hour now.

Update 05:41am Pacific: http://www.dreamhoststatus.com/2008/07/31/network-blip-for-some-customers/ Seems it did not go as well as it first appeared.