Servers that may be affected by this are:

absinthe achilles adams ak alondra altoid atlantic aviation babyruth badillo bandini barranca baseline bazooka beaudry bellflower betsybird beverly blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bundy butler butterfinger cahuenga centinela cerritos cheese cheeto cheezit chiclet cienega cognac colbert cookie crackerjack cupcake dingdong dorito drumstick dumdum fairfax federal fireballs firestone fletcher flower formosa frito fruitland fudge fundip garth garvey good gower grand granola grappa gumdrop gummi hillhurst hillmont hoho holt hope hyperion jalapa jefferson juniormint jurupa larchmont laurel lemonhead licorice lifesaver limbo-spunky4 look lucas mariposa mayo mazalien melrose milliken mills monroe moorpark nash necco nightlife nilla nothingless nougat nuno nutrageous oakdale obvious ocean ord overland oxnard paramount payday pernod pez piggy plenty popcorn pretzel ps4769 ps5117 ps6496 ps6519 rampart rangerxxx razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rum saltine sanborn schnapps scotch selma sepulveda shoutitlouder skittle skor skybar smores snicker soto sourpatch splashy spree stravinsky suburbview sugardaddy sunset syt taffy tampa tequila toffee tootsie truffle twinkie twix twizzler union vanowen vermouth vine virgil washington wayne western whittier whopper wilmington wilshire wiredgeek wmo york

UPDATE 6:57am Pacific: We are also adding Spunky email to the list of services that are down due to this, as some email accounts are hosted on that file server.

We have also discovered the cause for this downtime.  The shelves that hold this file server are redundantly powered, but it seems that both of the power supplies have taken a dive.  We are currently working to restore power to this shelf and hope to have that completed soon.  Check back here for more updates!

UPDATE 7:33am Pacific: We’re almost there!  We ended up replacing the shelf for this file server, as we discovered that one of the power strips died and the power supply connected to the hot strip was also dead!  So, this normally wouldn’t have caused an outage like this, save for the surprise dead power strip.  The file server is coming back online now and we’ll post the all clear here when we have completed some tests.

UPDATE 10:45am Pacific: The file server issues have been resolved and all services appear to be working correctly now. We are still however dealing with the newest status post about network issues we are dealing with.

http://www.dreamhoststatus.com/2009/06/08/emergency-network-outage/

So if you are noticing problems still it is likely related to that outage now. Sorry about the file server downtime. Typically our backup power supply would have saved us from this outage. This issue is resolved.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.

UPDATE - 12:45pm PST: This problem should have been fixed for the last couple of hours, but we’ve been testing things out to make sure everything is back to normal. This issue was caused by a php upgrade to our homie server image that was done for security purposes and that caused the server apaches to become unresponsive until they were restarted. We scheduled the restarts as soon as we isolated the problem, but due to the amount of servers that needed this done it took a while to get them all updated. We’ve tested all the servers affected, along with a large pool of customer domains on these servers, and they seem to be working fine now. If you have any additional issues, it was likely not due to this incident and you should contact support for further assistance. We’re sorry for any inconvenience this caused you.

Please stay tuned for updates.

Update: 16:55 April 7th: First off we would like to apologize for the recent database errors we have been experiencing that cause your panel access to go down. We are getting all our heads together to try to resolve these central database issues once and for all. We have stabilized the central database machines now and we are working on finding the source of these issues. All services are up and running currently but we are watching the database closely. Once again we are really sorry about the problems this has caused you and we aim to have this repeat issue completely resolved as soon as possible.

Update: Thankfully the network issue was resolved in a short period of time. There was a misconfiguration in our core router, and our network administrators patched it up.

So far we’ve looked for bad users (and have in fact removed a few which helped). Unfortunately that was not sufficient as there was more going on. As it turns out the machine was running out of file handles which was then causing everything to back up - one of our admins corrected this issue yesterday and everything was looking vastly improved but it appears that problems are back today. We’re waiting to hear whether this means there is another issue or if it’s just that the previous fix was not sufficient and needs extra attention. Once we have more details we will update this post.

Update: it did turn out to be some bad scripts under a pair of users that were causing the more recent load issues - we’re still keeping a close eye on it but so far the load is looking significantly better and the machine is more stable.

Update: while it’s remaining stable for longer periods of time we’re still seeing load spikes and continue to investigate.

Update: after stopping a few DoS attacks yesterday the server appears to be quite a bit more stable - the load is generally under 10 or topping out in the mid to high 20s (which on this machine is actually around 4-15 load as it has double the hardware of our older servers). That said it did spin out of control once today so we’re going to continue to watch it and clear out any lingering problems (things are vastly improved already though).

Update: 11:05pm PST, all back online.

The modifications and upgrades went well. The debug level was also changed to provide more information in case of another crash. Hopefully there won’t be any further crashes, but if there are this should help us troubleshoot and resolve the problem more quickly.

NOTE: This also effects access to our WebMail interfaces (SquirrelMail and RoundCube) and DreamHost Apps sites.

UPDATE: 11:28am PST It’s back up again! We’re keeping a close eye on the database server for a while though, just in case.

UPDATE: 12:16pm PST Looks like it’s down again. The admins are investigating. Stay tuned for more updates.

UPDATE: 12:40pm PST It’s UP again.

UPDATE: 12:57pm PST It’s down again. Admins are still investigating the cause of these outages.

UPDATE: 1:15pm PST It’s UP again.

UPDATE: 1:30pm PST Basically what’s happening is we’re 99% sure the memory (RAM) issue has been resolved. However the database process keeps dying without any useful information as to why. Fortunately however it seems to recover automatically in about 10 minutes or so without any data corruption (which is awesome). Our top database guru is on the case monitoring it very closely to try to figure it out and get it stable again. Again, we sincerely apologize for any inconvenience this is causing. We’re working as quickly as possible to get this problem resolved once and for all.

psmysql3621  psmysql3623  psmysql3833  psmysql4176  psmysql4223  psmysql4248  psmysql4285  psmysql4294  psmysql4299  psmysql4307  psmysql4603  psmysql5209  psmysql5229  psmysql5720  psmysql5954  psmysql6085  psmysql6099  psmysql6312  psmysql6314  psmysql6356  psmysql6603  psmysql6948  psmysql6994  psmysql7025  psmysql7028 psmysql7170

Update 13:41 PST: After a drive to our other data center, a drive swap and a bit of work to get the machine happy with the new hardware the host server is back up and fully functional.  If you are still experiencing problems please contact our support team from the webpanel.

Update 4:43pm: All set! If you’re still having a problem with pan, please contact a dreamhost support robot!