Update - 9:55am PST: This issue should be resolved now. We’ve restarted the individual cluster mail services and things seem to be running smoothly now. We’ve tested several of the affected mailboxes and have verified that they are working. If you continue to have issues, it was likely not related to this incident and you should contact support for further assistance.

Servers possibly affected are:

aeroquest1 artdesign astro bacon bandr baycouplesdb bcimages bcmweb bcreports bean bender bensons billbiz blazer bloodhoundgang bmw boney bootifull brookeburke bscmanagement btia c100webserver cadet calgary calmini carlotta catering cbjdigital1 cgbny charleston clickmeister cobb cobblers codlib cornerhw cowboy creek dagon dante deimos destro devtvr dicta dissenter dixie dsl eightoneeight embassy esdraelon evangeline evolution01 factorycat falcor fepi foreplay freeco futuretrax gauche gazoline gendou gmlb1 gnu greysoft1 greyson groundcontrol hoodcore hutllan hypnopics idpfilm ifpserver ifthen inaux infoword jbeck kidalex killeroldies kittens kittenwar kmi kylie labcomp lifetimesvoice lincoln loki lowest-airfare luke matisse max mbserver meep motmot mqseries mushroomhead naish nassweb neversoft nwtekno obraxus odt oshea p5audio padrino pappocom pariso perfecto peterpan phobos picasso ping polaris principa prometheus reallifelab rlharvester saints sauron sax scudettoweb searchmeister secrets settlements shazam silversun singe sitepartner suba sunimage sunmachine swingdater tagrel tekhost telkoth thetinagrant timber toddmuchmore trackseven traveltracer utopia vantec vmachine wbcentral webwerks wireandtransit wlusp womble wpcf yaleweb zartan

We will update this post with more information as soon as we can!

UPDATE 1:32AM (Pacific): All services should be restored now, but please contact Support if you are still experiencing any downtime, or stability issues!

Servers that may be affected by this are:

absinthe achilles adams ak alondra altoid atlantic aviation babyruth badillo bandini barranca baseline bazooka beaudry bellflower betsybird beverly blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bundy butler butterfinger cahuenga centinela cerritos cheese cheeto cheezit chiclet cienega cognac colbert cookie crackerjack cupcake dingdong dorito drumstick dumdum fairfax federal fireballs firestone fletcher flower formosa frito fruitland fudge fundip garth garvey good gower grand granola grappa gumdrop gummi hillhurst hillmont hoho holt hope hyperion jalapa jefferson juniormint jurupa larchmont laurel lemonhead licorice lifesaver limbo-spunky4 look lucas mariposa mayo mazalien melrose milliken mills monroe moorpark nash necco nightlife nilla nothingless nougat nuno nutrageous oakdale obvious ocean ord overland oxnard paramount payday pernod pez piggy plenty popcorn pretzel ps4769 ps5117 ps6496 ps6519 rampart rangerxxx razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rum saltine sanborn schnapps scotch selma sepulveda shoutitlouder skittle skor skybar smores snicker soto sourpatch splashy spree stravinsky suburbview sugardaddy sunset syt taffy tampa tequila toffee tootsie truffle twinkie twix twizzler union vanowen vermouth vine virgil washington wayne western whittier whopper wilmington wilshire wiredgeek wmo york

UPDATE 6:57am Pacific: We are also adding Spunky email to the list of services that are down due to this, as some email accounts are hosted on that file server.

We have also discovered the cause for this downtime.  The shelves that hold this file server are redundantly powered, but it seems that both of the power supplies have taken a dive.  We are currently working to restore power to this shelf and hope to have that completed soon.  Check back here for more updates!

UPDATE 7:33am Pacific: We’re almost there!  We ended up replacing the shelf for this file server, as we discovered that one of the power strips died and the power supply connected to the hot strip was also dead!  So, this normally wouldn’t have caused an outage like this, save for the surprise dead power strip.  The file server is coming back online now and we’ll post the all clear here when we have completed some tests.

UPDATE 10:45am Pacific: The file server issues have been resolved and all services appear to be working correctly now. We are still however dealing with the newest status post about network issues we are dealing with.

http://www.dreamhoststatus.com/2009/06/08/emergency-network-outage/

So if you are noticing problems still it is likely related to that outage now. Sorry about the file server downtime. Typically our backup power supply would have saved us from this outage. This issue is resolved.

UPDATE - 12:45pm PST: This problem should have been fixed for the last couple of hours, but we’ve been testing things out to make sure everything is back to normal. This issue was caused by a php upgrade to our homie server image that was done for security purposes and that caused the server apaches to become unresponsive until they were restarted. We scheduled the restarts as soon as we isolated the problem, but due to the amount of servers that needed this done it took a while to get them all updated. We’ve tested all the servers affected, along with a large pool of customer domains on these servers, and they seem to be working fine now. If you have any additional issues, it was likely not due to this incident and you should contact support for further assistance. We’re sorry for any inconvenience this caused you.

Update - 3:16pm PST: This also affected a few other servers as well - dio, deathlok and egon. The batman, deathlok and egon servers are on their way back up now, but we’re still working on dio. We hope to have that back up and running as soon as we can as well. We’ll resolve this post once we’re sure these servers are running smoothly again.

Update - 10:30pm PST: We’re still working on getting these machines back up and restored to their former glory. Egon should be completely restored now. Batman, deathlok and dio are back up and working, however, some data is still not there. We’re copying over that from backups now, so if you’re still missing data, hold tight, we should have that restored soon. Johnnyblaze seems to be the hardest hit, so we’re working feverishly to get this back in working order.

Update - 8:15pm PST May 1: Our recovery efforts are now complete and to those of you who have lost data we sincerely apologize and you should have a message in your inbox detailing any compensation due to lost data. If you are still having problems or if you did not receive direct contact from us about data you have lost please contact support immediately.

Update: Thankfully the network issue was resolved in a short period of time. There was a misconfiguration in our core router, and our network administrators patched it up.

Update: Sorry we didn’t include this earlier. Here’s a list of servers that may be experiencing issues:

barker bonanno castellano castiglia genovese mangano mcgurn rothstein scalise schultz spillane spilotro

Update: This issue was caused by a power strip failure in one of our server racks. We have the datacenter engineers on their way to the datacenter to resolve the issue. We don’t have an ETA on when things will be fixed, but we are doing all that we can to remedy this issue as soon as we possibly can.

Update: For the most part, this issue has now been resolved. The datacenter engineers were quick to respond and resolve the problem with the circuit breaker. There was a small handful of servers that weren’t booting up correctly, so we had to go through and manually reboot them. We have most of the services back up and running, but there are still a few PS servers that we’re still working on. We should have those all back up and running shortly. We’ll post one final update once we’re confident everything is resolved.

Update (1:39am): Alright, whew. Everything should be fine now. What actually happened was a circuit in the datacenter blew and it just so happened that the switches that your machines are connected to were out because of that. It took some time to get a tech from the datacenter in there to reset the breaker and then after that I had to go and manually check each machine. The DreamhostPS and PS Mysql customers had some networking issues that also slowed things down. I apologize for the downtime! If you’re still having trouble, please send a message to our support staff.

arkanoid bob boppity caesar deblume defender dinar dinero ellipsis euro fidget forint franc gage galaga gap go grizli guilder happy ikari ike intheroo ivan joust katamari klax korben kroner limbo-randy mark marvin nimitz obiekty24 orbital paperboy patton peso pound ps5500 ps5643 qbert quid replicated researchdirector rygar scagnetti schilling scipio shekel slaughter tetris ultima wasabi yen yoda zorg

Someone’s already been dispatched to the data center and we’ll get this back up and running as soon as we can.  The expected downtime is 30-45 minutes.  Sorry everybody!

UPDATE (10:49 am Pacific): Looks like some of the networking hardware has failed and needs to be replaced, so this is taking longer than expected.  Folks are working on this as quickly as they can and I’ll update this post as soon as there are any new developments.

UPDATE (1:42 pm Pacific): The hardware has been fixed. Right now, the drives are being checked for integrity before the machine can be booted back up. Unfortunately, this will take another few hours. Very sorry about the continued inconvenience. We’ll update this post as soon as the drive check is finished.

UPDATE (5:51 pm Pacific): Unfortunately, bad news: the file server came back up, but then it froze again. Our admins continue working on a resolution. We appreciate your patience and understanding.

UPDATE (10:38 pm Pacific): Things are looking up, we’ve got the file server back online and are fixing the mounts to the webservers now. There will be another update here when all is well.

UPDATE (1:40 am Pacific): Got the final word from the Admin team that this issue is now resolved!  If you are seeing any errors (such as “bad_httpd_conf”) on your websites, or any other issues, please contact Support and we’ll get those fixed up for you!

UPDATE (March 5th): Sorry about the extended problems some of you have been dealing with. If you are still noticing problems we will need to hear from you in order to get your site working for you. During the file server outage we had to lock our configuration files and exclude users with their data on the file server spoiler. Once spoiler was back up we removed these blocks and reconfigured our web servers. This fixed most of the problems but there were still others that had to be addressed directly. So if you still have problems with bad httpd conf or other issues please contact support for further assistance.

Update 05:53 AM PST: Hobgoblin is back up and serving content. The other 5 servers should be back up within 10-15 minutes. More updates to come.

Update 06:22 AM PST: All services on all the above servers should be restored. If you are still having issues accessing your site please contact support.

Here are the machines that could potentially be affected:

7up amp arizona bambi barqs boba cactuscooler calpico chai clamato codered crush crystallight dads dasani dietrite eggnog espresso fiji frappe fresca fuze gatorade geyser gingerale grapico hi-c hiball horchata inko jolt koolaid lipton martinellis milk mrpibb mug nehi nesbitt nestea niagara oj orangebang pellegrino pepsi perrier powerade propel punch refresco riptide rockstar rootbeer schweppes seagull seltzer silk simpledollar snapple sprite squirt sunkist sweetums tea tizer tropicana v8 worldtown yoohoo

This only affects web services for these machines.

Update 11:27pm: Unfortunately this is taking longer than anticipated. It looks like this may take another hour or two.

Update: 12:49 AM PST: The file-server is back up and serving data. Please contact support if you are still having issues accessing your site and you are on any of the above servers.