Power problems affecting Gambino, Bruno and DreamhostPS machines

DreamhostPS affected:

ps10079 ps10688 ps11564 ps8662 ps8677 ps8700 ps8725 ps8871 ps8881 ps8929 ps9041 ps9144 ps9590 ps10080 ps10891 ps11652 ps8665 ps8678 ps8701 ps8726 ps8872 ps8891 ps8935 ps9045 ps9149 ps9674 ps10213 ps10902 ps8650 ps8666 ps8683 ps8703 ps8728 ps8873 ps8908 ps8960 ps9097 ps9161 ps9813 ps10222 ps11154 ps8659 ps8668 ps8684 ps8714 ps8773 ps8876 ps8920 ps8963 ps9135 ps9168 ps9840 ps10367 ps11165 ps8660 ps8674 ps8690 ps8715 ps8774 ps8878 ps8926 ps9034 ps9138 ps9580 ps10582 ps11174 ps8661 ps8675 ps8691 ps8724 ps8855 ps8880 ps8928 ps9040 ps9140 ps9585 ps10197 ps10366 ps10633 ps10870 ps11118 ps11614 ps9073 ps9091 ps9472 ps9681 ps9913 ps9932 ps10202 ps10368 ps10634 ps10871 ps11121 ps9054 ps9075 ps9093 ps9474 ps9687 ps9914 ps9938 ps10203 ps10375 ps10636 ps10872 ps11178 ps9059 ps9077 ps9096 ps9476 ps9702 ps9915 ps9963 ps10206 ps10385 ps10643 ps10886 ps11300 ps9061 ps9081 ps9437 ps9478 ps9710 ps9917 ps9991 ps10218 ps10386 ps10672 ps10894 ps11304 ps9063 ps9083 ps9441 ps9480 ps9738 ps9920 ps9999 ps10278 ps10387 ps10686 ps10924 ps11336 ps9065 ps9085 ps9445 ps9482 ps9741 ps9923ps10280 ps10388 ps10745 ps11066 ps11588 ps9067 ps9087 ps9446 ps9488 ps9744 ps9925 ps10335 ps10408 ps10869 ps11103 ps11611 ps9069 ps9089 ps9447 ps9490 ps9757 ps9927

Something tripped the circuit that these machines were on and the breaker had to be reset despite not being in any way overloaded. They’re back up now and we have a datacenter technician looking into the cause of it.

Comments: 11

Posted In: General Outages

Posted By: Leon

Web panel down

The web panel is experiencing loading problems right now. Our admins are working on getting it usable again. Please check back here for updates.

We apologize for all inconvenience this is causing.

UPDATE 12:30pm PST: We’re back, web panel is working again, thanks for your patience!

Comments: 11

Posted In: Control Panel Outages

Posted By: andrea

Personal backup server problems

The backup server is up and operational for most of our customers, but some of you may experience connection issues or unable to see or access data previously uploaded. We are in the process of replacing some hardware on one of the server’s volumes. Please check back here for updates.

We apologize for all inconvenience this is causing.

UPDATE 1:20pm PST: Data rebuilding on above mentioned disk volume is in progress. No data loss is expected, however, due to the large amount of data on it will take a long time to complete. Our current, very rough estimate is early next week. Unfortunately, that also means that access and uploading new data to that volume is suspended until then. Again, very sorry about the inconvenience. Please continue checking back for updates.

Please also note: your user may have been temporarily affected by the problem this morning, even though it’s not actually on the bad volume. Please check as most of you have data access again.

UPDATE 2009-06-22 4:30PM PST: Our admins are still diligently working towards repairing the faulty drive. The volume rebuild has not completed. We thought that it would be finished by the end of the day. Realistically the process will be completed by tomorrow. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for further updates.

UPDATE 2009-06-23 3:03PM PST: There was a hiccup on the fileserver that the backup machine is mounted to. This caused the drive rebuild to come to a halt. The rebuild is chugging along again. However, its only about 35% complete. Unfortunately its going to be another day or so before the process finishes. Again we apologize for the inconvenience and appreciate your patience. Please check back for future updates.

Comments: 8

Posted In: Single Server Issue

Posted By: andrea

Web server falco is having problems

The machine has been having I/O issues and it looks like it will need a new raid card. A technician is on his way to the datacenter to replace it. Check this post for updates.

We apologize for all the inconvenience this downtime is causing.

UPDATE: 6/19/2009 12:15pm (Pacific): Falco is back UP and running again. Please check your sites and let us know if you run into any problems.

Comments: 6

Posted In: Single Server Issue

Posted By: andrea

Dedicated hosting stability issues

A routine backup script is unfortunately generating high loads on some of our Dedicated Customer Servers, which is causing some of them to crash.  We sincerely apologize for this and we are actively addressing the issue, so we hope to have the stability issues worked out as soon as possible.

Servers possibly affected are:

aeroquest1 artdesign astro bacon bandr baycouplesdb bcimages bcmweb bcreports bean bender bensons billbiz blazer bloodhoundgang bmw boney bootifull brookeburke bscmanagement btia c100webserver cadet calgary calmini carlotta catering cbjdigital1 cgbny charleston clickmeister cobb cobblers codlib cornerhw cowboy creek dagon dante deimos destro devtvr dicta dissenter dixie dsl eightoneeight embassy esdraelon evangeline evolution01 factorycat falcor fepi foreplay freeco futuretrax gauche gazoline gendou gmlb1 gnu greysoft1 greyson groundcontrol hoodcore hutllan hypnopics idpfilm ifpserver ifthen inaux infoword jbeck kidalex killeroldies kittens kittenwar kmi kylie labcomp lifetimesvoice lincoln loki lowest-airfare luke matisse max mbserver meep motmot mqseries mushroomhead naish nassweb neversoft nwtekno obraxus odt oshea p5audio padrino pappocom pariso perfecto peterpan phobos picasso ping polaris principa prometheus reallifelab rlharvester saints sauron sax scudettoweb searchmeister secrets settlements shazam silversun singe sitepartner suba sunimage sunmachine swingdater tagrel tekhost telkoth thetinagrant timber toddmuchmore trackseven traveltracer utopia vantec vmachine wbcentral webwerks wireandtransit wlusp womble wpcf yaleweb zartan

We will update this post with more information as soon as we can!

UPDATE 1:32AM (Pacific): All services should be restored now, but please contact Support if you are still experiencing any downtime, or stability issues!

Comments: 11

Posted In: Multiple Server Issue

Posted By: patrickm

Saltine Moving to New Hardware

The web server, saltine, is being moved to new hardware as it is failing come back up from a reboot. Saltine should be back up within the next 30-45 minutes… Check back for more updates!

Update 4:53PM: Sorry that it took longer than expected but saltine is back up and happy!

- The Happy DreamHost Server Failover Robot

Comments: 11

Posted In: General Outages, Single Server Issue

Posted By: anthony

Falco is fsck’ing

The web-server Falco is currently down as errors were detected on its file-system bringing it into read-only mode. An fsck (file system check) is currently running on it to get it back to normal status. This typically takes around an hour but could be up to two.

Please check here for updates.

Update 11:40 PM PST: The fsck is still running strong. no ETA as there is really no way to know for sure how long a fsck will take. It is very rare for it to take over 2 hours though.

And I apologize for typo’ing the server name as ‘faclo’ as it is indeed ‘falco’ as one of the commenter’s mentioned.

Update 01:29 AM PST: Unfortunately this is taking longer than expected. Its on pass 1B which is rescanning for multiply-claimed blocks. I will update this again as soon as its completed.

Update 2:55 AM PST: Sorry for the delay guys! It looks like the fsck finally finished and the machine is comming back up now and so far appears to be good now. It looks like the fsck might have taken a while due to the large number of files on the machine. This machine appears to have 33 million inodes being used which is quite a bit more than our average server.

Comments: 21

Posted In: Single Server Issue

Posted By: Sandon

favicon.ico, favicon.gif and robots.txt update

In the interests of server stability, we have proactively started creating favicon.ico, favicon.gif, and robots.txt files in domain directories. We are not overwriting any of these files, only creating them should they not exist.

The reason we are doing this is due to software packages using mod_rewrite rules to redirect requests for non-existent files to a default handler (such as a php script, a rails application, etc), which is an acceptable and perfectly valid use of the mod_rewrite rules, however when such a site is missing files, and starts receiving a heavy load of traffic, it can severely adversely affect server performance and stability.

Most browsers request the favicon.ico file by default when making a request, and additionally some even ‘pre-load’ the favicon.ico file from linked sites when visiting other sites. Some browsers even look for a favicon.gif file, though this usage appears to have dropped off considerably, we decided we should probably go ahead and create these as well, just in case.

Additionally, well-behaved web crawlers will request the robots.txt file to see what they should or should not be indexing at the site operator’s request. While under normal circumstances they generally only request the file once, prior to indexing, we’ve seen behavior where a web crawler will repeatedly request the file, in itself causing server instability and load issues.

To attempt to alleviate these problems, we’ve started creating the favicon.ico, favicon.gif, and robots.txt files in all domains across our system. The files are blank, 0 byte files, and simply serve to prevent things like WordPress from causing extremely high loads and server instability in the event of a traffic surge to the site, simply because it’s doing a whole lot of unnecessary work to generate a 404 page, which itself isn’t useful to the browser either.

You can easily replace these files with those of your own creation simply by uploading a new one to the server. We only create them where they do not previously exist. If for some *crazy* reason, it decides to ‘create’ the files again anyways, we simply use the ‘touch’ command, which on an existing file will simply update the last modified timestamp, not affecting any of the contents of the file whatsoever.

Please keep in mind that the creation of these blank files is entirely harmless, and will only serve to speed things up on your site and system-wide. It is in *your* best interests that we create these files.

If you have any questions, please file a support ticket and we’ll be happy to address any of your concerns.

Sincerely,
The Happy DreamHost Server Stability Robot

Comments: 34

Posted In: General Outages

Posted By: kitchen

Ram Upgrades!

We will be performing quick RAM upgrades on some of our more loaded servers today at 4pm. Each server should only be down for 5 minutes or so. We will do everything possible to minimize the downtime and hopefully they should all come up performing a lot better! Only HTTP, SSH, and FTP should be affected by this maintenance.

The list of servers is:

greene
hulk
schultz
bonanno
castiglia
burke
fay
watanabe
gambino
barker

Please contact support if you have any questions.

Happy DreamHost Special-Talent Upgrade Team!

Update — These are all finished as of Tue Jun 9 17:30:56 PDT 2009

Comments: 9

Posted In: General Outages

Posted By: micah

Moving Persephone to New Hardware

On numerous occasions persephone has had problems related to it’s network interface card and we think that it’s high time to get everything over to a new piece of hardware before it tanks in the middle of the night.  Estimated downtime is 20-30 minutes.

UPDATE: The migration to new hardware is complete, if you are still experiencing issues please contact support via the panel.

Comments: 5

Posted In: General Outages, Network Outages

Posted By: Kyle