DreamHost Status » Single Server Issue

Falco Getting New Raid Card

mir — Sat, 27 Jun 2009 20:22:22 +0000

Our admin team is replacing that hardware - it should be back up in the next half hour or so and we’ll update this post as soon as that is the case.

Thank you for your patience!

Update 1:49PM: Falco is now back up and all better!

Terranova getting a break.

mir — Wed, 24 Jun 2009 19:25:53 +0000

It looks like this server may not have an optimal distribution of usage (meaning it seem to be getting hit with a lot of usage at the same time which keeps spiking the load) so we are moving some customers off to help with that. Please be patient with this process as the moves will also add to the server load (but once we have completed the process over the next week or so the performance should improve and we’ll be watching to make sure that it does).

Thanks!

avalanche has fallen and can’t get up.

Sandon — Sat, 20 Jun 2009 09:22:14 +0000

It looks like the web-server ‘avalanche’ was unresponsive and did not come back up after being remotely power-cycled. We have dispatched someone to the data center to check out what the problem is.

Update: 02:39 AM PST: The issue has been resolved and the server is booting now. it should be complete back up within 10 minutes.

Update: 02:45 AM PST: Ok, All services on this machine should be 100% restored. If you are on avalanche and still having any issues please contact support.

Personal backup server problems

andrea — Fri, 19 Jun 2009 18:20:35 +0000

The backup server is up and operational for most of our customers, but some of you may experience connection issues or unable to see or access data previously uploaded. We are in the process of replacing some hardware on one of the server’s volumes. Please check back here for updates.

We apologize for all inconvenience this is causing.

UPDATE 1:20pm PST: Data rebuilding on above mentioned disk volume is in progress. No data loss is expected, however, due to the large amount of data on it will take a long time to complete. Our current, very rough estimate is early next week. Unfortunately, that also means that access and uploading new data to that volume is suspended until then. Again, very sorry about the inconvenience. Please continue checking back for updates.

Please also note: your user may have been temporarily affected by the problem this morning, even though it’s not actually on the bad volume. Please check as most of you have data access again.

UPDATE 2009-06-22 4:30PM PST: Our admins are still diligently working towards repairing the faulty drive. The volume rebuild has not completed. We thought that it would be finished by the end of the day. Realistically the process will be completed by tomorrow. We sincerely apologize for the inconvenience this has caused and appreciate your continued patience. Please check back for further updates.

UPDATE 2009-06-23 3:03PM PST: There was a hiccup on the fileserver that the backup machine is mounted to. This caused the drive rebuild to come to a halt. The rebuild is chugging along again. However, its only about 35% complete. Unfortunately its going to be another day or so before the process finishes. Again we apologize for the inconvenience and appreciate your patience. Please check back for future updates.

Web server falco is having problems

andrea — Fri, 19 Jun 2009 18:02:43 +0000

The machine has been having I/O issues and it looks like it will need a new raid card. A technician is on his way to the datacenter to replace it. Check this post for updates.

We apologize for all the inconvenience this downtime is causing.

UPDATE: 6/19/2009 12:15pm (Pacific): Falco is back UP and running again. Please check your sites and let us know if you run into any problems.

Saltine Moving to New Hardware

anthony — Tue, 16 Jun 2009 22:54:16 +0000

The web server, saltine, is being moved to new hardware as it is failing come back up from a reboot. Saltine should be back up within the next 30-45 minutes… Check back for more updates!

Update 4:53PM: Sorry that it took longer than expected but saltine is back up and happy!

- The Happy DreamHost Server Failover Robot

Falco is fsck’ing

Sandon — Tue, 16 Jun 2009 05:16:11 +0000

The web-server Falco is currently down as errors were detected on its file-system bringing it into read-only mode. An fsck (file system check) is currently running on it to get it back to normal status. This typically takes around an hour but could be up to two.

Please check here for updates.

Update 11:40 PM PST: The fsck is still running strong. no ETA as there is really no way to know for sure how long a fsck will take. It is very rare for it to take over 2 hours though.

And I apologize for typo’ing the server name as ‘faclo’ as it is indeed ‘falco’ as one of the commenter’s mentioned.

Update 01:29 AM PST: Unfortunately this is taking longer than expected. Its on pass 1B which is rescanning for multiply-claimed blocks. I will update this again as soon as its completed.

Update 2:55 AM PST: Sorry for the delay guys! It looks like the fsck finally finished and the machine is comming back up now and so far appears to be good now. It looks like the fsck might have taken a while due to the large number of files on the machine. This machine appears to have 33 million inodes being used which is quite a bit more than our average server.

Server “badillo” is having a bad day!

David — Mon, 08 Jun 2009 18:53:33 +0000

It looks like server “badillo” is having a bad day and needs new hardware. One of our admins is on the case and is moving all accounts on that server to new hardware as we speak. The whole process shouldn’t take too long. We’ll make another post once the process has been completed.

We apologize for any inconvenience this may cause.

UPDATE: 12:05 PDT - Unfortunately because there was also a problem with the spare server that was to be used another new server is being deployed as a replacement. This means the whole process could take about 1.5 hours to complete. We apologize for the longer than normal time it will take to restore services for your server.

UPDATE 1:01 P.M. PDT - Badillo is back up and running. Sorry for the inconvenience. If you have any further problems with this server, please contact support!

Exodus is down for the count.

Sandon — Sat, 06 Jun 2009 07:57:07 +0000

It looks like one of our web-server ‘exodus’ was unresponsive and had not came back up after we attempted to power-cycle it remotely. Right now an admin is en-route to the data center to check out the situation.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.

Tick down for maintenance

andrea — Thu, 04 Jun 2009 17:15:56 +0000

The web server ‘Tick’ is going down for maintenance. It should take less than an hour, but check back here for updates. Sorry about the inconvenience this may be causing.

Update 11:27am: Unfortunately, it turns out we need to migrate customer data to a more stable server, as even though Tick came back up and working, the same issues have resurfaced. Data is being transfered, and shouldn’t take longer than a day before all custromers on Tick are ready to go on the new machine. Tick is serving data for now, but let us know if you need assistance in the meantime. Again, sorry about all inconvenience this is causing.

Update 12:55pm: The data move has finished. If you used to be on Tick and now you are on Twofaced and have problems with your site or other services, please don’t hesitate to contact support with the details.