Archive for the 'General Outages' Category

Emergency OS patch on file server

Posted 24 minutes ago (May 9th, 2008 at 8:27 am PST)

Per an open ticket with Sun we need to apply two patches to one of our file servers. This is to hopefully fix a degraded zpool which will not finish a parity rebuild. There are exactly 78 users in the frisky cluster on this file server. This will bring your email and web services offline […]

DingDong and Pizarro down

Posted 13 hours, 37 minutes ago (May 8th, 2008 at 7:14 pm PST)

The HTTP servers DingDong and Pizarro are both currently unresponsive to our reboot efforts. We are working on getting manual reboots done in their respective data centers or evaluating whither moving to new hardware is necessary. Estimated downtime as long as 1 hour.
Update 7:44p
Pizarro is back up and seemingly stable on new hardware
DingDong is awaiting […]

Central database crash

Posted 1 day, 16 hours ago (May 7th, 2008 at 4:45 pm PST)

Our central database server crashed and restarted itself. It is currently replaying transaction logs and should be back in under an hour. This should not affect your websites, email, etc, but the user control panel (https://panel.dreamhost.com) and similar services are down until it comes online.
We are monitoring the situation and will report back where when […]

FTP problems (connection drops)

Posted 1 day, 23 hours ago (May 7th, 2008 at 9:49 am PST)

Some of our customers are experiencing problems related to their FTP service. This includes error messages while connecting or dropped connections. We’re looking into it, and will post an update as soon as we know more. Sorry about the inconvenience.
Please check back here for updates.
Update 5/7/08 11am Pacific — We have narrowed down […]

Webserver Hermes being moved to new hardware

Posted 1 day, 23 hours ago (May 7th, 2008 at 9:25 am PST)

Hermes crashed earlier and server isn’t coming back up when rebooted. It’s currently being migrated over to new hardware and should be back up shortly.
Sorry for the inconvenience this has caused.
UPDATE: Migration is complete. Your sites should be back up and running. Contact support if your sites are still down.

Email server emergency maintenance tonight (janky,randy,postal,spunky)

Posted 2 days, 17 hours ago (May 6th, 2008 at 3:26 pm PST)

About 30 minutes ago we had some major problems with one of our email load balancers that keeps email for janky,randy,spunky and postal clusters. Tonight at approximately 11:30 PM PST we will be doing some maintenance that may affect the performance / uptime of any customers that have email in these clusters. […]

Problems caused by apache service updates

Posted 2 days, 21 hours ago (May 6th, 2008 at 11:18 am PST)

We have noticed that any apache service updates initiated from the control panel are breaking the service. This includes anything that has to do with the domain’s web service, like adding a new domain, changing FTP users on domain, etc. This doesn’t just break the actual domain that initiated the change, but all domains on […]

Delay in quarantined junk mail delivery

Posted 5 days, 9 hours ago (May 3rd, 2008 at 11:47 pm PST)

We have noticed that messages that were quarantined by our junk filter are not getting delivered to the Junk Folder. We checked on the messages, and they are still on the servers. The reason they weren’t getting delivered was due to a connection problem to the mysql database that controls the Junk Folder. The problem […]

redhot getting new hardware

Posted 6 days, 20 hours ago (May 2nd, 2008 at 12:06 pm PST)

Redhot has had a hardware failure and needs to be replaced. We are currently working on moving it over to new hardware.
This should take about 30 minutes, and we will keep you posted as things progress!
Update: 1:06pm pacific: the failover has completed and services are up and running. If you experience any issues, […]

Mysql Maintenance

Posted 1 week ago (May 1st, 2008 at 9:47 pm PST)

These mysql servers will be experiencing some downtime tonight for maintenance:
jake
leo
midnight
snarf
Downtime is expected to be less than 30 minutes. It should happen shortly after midnight PDT.
–Update
The maintenance was preformed and the servers were back up within 15 minutes.