DreamHost Status » Network Outages

Moving Persephone to New Hardware

Kyle — Tue, 09 Jun 2009 19:25:24 +0000

On numerous occasions persephone has had problems related to it’s network interface card and we think that it’s high time to get everything over to a new piece of hardware before it tanks in the middle of the night. Estimated downtime is 20-30 minutes.

UPDATE: The migration to new hardware is complete, if you are still experiencing issues please contact support via the panel.

Emergency Network Outage

Patrick — Mon, 08 Jun 2009 17:16:23 +0000

One of our network links just went down due to failed hardware. We have replacement hardware on hand and are installing it now. During this installation much of our network will be inaccessible. This downtime should not exceed 10 minutes.

Check this space for updates.

– Update Mon Jun 8 11:09:52 PDT 2009 –

The hardware has been replaced and the network has been up and running properly for the past 30 minutes or so. There was some fallout due to the network being down, but that has also been cleaned up at this time.

If you are still having problems connecting to your sites, please contact support.

Exodus is down for the count.

Sandon — Sat, 06 Jun 2009 07:57:07 +0000

It looks like one of our web-server ‘exodus’ was unresponsive and had not came back up after we attempted to power-cycle it remotely. Right now an admin is en-route to the data center to check out the situation.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.

Network issues

Terri — Sat, 04 Apr 2009 01:49:03 +0000

We are currently experiencing networking issues. This is affecting some machines, as well as mysql services and oneclick installs. We are troubleshooting the issue and will fix it as soon as possible. We apologise for the inconvenience.

Update: Thankfully the network issue was resolved in a short period of time. There was a misconfiguration in our core router, and our network administrators patched it up.

Quick network cutover

Kelly — Thu, 12 Mar 2009 19:07:28 +0000

Tomorrow night at 8:00PM Pacific March 13, 2009 we will be doing a quick network cutover. We are moving our of some older racks in to newer purpose built networking space. The cutover should be nearly transparent, with at most a minute or two of downtime.

This affects all services for people in randy, frisky, swarthy, looney, blingy, and spacey clusters.

– Update –

The cutover went extremely well. All services are up and responding. If you are having any problems whatsoever, please contact support.

Randy cluster power failure

Ralph — Tue, 03 Mar 2009 17:30:13 +0000

The randy cluster, also home to the previous spoiler issue, is unfortunately having problems due to a failed power strip. Our administrators are actively working on this issue. More information should be posted soon. You can tell if you are in the randy cluster by looking at the ‘Email Server’ value in the ‘Account Status’ from the DreamHost web panel.

Update 9:35PST: Two circuits failed, the strips were not the problem, we are working with the datacenter staff to get them back online in an expedicious manner.

Update 10:05PST: Power has been restored to the failed circuits, service has been restored to the affected servers. If you are having trouble with your site still it is unrelated to the power outage, please contact support via the panel.

Update 11:00PST: We have evaluated this downtime and unfortunately this issue was not in our control. The problem was with three circuits that are managed by the data center.

Joker stabillity issues being worked on.

mir — Thu, 26 Feb 2009 17:33:26 +0000

I am terribly sorry for the issues customers have been having with this server, particularly when most of the users on it have been moved there in hopes of achieving improved performance. We’ve been investigating it but we haven’t properly posted our findings so that is the purpose of this message and we’ll update you here until we have things resolved.

So far we’ve looked for bad users (and have in fact removed a few which helped). Unfortunately that was not sufficient as there was more going on. As it turns out the machine was running out of file handles which was then causing everything to back up - one of our admins corrected this issue yesterday and everything was looking vastly improved but it appears that problems are back today. We’re waiting to hear whether this means there is another issue or if it’s just that the previous fix was not sufficient and needs extra attention. Once we have more details we will update this post.

Update: it did turn out to be some bad scripts under a pair of users that were causing the more recent load issues - we’re still keeping a close eye on it but so far the load is looking significantly better and the machine is more stable.

Update: while it’s remaining stable for longer periods of time we’re still seeing load spikes and continue to investigate.

Update: after stopping a few DoS attacks yesterday the server appears to be quite a bit more stable - the load is generally under 10 or topping out in the mid to high 20s (which on this machine is actually around 4-15 load as it has double the hardware of our older servers). That said it did spin out of control once today so we’re going to continue to watch it and clear out any lingering problems (things are vastly improved already though).

Panel offline tonight, 10pm PST

mikeh — Thu, 19 Feb 2009 01:56:53 +0000

We will be taking the panel offline (intentionally!) tonight, Wednesday February 18, starting at 10pm PST to attempt to address the database issues of the past two days. We expect to have it back online by 11:59pm PST.

Update: 11:05pm PST, all back online.

The modifications and upgrades went well. The debug level was also changed to provide more information in case of another crash. Hopefully there won’t be any further crashes, but if there are this should help us troubleshoot and resolve the problem more quickly.

Lira died

Sandon — Sat, 14 Feb 2009 14:16:52 +0000

It looks like the web-server ‘lira’ has died and is in some sort of reboot loop. It power’s for a second and then turns back off, then powers up again in a loop. I am moving this machine to new hardware now and it should be back up within 1-2 hours. I will update this post when it is back up and happy.

Update 08:05 AM PST: Ok, Lira is up and happy again on new hardware! The good news is that the old machine was a dated pentium 4 with 3 GB of ram and the new machine is a brand new quad core xeon with 4 GB of ram so it should be a significant upgrade for all the customers on lira. Please contact support if you are still having any issues accessing your site and you are on ‘lira’

Private MySQL Host Problems

Kyle — Wed, 11 Feb 2009 19:14:02 +0000

One of our Private MySQL host servers is having hardware problems this morning, we have a admin in route to move the machine to new hardware. Machines affected:

psmysql3621 psmysql3623 psmysql3833 psmysql4176 psmysql4223 psmysql4248 psmysql4285 psmysql4294 psmysql4299 psmysql4307 psmysql4603 psmysql5209 psmysql5229 psmysql5720 psmysql5954 psmysql6085 psmysql6099 psmysql6312 psmysql6314 psmysql6356 psmysql6603 psmysql6948 psmysql6994 psmysql7025 psmysql7028 psmysql7170

Update 13:41 PST: After a drive to our other data center, a drive swap and a bit of work to get the machine happy with the new hardware the host server is back up and fully functional. If you are still experiencing problems please contact our support team from the webpanel.