DreamHost Status » Network Outages

Randy cluster networking problems

andrea — Tue, 22 Apr 2008 23:17:50 +0000

We’re currently experiencing networking problems on our ‘randy’ cluster (check the Account Status on the right upper corner of your control panel to see if you’re affected by this outage - if your email server is listed as randy, you may be). This is affecting web, mail, and mysql connections as well. We’re looking into this, and will update you as soon as possible. Very sorry about the inconvenience.

Sorry for the delay in an update, the networking issue has been corrected and since this the web servers working normally again. We will continue to monitor the situation, but if you are still experiencing issues related to this please contact support with details.

Spunky Slowness and Downtime

greg — Thu, 03 Apr 2008 21:20:21 +0000

Since the spunky cluster move some servers have been experiencing a higher than average load. This affected website load times, website downtime issues, webmail connections, email connections, and other services that reside on the individual servers in the spunky cluster. The cause has been traced to a cross connection saturation which links two areas of the data center. Capacity was doubled and the load on all spunky servers dropped back to normal levels. If you’re still experiencing problems please open a support ticket since this issue is now resolved.

Networking problems in one of our datacenters

andrea — Wed, 19 Mar 2008 16:13:29 +0000

We are currently experiencing networking issues at one of our data centers, which is causing downtime on many clusters, and dedicated servers, affecting email and websites. Our networking admins are on the case, and will have this resolved as soon as possible.

Please check back here for updates. We apologize for all inconvenience this has caused.

Update: as of 10:30am Pacific Time, this outage is resolved. If you’re still seeing problems with your domains, please let support know, so we can look into it for you.

Network troubleshooting tonight.

jeremy — Thu, 31 Jan 2008 20:32:05 +0000

To help debug some issues we’ve been having with an Etherchannel trunk in one of our facilities, Cisco has asked us to bring one of the members of the trunk offline tonight at 9pm Pacific time. Even with the 1 member offline, we should have plenty of spare capacity to handle all of the traffic, so we’re expecting little to no impact on customer sites.

LAX data center connection issues

glen — Thu, 24 Jan 2008 04:08:16 +0000

We are currently experiencing connectivity issues to our LAX data center.

All servers in the following clusters would be involved:
Blingy, Frisky, Looney, Spacey, Swarthy

Updates to come.

Update: All Servers seem to actually be up, but there is heavy packet loss at the distribution switch making them seem down to the outside.

Update: Packet loss has stopped. Our admins do have significant network upgrades planned shortly as a long term fix connectivity issues like this. If you have any further downtime , let support know via our panel form.

Update: Spoke too soon, Packet Loss is back . Our admin team is working feverishly on this.

Update - Wed Jan 23 21:34:52 PST 2008 - After a reload the switch is working properly again. We are already in the process of a major upgrade to this particular part of the network to resolve this issue. The actual swap-out of equipment will happen as soon as possible. Until that time we will be taking steps to keep the current switch operating properly.

Quick network upgrade

mikeh — Mon, 14 Jan 2008 22:58:42 +0000

We need to perform a quick network upgrade right now to alleviate a saturating port between two of our datacenters and the internet. The outage should be intermittent, and only last around 1-2 minutes. Afterwards, DreamHost should be running faster, and with less packet loss!

Edit: This turned out to be a false alarm. The network upgrades didn’t cause any noticeable outages. Other outages you may have experienced during this time were a coincidence.

Janky email issues

andrea — Mon, 14 Jan 2008 21:38:22 +0000

Since earlier this morning, we’ve been experiencing slowness on the ‘janky’ cluster due to a file server problem (see your Account Status on the panel’s upper right corner to see if you’re affected). This mainly affects email access, but there are some high loads on web servers too. We’re working on resolving this as soon as possible, and appreciate your patience.

Update Mon Jan 14 15:11:45 PST 2008: The slowness with janky should be all better now.

LAX connectivity issues

kitchen — Sat, 12 Jan 2008 08:35:57 +0000

We are currently experiencing problems with connectivity at our LAX facility. This is affecting the following clusters:

blingy
frisky
spacey
swarthy
looney

as well as dedicated server customers.

We have our network engineers looking at the problem and hope to have a resolution shortly.

Update: things appear to be back to normal. We are still looking into the issue to try to figure out what caused the network outage.

Emergency maintenance window

Kelly — Fri, 11 Jan 2008 01:16:24 +0000

Tomorrow night, January 11, 2008 at 10:00PM Pacific time, we need to perform two maintenance tasks:

1. Reboot a switch in which one of its processes if hung. It is currently passing traffic, but various Layer 3 duties it performs are in a degraded mode. We have otherwise worked around the issue, but we need to get that process un-stuck. We will be using this opportunity to upgrade the switch to the latest software version in hopes that this will not happen in the future, as this doesn’t add any time to the reload.

This should not affect general site performance, however some internal functions will be affected. If you notice sub-optimal ssh times, or other DNS related issues while logged in to our servers, they should resume in about 5 minutes when the switch comes back online.

2. Update the firmware on the disk shelves attached to a filer that crashed today. Speaking with various technicians, the current version of firmware could have been the cause of the outage in frisky today. This will cause a similar outage while the shelves reload. It should take around 5-minutes. If your data is hosted on this filer, you will not have access to it for those 5 minutes. No email or other data shall be lost, and the new firmware should make it more stable.

We’re sorry about the short notice of this window. If either of these two problems becomes worse overnight, we will go ahead and apply the fixes immediately.

Updated: 10:06 Pm Pacific: All updates went smoothly. Total downtime for affected people was <5 minutes per update. Hopefully things will be smoother here on out!

Annoying Network Issues

Jason C — Sat, 15 Dec 2007 22:19:44 +0000

It appears as if we’re running into little flurries of bad connectivity on the network currently. Please know that we’re doing everything we can to level this back out. Hopefully it shouldn’t take too long to remedy, but we thought those of you seeing issues would appreciate the acknowledgement of a problem. As always we’ll let you know when things are taken care of.

Update: 6:52pm pacific: The network issues have been resolved. We had deployed some new networking hardware for the ‘blingy’ cluster and a couple of the ports were misconfigured, which caused random weird issues all around.