Networking issue at the Irvine datacenter (RESOLVED)

We’re experiencing connectivity issues at our Irvine datacenter. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update May 16th 1:39am PDT Tonight’s connectivity issues appeared to be a result of a drop out experienced by one of our transit providers. They are dispatching an investigation team to help assist us in our diagnosing the exact issue that occurred. We are not expecting any further problems, however we will continue to monitor the situation, as well as assist our transit provider on finding further details on this issue. We’re resolving this case for now, but if you have further issues, please contact support for assistance.

Previous updates available after the jump →

Comments: 32

Posted In: General Outages, Major Outage

Posted By: andrea

Transit issue between LAX and Alchemy datacenters (RESOLVED)

Currently, at this time we’re seeing network degradation between our LAX and Alchemy datacenters due to maintenance by our transit provider. We are investigating this issue and will have more info as it progresses. This issue should only affect a small amount of customers with mysql databases that are in opposite datacenters, or users using webmail, on our homiemail-sub5 mail cluster — Brian H

Update May 12th 2:10am PDT This issue should now be taken care of at this time. We were able to configure a different link with another provider, and restore service. We apologize for any issues this may have caused.

Previous updates available after the jump →

Comments: 36

Posted In: General Outages

Posted By: Brian

Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)

This is the second step in our DreamHost Network Improvement Series.

On Monday, we will be upgrading the Border Hardware at our Irvine Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4-hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, so your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 14th 10:08pm PDT: We started our maintenance at 10pm PDT, and experienced less than a minute of downtime. Our customer test domains are loading and are stable, and we are confident that this case can now considered to be resolved. Our Irvine datacenter now has new border hardware.

If you’re having issues with your sites, it’s likely unrelated, so please contact support with the details, so we can assist you.

Previous updates available after the jump →

Comments: 44

Posted In: Network Maintenance

Posted By: Ralph

Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)

This is the first step in our DreamHost Network Improvement Series.

On Wednesday we will be upgrading the Border Hardware at our LAX Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4 hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 10th 1:10am PDT: We are considering this maintenance resolved. Thank you for your patience as we worked through this first installment of our Network Improvement Series. We are excited about these changes and what they mean to our service and what they’ll mean to you, our customers, as well.

Previous updates available after the jump →

Comments: 79

Posted In: Network Maintenance

Posted By: Ralph

Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours

Previously this was to be done on Wednesday May 2nd, but we moved it to the 15th.

On Tuesday, May 15th, we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. The fiber ring is being tested to minimize errors and to ensure maximum throughput. We are giving ourselves a two-hour window starting at 10pm PDT Tuesday May 15th. Service affected will be web and email. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup, so this should only affect a small subset of customers. This maintenance should not take the full two hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.

The DreamHost panel is up and running again. Sorry about the inconvenience!

Previous updates available after the jump →

Comments: 155

Posted In: General Outages

Posted By: Ralph

Networking Issue at Irvine DataCenter (RESOLVED)

We have noticed connection issues in our Irvine datacenter. This is resulting in site and database downtime. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update April 28th 1:20pm PDT We apologize for the continued down time but our network engineers have reported progress and we believe we’ll be fully operational again shortly.

Update April 28th 1:42pm PDT We would like to report that all of the sites we have tested that were experiencing issues are back up and loading for us. We’re continuing to monitor things closely but it appears that this matter has been resolved. We will provide more details when we have been able to assimilate all of the feedback from our admin and networking team members.

Update May 2nd 2:20pm PDT While this issue has been fixed since our last update, we kept this post unresolved until we were able to get more information about the specifics of it. This issue was caused by another DoS attack. During the process of blocking the external origin of the attack, the core routers became unreachable and therefore blocked traffic to Irvine. We had to reboot the routers to solve the problem and return to a stable state. We have already planned on replacing these routers with much more powerful ones in the next couple of weeks to help avoid future issues like this.

Comments: 163

Posted In: General Outages

Posted By: danwy

Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)

On Wednesday May 2nd we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles based data centers.  The fiber ring is being tested to minimize errors and to ensure maximum throughput.  We are giving ourselves a 2 hour window starting at 10pm PDT Wednesday May 2nd.  Service affected will be strictly web content. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup so this should only affect a small subset of customers.  This maintenance should not take the full 2 hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 2nd 7pm PDT: This maintenance window has been cancelled and will be moved to Monday May 7th at 10pm. A new status post will be created to keep track of the outage.

Comments: 55

Posted In: General Outages

Posted By: Ralph

Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)

In our continued efforts to upgrade our network, we will be scheduling maintenance on a piece of our border network equipment on April 25th. The action being performed is a simple software upgrade which will improve resource utilization on the new Juniper router we installed. We anticipate the downtime itself to last no longer than 10 minutes; however, we will be on high alert for the 2 hours preceeding the window as well as during and after the upgrade to make sure there are no complications. We sincerely appreciate your patience and understanding as we strive to improve our service for you.

Update April 25th 10:54pm PDT We’re happy to report that this upgrade is done, and happened without any issues! All service should be working as expected. If you’re seeing problems with your domains, please contact support immediately. — Brian H

Update April 25th 10:21pm PDT First part of upgrade is complete. We are starting the final upgrade now, then one more reboot and we will be upgraded. We will continue to monitor after upgrade is complete.

Update April 25th 10:12pm PDT The reboot of the network equipment is happening right now.

Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

Update April 25th 7:15pm PDT Maintenance is scheduled to start tonight at 10pm. Please keep an eye on this post for updates once the maintenance starts.

Comments: 54

Posted In: General Outages

Posted By: Ralph

WordPress 3.3.2 update available for one-click installs.

WordPress 3.3.2 is now available for new installs and upgrades via our One-Click Installer system.

As this is a security release, it’s a recommended upgrade for all users who run WordPress.

Auto-upgrades have been queued for all customers who have them enabled, but will take some time to process.

You can check if you have automatic upgrades enabled (and upgrade manually if you prefer not to wait) from the One-Click Installer section of the DreamHost Panel:

* Log into the DreamHost Panel, and go to the One-Click Installer Section https://panel.dreamhost.com/?tree=goodies.installer
* Click on “Managed Installed Applications”
* Click on your domain, and a list of your installs will show
* Under the “Upgrade Action” column, you can view and modify the status of auto-upgrades for each install
* Click on the “Upgrade to 3.3.2″ button to the right of the install if you wish to upgrade right away

If you installed WordPress yourself, rather than using our One-Click Installer system, you can upgrade by logging into your WordPress Dashboard and clicking “Please update now” from the alert at the top of the screen.

You can see full release notes at WordPress.org:
http://wordpress.org/news/2012/04/wordpress-3-3-2/

Comments: 12

Posted In: DreamHost Features, One-Click Installer

Posted By: Brandon

Network Issue at Irvine DataCenter (RESOLVED)

We are noticing connection issues to our Irvine datacenter. Our network engineers and system administration team are working to restore connectivity.  We will update this post once we have more information.

Update Apr 17, 4:07pm PDT: The issue is now resolved, and lasted for about 20 minutes. Our network engineers and system administrators worked quickly to recover the network, and are continuing to work on any possible residual issues. We will make sure to keep you updated, as well as provide you with details regarding the root cause as soon as we have them. We sincerely appreciate your patience while we continue to get this resolved. -Seohee

Update Apr 17, 5:45pm PDT: We sincerely apologize for the network issues experienced at our Irvine datacenter. This particular incident was a routing table issue and was not associated with the previous outages we experienced at Irvine. This outage was smaller in scope and took down services for 20 minutes. The configuration issue that caused this took down 1 of 4 subnets used in Irvine. The fix is in place now and we will avoid any further problems due to this particular issue. We are continuing to work through significant network improvements and have made this our top priority.

Comments: 128

Posted In: General Outages

Posted By: Dan J