Update May 16th 1:39am PDT Tonight’s connectivity issues appeared to be a result of a drop out experienced by one of our transit providers. They are dispatching an investigation team to help assist us in our diagnosing the exact issue that occurred. We are not expecting any further problems, however we will continue to monitor the situation, as well as assist our transit provider on finding further details on this issue. We’re resolving this case for now, but if you have further issues, please contact support for assistance.

Update May 16th 12:47am PDT We are happy to report that service has been restored. There was a slight outage that recovered and we have our admin on site making sure no further incidents occur. We are currently diagnosing the cause of the connectivity issues. We apologize for the inconvenience.

Update May 12th 2:10am PDT This issue should now be taken care of at this time. We were able to configure a different link with another provider, and restore service. We apologize for any issues this may have caused.

Update May 12th 1:31am PDT We’re still seeing problems with inter-datacenter communications and are working with our network provider to get this resolved. An update as to what actually happened; we were aware of this planned maintenance, and didn’t expect it to be a customer facing issue. We apologize for this, and will announce all carrier planned maintenance windows in the future.

On Monday, we will be upgrading the Border Hardware at our Irvine Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4-hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, so your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 14th 10:08pm PDT: We started our maintenance at 10pm PDT, and experienced less than a minute of downtime. Our customer test domains are loading and are stable, and we are confident that this case can now considered to be resolved. Our Irvine datacenter now has new border hardware.

If you’re having issues with your sites, it’s likely unrelated, so please contact support with the details, so we can assist you.

Update May 14th 12:23pm PDT: We are going ahead with this maintenance tonight, May 14th, starting 10pm PDT. We will update this post with any status changes and progress reports.

Update May 10th 11:20pm PDT: We will need to reschedule this maintenance for another window unfortunately. The changes were successful however we were not comfortable with the state of the physical set up so we reverted back to a stable state. We will be creating a new status post once we find the best time to reschedule this event. Apologies we were unable to finalize this upgrade.

Update May 10th 11:10pm PDT: We have rolled back the changes. There were complications with the new hardware install. More information to come shortly. The sites we’re using to test are now back up, but on the old hardware.

Update May 10th 10:50pm PDT: Still working on getting everything working correctly on the new hardware. Service is still affected at this time.

Update May 10th 10:35pm PDT: Cut over is starting now. Service will be interrupted as we switch out the old hardware with the new. Please stand by. Your patience is greatly appreciated.

Update May 10th 10:20pm PDT: Final checks are being run and then the cut over will happen shortly. Next update should announce that the cut over to the new hardware has begun.

Update May 10th 10:10pm PDT: Finalizing router configurations before we start actually cutting over. Once we start cutting over, service WILL be down. We expect downtime itself to be minimized as much as possible.

Update May 10th 9:50pm PDT: Our maintenance team is coming together and getting ready to work. All important updates will be provided as soon as I have them. Thank you for your patience.

Update May 10th 3:10pm PDT: The maintenance on our border routers will begin at 10pm PDT. We expect the maintenance to complete by 11pm PDT, however for planning purposes we are specifying a 4 hour maintenance period through 2am PDT. Here is our expected sequence of events during the maintenance:

- Network services in our Irvine data center will be taken offline between [10-10:30pm PDT], resulting in all web hosting services in that data center being offline.

- A new border router that has been set up and prepared for operation will be connected to the network and tested, with an expected duration of [30 minutes] for the testing to be completed.

- Once the testing is completed and any issues are addressed to make the router fully operational, network services will be restored in Irvine and all web hosting services will be back online; we expect this to occur by [11pm PDT], however it is possible that network services will not be restored until [12 midnight PDT].

- In the event that the new router does not pass all tests by [11.30pm PDT], we will restore services on the current network router hardware, so that in any scenario, all network services are restored by [12 midnight PDT].

We will continually provide updates through the maintenance, especially when the outage begins and when we feel everything has returned to normal.

On Wednesday we will be upgrading the Border Hardware at our LAX Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4 hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 10th 1:10am PDT: We are considering this maintenance resolved. Thank you for your patience as we worked through this first installment of our Network Improvement Series. We are excited about these changes and what they mean to our service and what they’ll mean to you, our customers, as well.

Update May 9th 12:20pm PDT: Everything remains stable from what we are seeing. We are still working on getting our backup provider up and working correctly. Once that happens we will give everything the all clear.

Update May 9th 11:45pm PDT: All test sites are working correctly again. We are nearing the point of calling this complete however we are still verifying that all is clear. So far everything has gone off without any major surprises.

Update May 9th 11:30pm PDT: We are nearing the end of the maintenance. We are already seeing service being restored to some sites we are checking. Thank you again for your continued patience. We hope to have the maintenance window closed well before the full 4 hours. We will continue to keep you updated.

Update May 9th 11:10pm PDT: We are in the process of switching over to the new hardware. Service interruption will be happening at this time and we will do our best to limit the downtime as much as possible. Thank you for your patience.

Update May 9th 10:50pm PDT: We are currently switching over to the new protocol that our new juniper hardware requires. We expect a minor blip at this point of the upgrade.

Update May 9th 10:20pm PDT: We are finalizing router configurations from the old hardware to the new. Once that is done the actual cut over to the new border will take place.

Update May 9th 10:10pm PDT: We are still finalizing some preparations. The actual part of the maintenance that will cause service interruption has not started yet. We will keep you updated.

Update May 9th 9:10pm PDT: We are getting ready for the maintenance to begin. We still expect the start time to be 10pm PDT. Just letting you know we are here and ready to do this.

On Tuesday, May 15th, we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. The fiber ring is being tested to minimize errors and to ensure maximum throughput. We are giving ourselves a two-hour window starting at 10pm PDT Tuesday May 15th. Service affected will be web and email. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup, so this should only affect a small subset of customers. This maintenance should not take the full two hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.

The DreamHost panel is up and running again. Sorry about the inconvenience!

Update May 15th 11:26pm PDT We are continuing to work with our fiber ring provider to find out regarding the unexpected issues during our previous attempts. We will update this post again when we have more information. All web hosting services are still up, and we’re looking into the downtime the DreamHost panel is experiencing. Sorry about the inconvenience.

Update May 15th 11:03pm PDT We had to roll back our second attempt as well, and are currently talking to our fiber ring provider to find out regarding the unexpected issues. We will update this post again when we have more information. All web hosting services are up, but we’re looking into the downtime the DreamHost panel is experiencing. Sorry about the inconvenience.

Update May 15th 10:57pm PDT Our second attempt for the maintenance is about to start. Please see above for the details on possibly affected services. We’ll monitor service status and will post periodic updates as the maintenance progresses.

Update May 15th 10:51pm PDT We rolled back the first attempt and looking into the unexpected downtime it caused. We’ll update this post again with further developments and maintenance schedule as soon as we can. The maintenance window is still in place until midnight tonight. Thanks for your patience!

Update May 15th 10:39pm PDT We have started the maintenance of the fiber ring between the two Los Angeles based datacenters. We will post periodic updates as it progresses. We appreciate your patience during the process and apologize for the inconvenience it’s causing.

Update April 28th 1:20pm PDT We apologize for the continued down time but our network engineers have reported progress and we believe we’ll be fully operational again shortly.

Update April 28th 1:42pm PDT We would like to report that all of the sites we have tested that were experiencing issues are back up and loading for us. We’re continuing to monitor things closely but it appears that this matter has been resolved. We will provide more details when we have been able to assimilate all of the feedback from our admin and networking team members.

Update May 2nd 2:20pm PDT While this issue has been fixed since our last update, we kept this post unresolved until we were able to get more information about the specifics of it. This issue was caused by another DoS attack. During the process of blocking the external origin of the attack, the core routers became unreachable and therefore blocked traffic to Irvine. We had to reboot the routers to solve the problem and return to a stable state. We have already planned on replacing these routers with much more powerful ones in the next couple of weeks to help avoid future issues like this.

Update May 2nd 7pm PDT: This maintenance window has been cancelled and will be moved to Monday May 7th at 10pm. A new status post will be created to keep track of the outage.

Update April 25th 10:54pm PDT We’re happy to report that this upgrade is done, and happened without any issues! All service should be working as expected. If you’re seeing problems with your domains, please contact support immediately. — Brian H

Update April 25th 10:21pm PDT First part of upgrade is complete. We are starting the final upgrade now, then one more reboot and we will be upgraded. We will continue to monitor after upgrade is complete.

Update April 25th 10:12pm PDT The reboot of the network equipment is happening right now.

Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

Update April 25th 7:15pm PDT Maintenance is scheduled to start tonight at 10pm. Please keep an eye on this post for updates once the maintenance starts.

As this is a security release, it’s a recommended upgrade for all users who run WordPress.

Auto-upgrades have been queued for all customers who have them enabled, but will take some time to process.

You can check if you have automatic upgrades enabled (and upgrade manually if you prefer not to wait) from the One-Click Installer section of the DreamHost Panel:

* Log into the DreamHost Panel, and go to the One-Click Installer Section https://panel.dreamhost.com/?tree=goodies.installer
* Click on “Managed Installed Applications”
* Click on your domain, and a list of your installs will show
* Under the “Upgrade Action” column, you can view and modify the status of auto-upgrades for each install
* Click on the “Upgrade to 3.3.2″ button to the right of the install if you wish to upgrade right away

If you installed WordPress yourself, rather than using our One-Click Installer system, you can upgrade by logging into your WordPress Dashboard and clicking “Please update now” from the alert at the top of the screen.

You can see full release notes at WordPress.org:
http://wordpress.org/news/2012/04/wordpress-3-3-2/

Update Apr 17, 4:07pm PDT: The issue is now resolved, and lasted for about 20 minutes. Our network engineers and system administrators worked quickly to recover the network, and are continuing to work on any possible residual issues. We will make sure to keep you updated, as well as provide you with details regarding the root cause as soon as we have them. We sincerely appreciate your patience while we continue to get this resolved. -Seohee

Update Apr 17, 5:45pm PDT: We sincerely apologize for the network issues experienced at our Irvine datacenter. This particular incident was a routing table issue and was not associated with the previous outages we experienced at Irvine. This outage was smaller in scope and took down services for 20 minutes. The configuration issue that caused this took down 1 of 4 subnets used in Irvine. The fix is in place now and we will avoid any further problems due to this particular issue. We are continuing to work through significant network improvements and have made this our top priority.