Wednesday May 16th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 1 hour (POSTPONED)

We will be attempting to perform fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. This maintenance was scheduled for last night but we had to postpone it due to problems after the cut over to the new fiber. Our provider came out to diagnose the problems of our last attempt last night, and we are confident that we can finish this in a one hour window tonight. If there are problems we have a way to immediately roll back the change. Please stay tuned for further updates.

Update May 16th 10:05pm PDT: We have canceled this update until another date. We will create a new post once we decide it is time to try for this maintenance again. We felt we were too hasty with not enough proper warning that we would be doing this maintenance tonight. Thank you for you patience.

Comments: 3

Posted In: General Outages

Posted By: Ralph

Immediate Maintenance Required on Discussion List Servers ( RESOLVED )

Our system administration is performing maintenance on our discussion list servers. We expect all discussion list services to be down approximately one hour. We apologize for the inconvenience but it was deemed necessary as we’re setting up an improved (and more reliable) backup system for the lists.

UPDATE May 16th 3:18PM PDT The maintenance has completed. You are archives are now accessible. Again, we apologize for any inconvenience this has caused. If you notice any problems, please contact support for assistance. Previous updates available after the jump →

Comments: 0

Posted In: General Outages

Posted By: Oscar

Networking issue at the Irvine datacenter (RESOLVED)

We’re experiencing connectivity issues at our Irvine datacenter. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update May 16th 1:39am PDT Tonight’s connectivity issues appeared to be a result of a drop out experienced by one of our transit providers. They are dispatching an investigation team to help assist us in our diagnosing the exact issue that occurred. We are not expecting any further problems, however we will continue to monitor the situation, as well as assist our transit provider on finding further details on this issue. We’re resolving this case for now, but if you have further issues, please contact support for assistance.

Previous updates available after the jump →

Comments: 38

Posted In: General Outages, Major Outage

Posted By: andrea

Transit issue between LAX and Alchemy datacenters (RESOLVED)

Currently, at this time we’re seeing network degradation between our LAX and Alchemy datacenters due to maintenance by our transit provider. We are investigating this issue and will have more info as it progresses. This issue should only affect a small amount of customers with mysql databases that are in opposite datacenters, or users using webmail, on our homiemail-sub5 mail cluster — Brian H

Update May 12th 2:10am PDT This issue should now be taken care of at this time. We were able to configure a different link with another provider, and restore service. We apologize for any issues this may have caused.

Previous updates available after the jump →

Comments: 36

Posted In: General Outages

Posted By: Brian

Network Improvement Series Maintenance #2 Monday May 14th 10pm PDT 4 hours – Upgrading Border Hardware at our Irvine DC (RESOLVED)

This is the second step in our DreamHost Network Improvement Series.

On Monday, we will be upgrading the Border Hardware at our Irvine Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4-hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, so your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 14th 10:08pm PDT: We started our maintenance at 10pm PDT, and experienced less than a minute of downtime. Our customer test domains are loading and are stable, and we are confident that this case can now considered to be resolved. Our Irvine datacenter now has new border hardware.

If you’re having issues with your sites, it’s likely unrelated, so please contact support with the details, so we can assist you.

Previous updates available after the jump →

Comments: 44

Posted In: Network Maintenance

Posted By: Ralph

Network Improvement Series Maintenance #1 Wednesday May 9th 10pm PDT 4 hours – Upgrading Border Hardware at our LAX Data Center (RESOLVED)

This is the first step in our DreamHost Network Improvement Series.

On Wednesday we will be upgrading the Border Hardware at our LAX Data Center. This is an important step in our plans to provide you the best in network performance and stability. We will be giving ourselves a 4 hour window to complete the installation and configuration of the new border, however, we expect that any actual outage to be minimal. Our team will stay on hand to make sure the transition is as smooth as it can be. Any sites or email hosted in this data center will have no network path to them, your site will be down and your email should queue up on servers sending messages to you. We thank you for your patience as we continue to work to improve your DreamHost experience.

Update May 10th 1:10am PDT: We are considering this maintenance resolved. Thank you for your patience as we worked through this first installment of our Network Improvement Series. We are excited about these changes and what they mean to our service and what they’ll mean to you, our customers, as well.

Previous updates available after the jump →

Comments: 79

Posted In: Network Maintenance

Posted By: Ralph

Tuesday May 15th Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours

Previously this was to be done on Wednesday May 2nd, but we moved it to the 15th.

On Tuesday, May 15th, we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles-based data centers. The fiber ring is being tested to minimize errors and to ensure maximum throughput. We are giving ourselves a two-hour window starting at 10pm PDT Tuesday May 15th. Service affected will be web and email. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup, so this should only affect a small subset of customers. This maintenance should not take the full two hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 16th 12:17pm PDT We have to postpone the maintenance while our fiber ring provider works out the problem on their that caused the unexpected issues. We’ll put up a new status post as soon as we know the new schedule.

The DreamHost panel is up and running again. Sorry about the inconvenience!

Previous updates available after the jump →

Comments: 155

Posted In: General Outages

Posted By: Ralph

Networking Issue at Irvine DataCenter (RESOLVED)

We have noticed connection issues in our Irvine datacenter. This is resulting in site and database downtime. Our network engineers and system administration team are onsite working to restore connectivity. We will update this post as soon as we have more information.

Update April 28th 1:20pm PDT We apologize for the continued down time but our network engineers have reported progress and we believe we’ll be fully operational again shortly.

Update April 28th 1:42pm PDT We would like to report that all of the sites we have tested that were experiencing issues are back up and loading for us. We’re continuing to monitor things closely but it appears that this matter has been resolved. We will provide more details when we have been able to assimilate all of the feedback from our admin and networking team members.

Update May 2nd 2:20pm PDT While this issue has been fixed since our last update, we kept this post unresolved until we were able to get more information about the specifics of it. This issue was caused by another DoS attack. During the process of blocking the external origin of the attack, the core routers became unreachable and therefore blocked traffic to Irvine. We had to reboot the routers to solve the problem and return to a stable state. We have already planned on replacing these routers with much more powerful ones in the next couple of weeks to help avoid future issues like this.

Comments: 163

Posted In: General Outages

Posted By: danwy

Wednesday May 2nd Network Maintenance – fiber ring between our two Los Angeles based data centers – 10pm PDT – 2 hours (MOVED TO MONDAY – NEW POST)

On Wednesday May 2nd we will be performing fiber maintenance that will confirm our transfer capacity between our Los Angeles based data centers.  The fiber ring is being tested to minimize errors and to ensure maximum throughput.  We are giving ourselves a 2 hour window starting at 10pm PDT Wednesday May 2nd.  Service affected will be strictly web content. Websites that have their hosting server in one data center and their MySQL databases in the other will notice database connection errors. MySQL and web server in different data centers is not our typical setup so this should only affect a small subset of customers.  This maintenance should not take the full 2 hours to complete. We will be using the maintenance window to monitor the changes and make sure no complications arise. This upgrade is extremely important to our reliability focus and efforts.

Update May 2nd 7pm PDT: This maintenance window has been cancelled and will be moved to Monday May 7th at 10pm. A new status post will be created to keep track of the outage.

Comments: 55

Posted In: General Outages

Posted By: Ralph

Wednesday April 25th Network Maintenance Irvine Data Center 10pm PDT – 2 hour window (RESOLVED)

In our continued efforts to upgrade our network, we will be scheduling maintenance on a piece of our border network equipment on April 25th. The action being performed is a simple software upgrade which will improve resource utilization on the new Juniper router we installed. We anticipate the downtime itself to last no longer than 10 minutes; however, we will be on high alert for the 2 hours preceeding the window as well as during and after the upgrade to make sure there are no complications. We sincerely appreciate your patience and understanding as we strive to improve our service for you.

Update April 25th 10:54pm PDT We’re happy to report that this upgrade is done, and happened without any issues! All service should be working as expected. If you’re seeing problems with your domains, please contact support immediately. — Brian H

Update April 25th 10:21pm PDT First part of upgrade is complete. We are starting the final upgrade now, then one more reboot and we will be upgraded. We will continue to monitor after upgrade is complete.

Update April 25th 10:12pm PDT The reboot of the network equipment is happening right now.

Update April 25th 10:00pm PDT Maintenance has started. You will notice an interruption of service as we upgrade the hardware. We will keep you posted as we get close to completing this upgrade.

Update April 25th 7:15pm PDT Maintenance is scheduled to start tonight at 10pm. Please keep an eye on this post for updates once the maintenance starts.

Comments: 54

Posted In: General Outages

Posted By: Ralph