We now have a critical patch from our file server vendor which they believe will fix the crashing problems we’ve had. It takes 20 minutes to install, and unfortunately requires a reboot. We’ve decided to schedule this upgrade for 11:30pm PDT tonight, Sunday April 6th, 2008. That is pretty much the lowest usage time across all our users in a week, and we’ve decided we can’t wait another week before applying this patch in order to give more notice.

Our apologies for the very short notice! We received notice that the patch was ready this morning. Hopefully by midnight (PDT) tonight, Blingy will finally be happy again…

UPDATE 12:15AM PDT:

Still haven’t started the upgrade yet.. it took them a little longer to get the patch than expected, however, we should have it in the next 2 minutes, and then the upgrade will begin, probably around 12:30 now, and be done before 1am Monday morning PDT. Hang in there!

UPDATE 12:54AM PDT:

A few more bumps! Still no changes.. and now we’re afraid we’ll have to push off the upgrade to about 1:45AM (9:45AM GMT) it looks like.. hopefully all finished by 2:15AM PDT (10:15AM GMT), our apologies to blingy customers in Europe.

UPDATE 02:10AM PDT:

All done! The actual reboot only took about 3 minutes, and then there were some residual effects from the reboot on various web and mail servers for up to ten minutes after that. But things seem settled down now and the upgrade “took”. We all hope it did the trick. BIG TIME.

I apologize for this mistake. I was intending to be editing our non-live access-list, and edited the live one by mistake.

Update 9:54 PM Pacific: We have unblocked the firewall and our routing is coming back online. Sites should start working again as soon as our router finishes loading its tables. I use “resolving” because it is still processing the routes.

Update: Things seem to be working again, we apologize for the outage.

Our planned downtime window is 2 hours, during which all services including email, webmail, websites, ftp, ssh and the web panel will be offline. We will post updates here as things progress, so please keep an eye out and we’ll let you know when things are back online!

We are going to try to take care of as many things as we can during this maintenance window, so to minimize required outages in the future.

We greatly apologize for the inconvenience and thank all of you for your patience!

– UPDATE 12:18 AM Sat Oct 20

The router is back up, again, after the redundancy test we performed on the upgrade failed miserably. We are now running the new version of IOS, fortunately, so all is not lost. We are currently seeing if the network and the servers are back up and running.

– UPDATE 1:34 AM Sat Oct 20

We believe that all systems are Go! Please write in to support if you are still having any problems. We will continue to monitor this throughout the night. We apologize for the extended outage, we will be contacting Cisco regarding that bug in the redundant failover system.

Update: Services seem to be returning to normal, watch this space for more updates.

The impact of this upgrade is that it will bring many of our internet uplinks offline, and make a good portion of our network completely unreachable until the router comes back online.

Update 23:27 PM: We reloaded the router with the latest revision of firmware per Cisco’s (And our CCIE consultant’s) suggestion. All seems to be well, the router is passing traffic and the servers appear to have recovered gracefully.

UPDATE 11:53am: Should be all back to normal now. One of our mail load balancing servers failed and was giving the errors while it was being switched over to a backup server. If you continue to have problems, please contact support.

We apologize for the inconvenience.

UPDATE–

After working with our vendor we seem to have the problem under control. We are leaving this post in an “unresolved” status still until we are 100% sure the problem is not coming back as it did yesterday but so far everything is looking much better.

UPDATE– Thu May 24 10:42:09 PDT 2007

Things certainly seem better. We’re leaving this post as “unresolved” for now while we troubleshoot the outage but as of this moment, DNS resolution appears to be working fine.

RESOLVED —
The problems regarding dns and our load balancer have been resolved. It was a combination of problems that caused the outage. The main problem was that the default udp settings on the load balancer did not have any timeout values set (BAD!!!). This was causing any udp connections that didn’t complete to fill up connection tables on the load balancer causing its CPU to spike and stop serving new requests. We sincerely apologize for this outage and are working on new plans that would prevent a similar outage in the future.

It was discovered by ABM Engineering during the power monitoring
equipment installation, that a Phase C Conductor Cable on UPS #4
has been compromised to the point that immediate action to
repair it is necessary or a ground fault will occur to the building
systems.

Since only one of our three data centers is losing power, not all servers will be shut down. Unfortunately, our core routers and our upstream providers are in that data center, so there will be no network to any servers during the window. Gaah.

Since we have advanced (albeit limited) warning of this event, we will be onhand to physically power off all of our equipment at 11:15PM PST. Barring any unforeseen issues (on our end or the building engineer’s) we plan on having everything back up by 4AM PST, hopefully much sooner.

This outage will affect everyone’s sites and email. Email sent to our customers during this outage will be deferred on the sending server and will be delivered after service is restored.

We apologize for the inconvenience this no doubt causes, as well as for the short notice. We will be posting updates as possible during the outage.