Emergency maintenance window

Tomorrow night, January 11, 2008 at 10:00PM Pacific time, we need to perform two maintenance tasks:

1. Reboot a switch in which one of its processes if hung. It is currently passing traffic, but various Layer 3 duties it performs are in a degraded mode. We have otherwise worked around the issue, but we need to get that process un-stuck. We will be using this opportunity to upgrade the switch to the latest software version in hopes that this will not happen in the future, as this doesn’t add any time to the reload.

This should not affect general site performance, however some internal functions will be affected. If you notice sub-optimal ssh times, or other DNS related issues while logged in to our servers, they should resume in about 5 minutes when the switch comes back online.

2. Update the firmware on the disk shelves attached to a filer that crashed today. Speaking with various technicians, the current version of firmware could have been the cause of the outage in frisky today. This will cause a similar outage while the shelves reload. It should take around 5-minutes. If your data is hosted on this filer, you will not have access to it for those 5 minutes. No email or other data shall be lost, and the new firmware should make it more stable.

We’re sorry about the short notice of this window. If either of these two problems becomes worse overnight, we will go ahead and apply the fixes immediately.

Updated: 10:06 Pm Pacific: All updates went smoothly. Total downtime for affected people was <5 minutes per update. Hopefully things will be smoother here on out!

.

11 Responses to “Emergency maintenance window”

  1. 1
    xxxxxx Says:

    Oh lord…

  2. 2
    Lord Says:

    Yes my son?
    What? Oh, you didn’t realize that things break and need to be repaired? Well, sorry, perhaps I’ll give the next generation a little more common sense. It seems to be lacking these days…

  3. 3
    Doug Says:

    It needs to happen now. Control panel goodies installer is severely backed up and I just got a PHP mail() message I sent five hours ago.

  4. 4
    Sphere of Hip-Hop Says:

    LOL @ lord.

  5. 5
    Mike Says:

    another LOL @ Lord :P

    Yep they are working on it, you can tell by the lack of any non-static website on my account :]

    Good luck anyways, and don’t break too much stuff…

  6. 6
    Mike Says:

    Wait… its 10am in America now, not 10pm, something is broken anyway :S

  7. 7
    MsU Says:

    Whew! Glad to see it’s not just me.

  8. 8
    James Says:

    My database just went down… It’s 12:19am Jan 12 PST

  9. 9
    Patrick Says:

    website & ssh down for me.

  10. 10
    Nathan Says:

    My wordpress site says “error connecting to database”

  11. 11
    Taha Ali Adil Says:

    oh my GOD … its shake My heart

Leave a Reply