Network fileserver troubles

One of our fileservers is having some serious problems, and is causing several of our shared hosting servers to become unresponsive. We’re working to swap out the filer now and get everything back online.

Watch this space for updates.

We think we have the problem narrowed down now and fixed. The fileserver has been running a consistency check on the volume for about the last hour now, and should be done in about 30 more minutes. Once this is done we can get service restored!

It looks like the file system check wasn’t as finished as I thought it was. It will be several more hours until it finishes. We’re very sorry about the downtime, we are working as fast as we can.

Update 3:50pm pdt: We still have no eta one when this will be resolved. However, we recently issued some fixes to the affected mail servers that should allow queued mail for those not directly affected to start coming in.