Due to a hardware failure on “copenhagen”, our admin team is in the process of migrating the server to new hardware. It hopefully shouldn’t take too long to move everything over — but we’ll be posting updates here as the issue progresses.
Update – Oct. 25th, 5:24 PM (PST): Due to the state of the server, fixing everything up is going to take a little longer than normal. So we were kind of dead wrong on our initial estimate. Sorry about that! We’re doing our best to get this resolved as fast as we can, however.
Update – Oct. 25th, 6:27 PM (PST): As of this moment, we’ve decided to go ahead and mount your data from our most recent backups. During this time we will be restoring your content from our backups to the new hardware.
Update – Oct. 26th, 7:10 PM (PST): Due to the filesystem flipping itself into “read only” status earlier today, the restore is taking even longer than anticipated. It is still running and all content is expected to be restored eventually — but please note that the server will be running at reduced capacity (see: slower) until everything is back to normal.
Update – Oct. 26th, 9:25 PM (PST): One of our admins has restarted the Apache services on the server to help knock the load down. This should stabilize the sites that are up and running for the time being. Basically, the sites that are still missing are causing some Apache services to hang. That ends up driving the load up.
One of our techs who has experience with recovering servers that have seen problems like this will be taking a look at things shortly. Once we know where the recovery effort stands, we’ll make sure to update this article with more info.
Update Oct. 27th 12:32 AM PDT: I was able to get the array back into a workable state with no visible corruption. I started copying the data over to the new copenhagen now. An ETA before all data is restored is probably around 24 hours. Any data you upload that was newer than the original should not be over-written.
Update Oct. 27th 4:02 PM (PST): The data is still copying and on target to be finished in the (hopefully early) AM hours of the 28th. Once the copy has completed, we’ll drop another update here.
Update Oct. 27th 10:50 PM PDT: All of the users we didn’t originally have backup data for have been restored from the original hardware except the largest 16 users (so the vast majority have been recovered). Users have been recovered in the order of disk space usage (smallest first) so all the remains are the largest users on the server. If you are on copenhagen, have less than 10GB of data in your users directory and you are missing files please contact support. This will be listed as completely resolved once these final 16 users are copied.
Update Oct. 28th 10:28 PM PDT: For several hours now all the data that we were missing backups from recent moves to the server have been restored from the original hardware. If anyone else is missing data please contact support to have your user manually restored.