Filer problems with blingy cluster. |
We are currently having a problem with a filer which has crashed and is recovering at this time. While this is happening some customers in the blingy cluster will experience problems loading their websites/email. We apologize for the outage and service is expected to return to normal as soon as the filer recovers.
UPDATE 3:01:AM PDT
The filer has finished recovering and all services are back up and running. We are working with the filer vendor to find the source of the crash to prevent any further outages.
Update 24/03/08 10am: We’re working on the file server again to alleviate the load that’s causing problems with web, mail and mysql services. Sorry about that.
Update 27/03/08: We are doing emergency data moves to quell the stem of problems recently caused by your file server. During these moves, your data may be inaccessible. We are moving as we can off as fast as possible. Very sorry about the continued inconvenience!
Update 27/03/08 This series of moves has finished. We are going to keep an eye on things to see how much it helped and may have to do more moves tonight and tomorrow morning to get everything working smoothly again. This post will be updated with more information as soon as possible.
Update 29/03/08
We are continuing to move data off of the problematic file server but it’s a bit of a catch-22 because customers on that machine are continuing to add data at a very high rate. It filled up this morning for a while causing device full errors as well as mail problems and issues serving websites (when these fill up it causes problems across the board). To explain in more detail, when we move data it does not immediately disappear (there is a ’snapshot’ created of the old data that remains in case there was a problem with the move - that ensures that we do not lose customer data but until the admin team can check the move to make sure it went through properly we cannot delete the old data). We just did some of that and have some breathing room again and of course more moves are still in progress but we are asking customer on this cluster to help us by holding up on any non-essential uploads of data for the next couple of days. As soon as we have a significant portion of the data removed the problematic file server will begin to function properly (and additional moves will go much more quickly and smoothly) but right now we’re having trouble moving data more quickly than it’s being added by people. If everyone could please limit uploads to absolutely essential data until we reach the turning point where everything is working this will be resolved much more quickly (in other words if for example you are setting up a repository of large files you’ll actually be better off waiting a couple of days and getting the all clear from us on this issue because you’ll be able to access that data reliably instead of cramming it on there now and slowing the recovery process).
In the meantime we’ll be doing everything we can to safely and quickly move data off and get things back to normal.
Added information: Some of the people recently moved to the new file server are seeing errors because the data did not get set up completely (loading the site will work but just show an empty index). The admin team has been running an rsync that will fully restore all data and should hopefully finish by 9 PM PST - once that is finished all site and email data will be available for those users.
Update 30/03/08
We’re still racing to keep ahead of new data being added so any help we can get on that front is greatly appreciated (we’re still asking for customers to limit uploads as much as possible to speed up the recovery process). Some customers who are being moved are seeing blank directories still but those are due to moves in progress and the data will be fully restored when those complete.
Update April 1, 2008
We seem to be ahead of the curve right now, we are moving data off the primary volume and on to a secondary one faster than new data is being uploaded. The volume hasn’t filled up completely in a few days. We are working closely with the technical support team to see how we can speed up the process further. Thank you for your patience.
Update, April 1, 2008
I apologize for the late update but we’ve been going over our options (while moving data of course). While we’re not seeing any real relief in terms of data uploads we do have some very large moves that are almost complete. Once those finish we can start deleting the data (for example one is around a half TB or around 500 GB which will be 4-5% of the total but it’s going to take until around Friday to delete it all, so we’re dealing with a ton of data). Tomorrow’s update should be earlier in the day and hopefully we’ll have some progress to reports from the large moves being complete.
Update, April 3, 2008
The data moves to other file servers has been running constantly, but last night and this morning some complications happened with the moves, requiring admin attention. To clear up some space there had to be a short interruption in file serving, this is now finished, space is available and the moves are continuing. The admins are fixing up the last of the web servers which were having issues after file serving was restored. Our apologies again for the continued issues.
Update, April 4, 2008
Today has been a pretty good day of progress. We were able to complete even more moves and free up more data from the file server. Moves have been going quicker and stability is dramatically improving. Monitoring of the servers and email in the blingy cluster today have shown a significant decrease in problems. Issues do still exist but the problem is noticeably getting better. We are also pleased to note that we have more storage that will be coming early next week. We believe that this will go a long way in helping us fix this major problem.
Update, April 4, 2008
Things are continuing to improve today - when I got in I was pleased to see that we had held firm and even gained a percent (the effected file server was down to 95% which is as low as I have seen it in the last week and since I have been working it has dropped to 94%). Performance should improve as we gain ground (this will speed up moving data off as well). This progress, along with the added storage space we are expecting early next week should hopefully allow us to restore service for our customers to normal.
5:45 PM PST : The moves we started just a while ago seem to be causing server problems, we’re looking into it and should have it resolved shortly (they were run just like the ones that had completed so we have to determine why these specifically caused an issue). Update: this resolved itself before we could detect the cause but we’re monitoring the situation to ensure that it’s not a recurring issue (we have no indication that it will be).
Update, April 8, 2008
Please see our other posting for details on the work we did on the effected file server:
http://www.dreamhoststatus.com/2008/04/06/30-min-blingy-downtime-tonight/
We are also continuing to offload data and are making good progress (it’s never as fast as we would like it to be of course). There’s excellent detail here in case you missed it:
http://blog.dreamhost.com/2008/04/07/another-anatomy/
which chronicles the situation and fills you in pretty much up to today. We’re seeing the data dip to 90% so we’re hoping to have it down in the 80’s by the end of the week (every percent we gain helps and as performance improves we can speed up the rate of moving but we’re still looking to hit critical mass where you get the proper level of performance).
Update, April 9, 2008
As we had hoped progress is speeding up as we free up more space - while the file server is showing 95% usage, around 11% of that is data that has already been moved and is no longer in use. Due to a software issue we haven’t been able to remove it yet (the admin team is working on the best way to execute that), but once that is gone we should be around 85% usage which is another large step forward.
In terms of effect, I have already seen improvement in site function for many customers as well as greatly increased speed in moving chunks of data off as well as receiving reports that mail is functioning quite a bit better. That said this issue remains at the High severity rating and in unresolved status as we have not reached a normal level of service. I can’t stress enough how sorry I am that our customers have had to put up with this but I thank those of you who have stuck with us (check the newsletter for details on what we’re doing for Blingy customers) and look forward to providing you with the level of service we strive for at DreamHost.
Update, April 9th, 2008 21:59 PDT
Unfortunately, we need to unmount the volume again to kill these snapshots before they leave us with 0 bytes of free space. In 2 hours (midnight) I will be taking the problematic volume offline to delete the phantom snapshots. Total downtime will be between 10 and 30 minutes. Sorry for the short notice and additional outage!
Update, April 10, 2008
Well the snapshot mentioned yesterday is gone and we’re actually at 83% used today which is below where we were hoping to see marked improvement (85%). Of course we’re still moving data off (which increases the usage on the file server) so that won’t fully translate to customer usage improvement but it should be quite a bit better and keep improving until we stop moving data.
Update, April 14, 2008
Okay, we’re finally getting ready to mark this as resolved.. things have seemed pretty much okay for a while now. But, just to be sure, we’re dropping the severity to Medium for now and leaving it as unresolved.
Update, April 17, 2008
We’re still hearing some reports of site slowness - we were able to resolve an issue causing high loads today which should help but we’re not going to consider this resolved until everyone is receiving good service.
| Severity: | Medium | Resolved: | No |
March 28th, 2008 at 1:17 pm
This is the result of overselling.
March 28th, 2008 at 1:22 pm
Actually, there is zero connection between the two and you’ve have shown that you have no idea of how any of this works, and that you’re just repeating regurgitated hype that’s puked all over little kiddie host boards like webhostingtalk.
March 28th, 2008 at 1:24 pm
“This is the result of overselling.”
Are you a customer Luis? Because if so, like me, you didn’t mind the reselling when you signed up.
Everyone can keep blaming it on this or on that, but that’s not going to do shit. What would do shit though is if someone from DH would actually GIVE US AN UPDATE!!!!!!!!!!!!!!!!!!
What do I tell my clients? “Hang in there” “It should be fixed soon” “I know you’re not getting what you paid for, but it is out of my hands…” Been there, done that. It’s getting to the point where all they want to hear is that “Everything is fixed and you won’t lose any more customers/business/MONEY!!!!”
Do what’s right DH. LET US KNOW WHAT THE HELL IS GOING ON……..please.
March 28th, 2008 at 1:29 pm
If your clients have a problem with uptime, then tell them to pay for a redundant setup. If they’re not willing to pay for uptime, then their sites aren’t important.
It costs more and makes no difference when things are good. But the people that pay to do things right never complain about downtime like this because they never have to see it.
March 28th, 2008 at 1:48 pm
JR Bob. Shared host != unrealible host. Noone screams for 1-2 hours of downtime. But it’s for weeks.
March 28th, 2008 at 1:58 pm
JR Bob lacks perspective. Shared hosting reliability should equal:
* “Oh, the site’s down right now. We’ll come back to it in an hour.”
* “Hold on, this page triggers a db query in it so it loads a little slowly.”
* “Hm. That didn’t work as I expect it to. Let me contact support. They’ll get back to me in about a day.”
For many, many sites this is an acceptable level of quality. 90% or 95% uptime is ok.
This is a sustained black out that about to hit a full week. That’s following a two month “brownout” with progressively worsening uptime/performance.
March 28th, 2008 at 2:05 pm
“If they’re not willing to pay for uptime, then their sites aren’t important.”
Well excuse me if my clients run small businesses and aside from trying to run that business have LIVES, which mean bills, kids, more bills, gas in their vehicles, some more bills, and TAXES.
You’re right though, JR Bob, I should just tell them that their lives are insignificant compared to corporate clients whose sites’ are obviously more important than theirs, and while I’m at it, maybe I’ll just tell them they should just pack it in all together because if they aren’t willing to pay me more money, well their just a big fat LOSER.
Give me a break dude. I’m in East Bumfuck, Tennessee. Aside from trying to get people to realize that e-mail and websites are two different things, times are tight and money is tighter. Think about that over your next latte at Starbucks.
March 28th, 2008 at 2:21 pm
Cry, cry, cry, all you want. You might as well, since you still haven’t figured out that it accomplishes nothing.
If they clients are that poor, then their sites aren’t worth the money of keeping up, which means their downtime isn’t the end of the world.
Redundancy between two shared hosts is cheap, so save the excuses.
I never said “shared == unreliable” — but the fact of the matter is that shared is the absolute least reliable form of hosting there is. That’s no big secret. But it’s easily cured with failover, but no one bothers to better their situation because it’s harder than throwing tantrums.
And this is far from a catastrophe. There are hosts that have been spammed here as better that are going into 48 hours of downtime on almost ALL customers — not just one cluster.
But seriously, you guys are sooooo riiiiight! I can tell your lack of ability to do anything other than complain has really paid off for you. While the smart people have their sites failed over, the dummies are whining in blog comments like the sky has fallen.
Keep up the great work! I’m sure your “clients” appreciate your hard work and dedication as you twiddle your thumbs and cry on a blog. THAT is what being a professional is all about.
March 28th, 2008 at 2:24 pm
I forgot to mention how funny it is when people that are poor because they don’t have a skillset worth paying for, think that pointing out that someone is doing way better than them is some sort of insult.
What do you do when people make fun of your old rusty Yugo? Reply with something clever like, “F You and the Ferrari you drove here in!”
That’ll teach those evil people that are smarter with their money than you are!
March 28th, 2008 at 2:26 pm
@ Preston - There is a big difference between a few hours here and there and the ongoing problems which have been popping up frequently lately, but don’t expect matters of degree to dawn on such folks.
March 28th, 2008 at 2:31 pm
I didn’t realize that asking for an update EVERY ONCE AND A WHILE was crying. I forgot, we’re on the bottom of the barrel of hosting so we shouldn’t expect much. Just like going into McDonald’s. I guess that would compare to shared hosting in the food business, but at least when they run out of ice cream they tell me why.
HERE’S THE FUNNIEST PART OF IT ALL - IF JR BOB IS SOOOOOOOO SMART AND HAS HIS SITES FAILED OVER…….WHY ARE YOU EVEN HERE? LOL!!!!!!!!!!!!!!!!!
March 28th, 2008 at 2:46 pm
@ JR Bob - Actually, I if you could read you’d understand that I was concerned for my clients, not me. I moved to Tennessee from NY for a reason…my money goes a lot further down here and I worked hard to have plenty of it, but I also don’t forget where I came from which gives me the ability to build relationships with my clients, not just milk them for everything they’ve got.
I don’t know when this turned into what it did, but my original concern remains…an update would be nice. DH obviously has way more REALLY REALLY POOR customers like me than Ferrari driving customers who use shared hosting like JR Bob.
@Not Me - well said
March 28th, 2008 at 3:02 pm
Man Oh! man! IMAP Keeps dropping me… What did I ever do to iMAP
March 28th, 2008 at 3:08 pm
To make fun of you idiots. Not all of us are too busy crying and trying to explain incompetence to clients to goof off.
Right, you’re rich, so spend a little more to make things better for your clients. Or are you only hero for the poor until it comes to spending money?
Basically, instead of your clients being in a bad position because they can’t afford to better it… really turns out that they’re in a bad position because you choose not to better it for them.
I don’t know any rich people with significant income from an online business that let their livelihood depend 100% on a single cheap shared hosting account being up all the time. If you’re making $100/month, fine. If you’re making $10,000 or more a month, it’s time to protect the investment.
You don’t have to be rich to be smart and I have no interest in driving a Ferrari.
I put sites where it’s most appropriate and accept the outcome of that decision. All shared hosts go through this crap. Just because they don’t do it the same week as Dreamhost, or OPENLY admit there are problems in public, doesn’t meant it’s not happening.
A business site with income that is important is going to have failover options, whether shared or dedicated. A parked site, personal blog, tiny-income site, whatever… is going on cheap shared hosting with zero redundancy, because that’s where it belongs.
The difference is I don’t throw the same cheap solution at every project. It’s not really harder to think and plan for problems than it is to host-hop, or sit around complaining and doing nothing to prevent it from happening again.
March 28th, 2008 at 3:23 pm
Come clean. Which of you is putting this anti-dreamhost propaganda on YT? youtube.com/watch?v=qS7nqwGt4-I
March 28th, 2008 at 3:27 pm
I never claimed to be rich.
Bottom line, you’re right man. You’re always right, and I’m sure that’s what your wife (or boyfriend, whichever you prefer) tells you too.
“To make fun of you idiots”
Well, I can just picture you getting a e-mail from your buddy saying, “Hey, Dreamhost has been having problems and some have been affected for almost a week.” RE: “Cool. Want to go to their status page and make fun of the dumb f*ckers for using shared hosting”, which is a clear indicator of your character……..
or, option B: You’re full of shit.
I’m leaning toward option B.
Anyway, that’s my contribution for the day. I came here to find an update, not an asshole. At least the asshole talks though
March 28th, 2008 at 3:30 pm
I did!
March 28th, 2008 at 3:34 pm
Blingy disk at 99%, about to fill up again (making everything inoperable)
$ ssh spyro.dreamhost.com df .
10.175.4.69:/fcvol1/blingy/yoshie
13711880192 13545685676 166194516 99% /home/.yoshie
March 28th, 2008 at 3:37 pm
You know what sucks? I cant even leave dreamhost until my damn site works again because I cant access my ftp and mysql to export all of my data
March 28th, 2008 at 3:38 pm
First you said:
Now it’s:
Which is it, because people that don’t link “being rich” to “having a lot of money” generally don’t need to sit around and remember “where they came from,” since they’re still there.
So you just like to give the impression that you have “a lot of money” but you really don’t. Is that what you’re saying?
Coming from the guy that isn’t right, and instead of admitting it, goes for the gay crack distraction instead.
Ask your miserable clients which one of us is right. Maybe when they hear the question and see how obvious the answer is, they’ll dump you and find a competent replacement that will look out for them a little better, instead of someone that takes the cheapest route and doesn’t offer better solutions for those that might require more.
Yeah, you’re smart. I’m really miserable and suffering just like you, so instead of going along with the crying, I take the opposite of my own side.
And since I clearly know what the alternatives are, there’s no way I could possibly know how to implement them and reap the rewards of it, rather than doing the opposite for the sake of complaining.
And seriously, this means A LOT coming from the guy that can’t keep his clients happy and doesn’t even seem to know if he’s rich or poor from one minute to the next. You’ve got a great handle on things!
March 28th, 2008 at 3:40 pm
What part of “not keeping local backups” seemed like a good plan to you?
March 28th, 2008 at 3:41 pm
What do local backups have to do with NO FUCKING SERVICE?
March 28th, 2008 at 3:41 pm
I do, but I want to dump everything that is current and do a one time move.
March 28th, 2008 at 3:44 pm
Assholes like JR Bob + Host spamming = The obvious need for a customer only status board DH
March 28th, 2008 at 3:47 pm
@JR Boob - FYI - Media Temple, The Planet, Rochen…the three other hosts I use for my “clients who are important” by your standards.
March 28th, 2008 at 3:50 pm
Dear Dreamhost:
Kill yourself violently. I’m going to MediaTemple. And you all should to. This if ridiculous.
March 28th, 2008 at 3:51 pm
Didn’t you just say I’m the one that can’t read? His post said he can’t LEAVE Dreamhost. He could with backups.
Any other tough questions you need answered?
Since I’m a customer, I’d still be here. Not too bright, are you?
The host spammers love that Dreamhost leaves it open. Actually, all spammers. The more the word gets out, the more spam will show up, since they rarely even delete it.
I believe there are directions for scheduled SQL dumps in the wiki, so you can always have a fairly recent copy.
Being up and running, but slightly outdated, is still better than being down… depending on how important the site is, or what type of content isn’t up-to-date.
March 28th, 2008 at 3:54 pm
And there you have it. Just be honest with your clients for once and tell them the reason they’re not up like the others is because you decided they’re not as important.
Basically, you’re complaining about your own decisions.
If you really have all these other hosting solutions up and running, it just makes you look even more foolish for not failing over to them.
March 28th, 2008 at 3:54 pm
“Since I’m a customer, I’d still be here.” or “To make fun of you idiots.”
I guess it’s a little clearer now?
March 28th, 2008 at 3:57 pm
“And there you have it. Just be honest with your clients for once and tell them the reason they’re not up like the others is because you decided they’re not as important.”
Really? ‘Cause I thought if they were POOR they weren’t important. Therefore, if they’re too poor to afford dedicated service, how is it my decision again?
March 28th, 2008 at 4:02 pm
The site moves has finished? Methinks not
[nariko]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.0G 1.3G 614M 69% /
/dev/sda3 6.9G 1.9G 4.7G 29% /usr/local
/dev/sda8 2.0G 79M 1.8G 5% /tmp
10.175.4.69:/fcvol1/blingy/abduct
13T 13T 187G 99% /home/.abduct
10.175.4.69:/fcvol1/blingy/zuni
13T 13T 187G 99% /home/.zuni
10.175.4.69:/fcvol1/blingy.localbackups/nariko
13T 13T 187G 99% /mnt/filerbackup
10.175.4.69:/fcvol1/blingy.crontabs/nariko
13T 13T 187G 99% /usr/local/var/spool/cron/crontabs
Don’t get me wrong, 187G is a nice chunk of disk space, but its only a temporary bandage. The whole blingy cluster should be way below 99% disk usage.
March 28th, 2008 at 4:04 pm
For all those justifying the continued downtime as a cheap webhosting solution, my webmail has been down for 3 days now (and on and off in Outlook) and I’m on a private server, at a cost of $75/month. The amount of money the majority everyone is paying is a non-issue. Dreamhost provides a service and they offer that service at a price they determine and we accept. That gives no excuse for downtime of this magnitude that we haev all been dealing with, and I believe Dreamhost deserves all the flak they are getting from disgrunted customers. Myself included.
March 28th, 2008 at 4:05 pm
You can be a customer and not be stupid. Well, you can’t, but it is possible.
Upgrade them and take a hit on the profit. It would be worth it if anything about your image was good enough to make it worth maintaining. Don’t want to help them out? Guess you forgot “where you came from”?
You should be able to handle it, at least on the days where you’re pretending to be rich and “remembering where you came from.”
Did you know that a dedicated server becomes a shared one when you start adding users? You can’t afford one dedicated server to put your Dreamhost clients on? I guess you’re back to not being rich? Just wait 5 minutes until you’re rich again, then order the server. I really don’t remember saying to give each Dreamhost client their own server.
Rackspace wasn’t on your list and I don’t see anything too expensive there, so what’s the problem?
Seriously, run the garbage you’re spewing here past your clients and see how they feel about you. If you’re smarter than me, you have nothing to lose.
March 28th, 2008 at 4:06 pm
Please fix it quickly!
March 28th, 2008 at 4:08 pm
I forgot to mention: All they need to do is turn off comments. Let people whine on their own sites and leave that status site for announcements only.
March 28th, 2008 at 4:10 pm
Where were you earlier? They asked if they should fix it quickly earlier, but some other guy said, “No, that’s okay, take your time.” So now, everyone has to wait.
March 28th, 2008 at 4:12 pm
In other words, they’re just going to rip anything you say out of its context and use it to build more straw men which they will then proceed to knock down.
March 28th, 2008 at 4:15 pm
I am furious. Not just because of the inconvenience this has caused me & my business; but also because DH has given me no choice but to find a new host which is also an inconvenience. I hope they don’t try to make leaving a problem. I don’t want to leave but this blingy problem is forcing me to seek a more reliable host. One that is prepared to deal with the inevitable technical problems with minimal interuption in services.
March 28th, 2008 at 4:27 pm
@Not Me
Point well taken. Sometimes it’s hard to ignore stupidity, but in the end it makes you feel stupid for even acknowledging it…
March 28th, 2008 at 4:29 pm
That’s why I keep replying to you.
March 28th, 2008 at 4:53 pm
I’m surprised the comments come so slowly in here, they really upset a lot of people judging by this http://forums.hostingheadaches.com/DreamHost/search.php?query=blingy
March 28th, 2008 at 5:01 pm
JR Bob: Is this your tempo? Just fat asses take weeks to buy a fileserver…
March 28th, 2008 at 5:11 pm
Yeah, that’s the problem. You figured it all out, just let them know and they’ll have it fixed right up.
March 28th, 2008 at 5:31 pm
Anyone know the ip of blingy’s mail server? I moved my MX records away and need to download old email.
March 28th, 2008 at 5:51 pm
Yeah, does anyone know how to get old emails back? I changed my MXs too.
-j
March 28th, 2008 at 6:17 pm
http://hostingheadaches.com/articles/Retrieving_Old_Emails_From_DreamHost_Accounts/
March 28th, 2008 at 6:29 pm
Oh lovely, my friends site on Spacey isn’t taking uploads, could anyone guess why? Wonder when that one will get posted…
March 28th, 2008 at 6:46 pm
The article on that other guy’s site should help out: http://hostingheadaches.com/How_to_upload_to_spacey_cluster_at_Dreamhost/
Reading through that would make way more sense than simply contacting support to see what the problem is.
March 28th, 2008 at 6:51 pm
That other guy’s site has a write-up about it: http://hostingheadaches.com/articles/Uploading_To_Spacey_Cluster_At_Dreamhost
Reading through that would make much more sense than contacting support to see what the problem is.
March 28th, 2008 at 6:54 pm
I think that Dreamhost got owned, the downtime is unbeliveable… i think they are playing cat-mouse!!!