Filer problems with blingy cluster. |
We are currently having a problem with a filer which has crashed and is recovering at this time. While this is happening some customers in the blingy cluster will experience problems loading their websites/email. We apologize for the outage and service is expected to return to normal as soon as the filer recovers.
UPDATE 3:01:AM PDT
The filer has finished recovering and all services are back up and running. We are working with the filer vendor to find the source of the crash to prevent any further outages.
Update 24/03/08 10am: We’re working on the file server again to alleviate the load that’s causing problems with web, mail and mysql services. Sorry about that.
Update 27/03/08: We are doing emergency data moves to quell the stem of problems recently caused by your file server. During these moves, your data may be inaccessible. We are moving as we can off as fast as possible. Very sorry about the continued inconvenience!
Update 27/03/08 This series of moves has finished. We are going to keep an eye on things to see how much it helped and may have to do more moves tonight and tomorrow morning to get everything working smoothly again. This post will be updated with more information as soon as possible.
Update 29/03/08
We are continuing to move data off of the problematic file server but it’s a bit of a catch-22 because customers on that machine are continuing to add data at a very high rate. It filled up this morning for a while causing device full errors as well as mail problems and issues serving websites (when these fill up it causes problems across the board). To explain in more detail, when we move data it does not immediately disappear (there is a ’snapshot’ created of the old data that remains in case there was a problem with the move - that ensures that we do not lose customer data but until the admin team can check the move to make sure it went through properly we cannot delete the old data). We just did some of that and have some breathing room again and of course more moves are still in progress but we are asking customer on this cluster to help us by holding up on any non-essential uploads of data for the next couple of days. As soon as we have a significant portion of the data removed the problematic file server will begin to function properly (and additional moves will go much more quickly and smoothly) but right now we’re having trouble moving data more quickly than it’s being added by people. If everyone could please limit uploads to absolutely essential data until we reach the turning point where everything is working this will be resolved much more quickly (in other words if for example you are setting up a repository of large files you’ll actually be better off waiting a couple of days and getting the all clear from us on this issue because you’ll be able to access that data reliably instead of cramming it on there now and slowing the recovery process).
In the meantime we’ll be doing everything we can to safely and quickly move data off and get things back to normal.
Added information: Some of the people recently moved to the new file server are seeing errors because the data did not get set up completely (loading the site will work but just show an empty index). The admin team has been running an rsync that will fully restore all data and should hopefully finish by 9 PM PST - once that is finished all site and email data will be available for those users.
Update 30/03/08
We’re still racing to keep ahead of new data being added so any help we can get on that front is greatly appreciated (we’re still asking for customers to limit uploads as much as possible to speed up the recovery process). Some customers who are being moved are seeing blank directories still but those are due to moves in progress and the data will be fully restored when those complete.
Update April 1, 2008
We seem to be ahead of the curve right now, we are moving data off the primary volume and on to a secondary one faster than new data is being uploaded. The volume hasn’t filled up completely in a few days. We are working closely with the technical support team to see how we can speed up the process further. Thank you for your patience.
Update, April 1, 2008
I apologize for the late update but we’ve been going over our options (while moving data of course). While we’re not seeing any real relief in terms of data uploads we do have some very large moves that are almost complete. Once those finish we can start deleting the data (for example one is around a half TB or around 500 GB which will be 4-5% of the total but it’s going to take until around Friday to delete it all, so we’re dealing with a ton of data). Tomorrow’s update should be earlier in the day and hopefully we’ll have some progress to reports from the large moves being complete.
Update, April 3, 2008
The data moves to other file servers has been running constantly, but last night and this morning some complications happened with the moves, requiring admin attention. To clear up some space there had to be a short interruption in file serving, this is now finished, space is available and the moves are continuing. The admins are fixing up the last of the web servers which were having issues after file serving was restored. Our apologies again for the continued issues.
Update, April 4, 2008
Today has been a pretty good day of progress. We were able to complete even more moves and free up more data from the file server. Moves have been going quicker and stability is dramatically improving. Monitoring of the servers and email in the blingy cluster today have shown a significant decrease in problems. Issues do still exist but the problem is noticeably getting better. We are also pleased to note that we have more storage that will be coming early next week. We believe that this will go a long way in helping us fix this major problem.
Update, April 4, 2008
Things are continuing to improve today - when I got in I was pleased to see that we had held firm and even gained a percent (the effected file server was down to 95% which is as low as I have seen it in the last week and since I have been working it has dropped to 94%). Performance should improve as we gain ground (this will speed up moving data off as well). This progress, along with the added storage space we are expecting early next week should hopefully allow us to restore service for our customers to normal.
5:45 PM PST : The moves we started just a while ago seem to be causing server problems, we’re looking into it and should have it resolved shortly (they were run just like the ones that had completed so we have to determine why these specifically caused an issue). Update: this resolved itself before we could detect the cause but we’re monitoring the situation to ensure that it’s not a recurring issue (we have no indication that it will be).
Update, April 8, 2008
Please see our other posting for details on the work we did on the effected file server:
http://www.dreamhoststatus.com/2008/04/06/30-min-blingy-downtime-tonight/
We are also continuing to offload data and are making good progress (it’s never as fast as we would like it to be of course). There’s excellent detail here in case you missed it:
http://blog.dreamhost.com/2008/04/07/another-anatomy/
which chronicles the situation and fills you in pretty much up to today. We’re seeing the data dip to 90% so we’re hoping to have it down in the 80’s by the end of the week (every percent we gain helps and as performance improves we can speed up the rate of moving but we’re still looking to hit critical mass where you get the proper level of performance).
Update, April 9, 2008
As we had hoped progress is speeding up as we free up more space - while the file server is showing 95% usage, around 11% of that is data that has already been moved and is no longer in use. Due to a software issue we haven’t been able to remove it yet (the admin team is working on the best way to execute that), but once that is gone we should be around 85% usage which is another large step forward.
In terms of effect, I have already seen improvement in site function for many customers as well as greatly increased speed in moving chunks of data off as well as receiving reports that mail is functioning quite a bit better. That said this issue remains at the High severity rating and in unresolved status as we have not reached a normal level of service. I can’t stress enough how sorry I am that our customers have had to put up with this but I thank those of you who have stuck with us (check the newsletter for details on what we’re doing for Blingy customers) and look forward to providing you with the level of service we strive for at DreamHost.
Update, April 9th, 2008 21:59 PDT
Unfortunately, we need to unmount the volume again to kill these snapshots before they leave us with 0 bytes of free space. In 2 hours (midnight) I will be taking the problematic volume offline to delete the phantom snapshots. Total downtime will be between 10 and 30 minutes. Sorry for the short notice and additional outage!
Update, April 10, 2008
Well the snapshot mentioned yesterday is gone and we’re actually at 83% used today which is below where we were hoping to see marked improvement (85%). Of course we’re still moving data off (which increases the usage on the file server) so that won’t fully translate to customer usage improvement but it should be quite a bit better and keep improving until we stop moving data.
Update, April 14, 2008
Okay, we’re finally getting ready to mark this as resolved.. things have seemed pretty much okay for a while now. But, just to be sure, we’re dropping the severity to Medium for now and leaving it as unresolved.
Update, April 17, 2008
We’re still hearing some reports of site slowness - we were able to resolve an issue causing high loads today which should help but we’re not going to consider this resolved until everyone is receiving good service.
| Severity: | Medium | Resolved: | No |
March 26th, 2008 at 8:18 pm
Yeah e-mail at least for me seems to be working… AT THE END OF THE WEDNESDAY BUSINESS DAY. Thanks.
Anyways, agreed with others that not having any status or contingencies is like being kicked after you’ve been punched to the ground.
I had my e-mails forwarded to another address but that was still relatively slow and then the DH e-mail went backup tonight so I have some double e-mails, some unique ones. It’s wreaking havoc with my Mail program’s sorting service and making it really difficult to sort through messages.
This is insane. Started Sunday we’re not at end of Wednesday no, (real) response from DH. Need to get my duckies in order and then switching to Hostgator.
March 26th, 2008 at 9:36 pm
tick tock: 31900976 bytes left
March 26th, 2008 at 9:52 pm
email down on one host, up (but super slow) on the other….
March 26th, 2008 at 9:59 pm
Site is screwed here. Not impressed at all, just moved my site over with the intention to get away from this sort of crap.
March 26th, 2008 at 10:00 pm
@Arghhhh
“My request for the refund or to be moved from the Blingy cluster has fallen on deaf ears!
I received the same reply that one of the other members received 2 days ago!!!! WTF?”
Well, did they not respond to your request for refund? Thats sad..
One month back I requested them to change my cluster. But they said that they don’t have the technology to move my things from one cluster to another. They just moved me to another server within Blingy. For a day or two things improved, but it got even worse than my previous server that. My bad luck at DH did not want to leave me until I was here. Finally I changed over to another reputed host, for just $7.95 (I will try them for one month first) and I am happy there.
March 26th, 2008 at 11:09 pm
This is simply unacceptable. My sites are completely down again. I’m losing money, patience, and respect. This problem must be resolved. I’ve sent an email requesting a refund, this email was not returned. If someone is asking you for a refund, show them respect. Don’t ignore it, or put it at the bottom of the pile, a very large pile probably. This is terrible service.
March 26th, 2008 at 11:23 pm
Hi Shane,
Recently a few of my friends are facing the same problem. We were all at DH, and after much research we have shifted over to Hostgators. Some of my friends have signed up there for one month, for just $7.95, while they are waiting for DH to refund them their money. They will test Hostgators for a month before paying them more. If they are not good they will ask the refund for even that $7.95. So far the service there looks really great. You too may try the same if you want.
March 27th, 2008 at 1:47 am
Is Hostgators work in clusters?
1 server for web server,1 server for database,1 server for mail?
Thanks
March 27th, 2008 at 2:42 am
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 2071416 1325860 640332 68% /
/dev/sda3 7224600 2804540 4052976 41% /usr/local
/dev/sda8 2071416 88012 1878180 5% /tmp
10.175.4.69:/fcvol1/blingy/abbreviation
13711880192 13711880192 0 100% /home/.abbreviation
10.175.4.69:/fcvol1/blingy/wycliffe
13711880192 13711880192 0 100% /home/.wycliffe
10.175.4.69:/fcvol1/blingy/wynette
13711880192 13711880192 0 100% /home/.wynette
10.175.4.69:/fcvol1/blingy.localbackups/toadstool
13711880192 13711880192 0 100% /mnt/filerbackup
10.175.4.69:/fcvol1/blingy.crontabs/toadstool
13711880192 13711880192 0 100% /usr/local/var/spool/cron/crontabs
March 27th, 2008 at 3:00 am
Fixed? No…
It has been what… days?
damn you
March 27th, 2008 at 3:33 am
Hey, yeah this is becoming beyond a joke now.
I’m now unable to access mail that was previously accessible…. it is telling me my login details are wrong. I hadn’t changed a SINGLE detail. I then proceeded to change my password for 1 account, perhaps that would fix it?
No.
I’m also getting multiple 403 Forbidden errors across the couple sites I maintain.
Its just not good enough guys.
March 27th, 2008 at 3:43 am
Hi Sam, can I too would like to try Hostgator. I have seen a lot of people happy with Hostgator. One of my friend once told me that he got an account there for just $0.01 for the first month. Do you have any idea how I can avail that?
March 27th, 2008 at 3:51 am
@Queen
Hi, you can do the following to get your first month for $0.01
1. Go to Hostgator
2. Chose their baby paln
3. Signup there.
4. Use the coupon: freemonth
And try them for one month. Then you may stay there and pay their regular charges from the next month onwards.
March 27th, 2008 at 3:53 am
And yes, follow this link:
hostgator
March 27th, 2008 at 4:06 am
is there some irony to the fact that this whole shooting match shut down completely as I was transferring all my files to a different hosting provider? I am shocked at how bad DH has been in the few short months I have been with them. DH came highly recommended and I moved EVERYTHING (all 5 of my sites) over to DH. Now, nothing works except my email which I routed to Gmail services a few weeks ago (because of these very problems)
March 27th, 2008 at 5:08 am
@Sam,
Ahhh - ok so asking to be moved to another cluster is useless! Well time to jump ship…..me thinks.
And Im going to ask AGAIN for a refund. I paid 12 months up front - along with probably everyone else!
I looked at your new host - what are the loads on their servers like? Hey, at this point Id be happy with anything under 10! LOL
Im either going to go there or jump onto a friends Dedicated server at Liquidweb.
I just wish that DH would update us a couple of times a day….then at least we would KNOW what was going on.
March 27th, 2008 at 6:16 am
My new load is 0.25.
March 27th, 2008 at 6:17 am
I wish people from HostGator would quit spamming this board. HostGator sucks just like all the rest of the cheap webhosts. It appears to me that people from Hostgator are posing as dissatisfied clients of DH to take advantage of this situation. If one does a simple search using ‘Hostgator sucks’, one will find plenty of dissatisfied customers there as well.
The fact that it appears that employees of Hostgator are posing as clients on this forum causes me to absolutely NOT consider using Hostgator. Doesn’t anybody find it just a little strange that almost all the posts on this board regarding other hosts recommend Hostgator? FUCK YOU HOSTGATOR, AND THE SWAMP YOU CAME FROM.
I’m sticking with DH for the time being, but if I do choose to change hosts, HG will NOT be my destination.
March 27th, 2008 at 6:59 am
My website is still down - when I attempt to navigate to it, it replies with a 500 internal server error. This past week, it was just extremely slow, but would still load after approx. 3 minutes, now it’s completely down:
http://www.divethegreatlakes.com
I will have to check when I signed up with them and search for something different.
Jerry
March 27th, 2008 at 7:04 am
I second that! I came to DH from HostGator bcs they were terrible, If you think DH is bad definitely stay away from TurdGator. iPower is another one to steer clear of, their CGI is a nightmare. All of these low cost hosting plans run on the same short fuse so you basically just end up crossing your fingers and hoping that you get on a good machine.
March 27th, 2008 at 7:54 am
Here are a few searchch results
“Dreamhost suc*ks*” 4,480 results
“Bluehost suck*s*” 540 results
“Hostgator sucks**” 662 results
March 27th, 2008 at 8:00 am
@Jkono
“All of these low cost hosting plans run on the same short fuse so you basically just end up crossing your fingers and hoping that you get on a good machine.”
I think dreamhost is the cheapest. And I guess(just a guess) dreamhost people go over the net and defend themselves. The typical defence is “I am at dreamhost since so and so many years. They are great. They provide so much for so little”
Thats cheap!
Damn it man..
dreamhost is frustrating me since past 2 months. But still I see people coming here and claiming that dreamhost is great.. how cheap you DH employes are.
March 27th, 2008 at 8:12 am
@Arghhhh
“I looked at your new host - what are the loads on their servers like?”
I have noticed it to fluctuates between 0.1 to 0.8. Usually it is around 0.2
“Im either going to go there or jump onto a friends Dedicated server at Liquidweb.”
Wish you good luck!..
March 27th, 2008 at 8:15 am
Something tells me “Sam” works for Hostgator.
Regardless, this downtime is just plain stupid. This has been going on for a week now! A brand new server could have been built and all files could have been copied in less time than it has taken to fix this cluster.
March 27th, 2008 at 8:15 am
I’ve been hearing a lot about hosts that suck, but what about decent ones (and I already know gator isn’t), they don’t neccessarily have to be cheap, but I don’t want to spend too much right now, my site is just starting to get a user base (though I’ve already heard from a few that I’m too slow and that such and such is much faster). This other guy I work with uses my site to display what can be done with LAMP and google maps (I get the work), but he’s complaining that it’s too slow and I need to do something or he has to use someone else.
It’s getting really frustrating
March 27th, 2008 at 8:44 am
well now its hard to know who to believe! LOL
Im in Aus, so I might just try one of our hosting co’s instead of US based ones.
I just cant keep my site and 80 regular users going like this.
March 27th, 2008 at 8:59 am
Spamming the boards by other hosts is definitely a turn off to me, but c’mon DH. This is happening way too much. Every time you turn around there’s an issue. I don’t want to leave, waaaaaaaaaaaaaaaaayyyyyyyyyyyyy too much work involved, but I also can’t afford to lose my clients because of downtime and shit just flat out not working.
They don’t understand what goes on. All they see is that their sites aren’t loading and lost revenue. This really sucks.
March 27th, 2008 at 9:48 am
ERROR: Connection dropped by IMAP server.
Query: FETCH 20:34 (FLAGS UID RFC822.SIZE INTERNALDATE BODY.PEEK[HEADER.FIELDS (Date To Cc From Subject X-Priority Importance Priority Content-Type)])
I keep on getting this error from DH. Its Been a wk now and nothing seems to be workinga nymore. DH, do something before i change from you guys to somwhere else. It sucks too bad DH. work on it quick.
March 27th, 2008 at 9:48 am
Good point, Hostgator is almost as cheap as Dreamhost and weird it’s like the only hosting provider getting mentioned.
This’ll look like spam too, but to those actually looking for a decent webhost a site a big site I contribute to uses (mt) Media Temple and I’ll likely try them. Have not tried them so I’m not sure but they seem to have big clients, good interface, seems reputable, etc.
None of this “buddy-buddy we’re sorry crap”. It’s fine to get all casual and chummy but not when people are pissed and the service has gone done.
March 27th, 2008 at 10:31 am
I have a question…since this has turned into the “this host sucks” board…Has anyone used godaddy for hosting? I am building a site for a client and they are currently on a godaddy account. I can’t find any control panel or anything there… I see their account info..but no buttons or levers to pull…BTW, my other 2 hosting accounts with DH are running fine and have been for 2 years.
March 27th, 2008 at 10:41 am
what idiot would upgrade the hard drives without coping the data first? Guess that will leave some of us without a website till all the data gets copied over.
March 27th, 2008 at 11:32 am
@Kal
“Good point, Hostgator is almost as cheap as Dreamhost and weird it’s like the only hosting provider getting mentioned.”
Because I was the only one talking about it. I am trying HG and its way better than DH. Thats the only point I am making. I didn’t do much research on Mediatemple. But that too looks good. By far the least sucking** results is for media temple.
Google search:
“Media Temple Suck*s” Just 183 results
“Mediatemple suck**s” Just 84 results.
Even if we add both the results its less than 1/2 of the “Hostgator Suc*ks” or “bluehost suck*s” results. I will try Mediatemple too.
March 27th, 2008 at 11:35 am
well, i am still stuck here.
I tried a couple other services, but for some reason I still preferred the DH way (This IS CHANGING NOW THOUGH) :@
This is too much, server goes down every 3 hours? silly thing.
March 27th, 2008 at 11:42 am
OMG!
I just noticed something ladies and gentlemen!
The NightmareHost website does NOT mention anything about their uptime! ROFL
yes! Now we should have noticed that. 99% of the hosting services i visited declare that they guarantee 99.9% uptime!
guess what? DH doesn’t say this! which means? They take 12 months to fix one problem
March 27th, 2008 at 12:15 pm
yhs, I’ve done some asking around on the 99.9% ‘uptime guarantee’ that many hosting services offer. For the most part, I’ve found this to be a marketing gimmick. It seems no one has a standard policy in the event that this uptime guarantee is not met (partial refund, etc). The only option would be to ‘cancel at any time’ - so you’re really not gaining much.
March 27th, 2008 at 12:25 pm
anyone heard anything on when this might be fixed? They said a few days a week ago!
March 27th, 2008 at 12:30 pm
WHAT THE FUCK.
March 27th, 2008 at 12:31 pm
From the status update e-mails:
“Some of you are also asking when this issue will be fixed, I wish I had concrete information for you, but if I had to estimate I would say by the end of the week. Unfortunately I can not guarantee the issue will be fixed by then, but we are focusing on this issue presently and that will go a long way to us finding a solution.”
March 27th, 2008 at 12:33 pm
Thanks Loni. Top notch customer service they have. I read that on Monday. No updates from them since then? Thanks alot DH.
March 27th, 2008 at 12:35 pm
Maybe I am looking at this wrong, but my website has been up nearly 100% since I moved here, and although my e-mail is down at this exact moment, my service has been good.
Any webserver is going to have the same kinds of problems, no matter what they promise. Hardware breaks, DDOS happens, and networks become overloaded.
This is not to say everything is peaches and cream, I am very irritated that my e-mail is down now, and has been very sporadic for more than a while, what I am saying, the only perfect server is a piece of cardboard on a stick in your front yard, but then you have to worry about rain and snow.
March 27th, 2008 at 12:41 pm
Don,
I agree that downtime happens - it’s to be expected and as someone who used to be in IT, I’m glad I’m not up all night for days in a row fixing this problem (I assume that some folks are).
greater than 1 week downtime for email must involve the longest sustained server room fire (no really, there has to be a fire department involved) in the history of the internet. When we finally get the update, it must be interesting.
March 27th, 2008 at 12:55 pm
I thought I was good…main domain is fine…I just tried all my subDomains and a redirect domain…down…ugh.
It’s times like this that some people can take time off…But others must work extra…I think I’ll take another break.
March 27th, 2008 at 1:00 pm
I dont have a problem with downtimes,
Afterall, we are all humans and shiz happens. BUT, having a downtime of a week AND not having an update status AND all you get as reply from support is an autogenerated message….now ALL that together is very irrtating.
Oh yes, and all these emails on my Gmail account from people asking me why? where? when? how? and all these OMGs drive me crazy.
Well, the least they can do is shove us all to different clusters whilst this one is being fixed!
March 27th, 2008 at 1:23 pm
@yhs
AMEN! Why aren’t they putting us on different clusters? Maybe there’s a good reason, but until they tell us, WHY DH? This isn’t like the power going out for a little while and missing your favorite TV show. A LOT OF PEOPLE ARE LOSING A LOT OF MONEY!!!
I wonder how much money DH is losing? A few people who ACTUALLY DO close their accounts?
March 27th, 2008 at 1:30 pm
they just added this to the critical announcement for me:
Note: We are doing emergency data moves to quell the stem of problems recently caused by your file server. During these moves, your data may be inaccessible. We are moving as we can off as fast as possible. Very sorry about the continued inconvenience!
March 27th, 2008 at 1:42 pm
Companies may lure you in using the promise of “bargain” web hosting, but how much of a bargain is it in real terms? Your savings up front come out of what an honest, reputable company would invest in things such as quality equipment which is frequently upgraded, thorough and seamless system redundancy, pro-active maintenance, fast and dependable customer service, among many other aspects that today’s webmasters require. Once they’ve accepted your payment, you’ll quickly discover that the lack of such necessities results in lost traffic, lost data, lost customers, lost income, and adding insult to these injuries, possibly even a lost mind while you wait, enraged, for days on end to get anything that could be even loosely termed “support.”
When you’re ready to sacrifice that comparatively small initial savings for the sake of finally seeing all those profits you’ve been missing out on instead, you won’t even need to waste still more time just searching for a reliable host, because you are already only a few easy clicks away from having an Astonishing Host! Our shared hosting plans start at $34.99 per month, but for the additional investment you get:
- Unmetered (often misleadingly labeled “unlimited”) Multi-Homed Premium Bandwidth!
- 5,000 MB Disk Space on State-of-the-art Hardware!
- 1,000 MB Guaranteed RAM (1450 MB Burstable!) for EVERY site, using our exclusive, patent-pending technology!
- 99.999% Uptime SLA with triple refund of that month’s fees!
- The stability and security of our newly-built datacenter equipped with strict, biometric access controls and multiple alternate power sources and cooling systems!
- Industry-leading redundancy and failover capabilities!
- 24/7 Telephone support from our dedicated USA-based call center staffed by only Level 2 and Level 3 Technicians, in addition to guaranteed 30 minute maximum email support response time!
- 4 hours of FREE consultation time with one of our award-winning web designers for the next 100 signups, so act quickly!
- Much, MUCH more!
STOP losing business and join the winning team at http://astonishinghost.com/ today!
March 27th, 2008 at 1:42 pm
It seems to me, that after a few hours on day 1 that ‘emergency data moves’ would take place. This is an echo of Preston comments on moving folks to servers that work.
March 27th, 2008 at 1:47 pm
people with private servers should have priority in this. We should be the first to have our sites fixed since we pay more than any of the shared hosting people
March 27th, 2008 at 2:18 pm
Here’s a little trick when researching new ISP’s:
google “new hosting service name sucks ” and see what you come up with
I’m thinking of switching to AN Hosting , I googled “AN Hosting sucks” and not very much came up. In fact, the first link I clicked on was a former DH customer ! He seems pretty satisfied.
Like my friend’s mom used to say, “If it happens once, shame on them, if it happens twice, shame on you.”
Not much use in complaining about DH anymore — I think they grew to fast for all of our own good.
March 27th, 2008 at 2:49 pm
Warning:
Look at the time since when the hosting company had been existant. Just the number of “thishost sucks” doesn’t give you any concrete info. Anhosting is a relatively new name. Be careful.