Lovejoy and Sanjay File Server Issues
A file server that hosts files for a small number of customers, mostly on the lovejoy and sanjay web servers, is having issues at the moment. Our admins are working to have the file server check its system and disks, which must happen before it can go live again. This is causing website and email downtime for the customers on these two web servers. We are working to have this fixed as soon as possible, when we have further information we will update this post.
Update - 03:45 PM PST: Recovery efforts are still moving forward. At this point it will take at least 16 more hours for disk checks to complete. And that’s in a perfect world. Please accept our most humble apologies for any and all problems this is causing the affected users of lovejoy and sanjay.
Update: 1:47 pm Pacific: The file server is back up and running. No file should have been lost, please let tech support know if anything seems amiss! We’re sorry it took so long to get up, once the filesystem checks finished the machine did not want to boot back up into its operating system.
Update - July 22nd 04:59 PM PST: Unfortunately the problems with the file server returned again. We are starting up another file and disk check on it. We will be watching the check closely to make sure that we do not miss any problems that we possibly missed. Our apologies and we will get things up for you as soon as we can.
Update - July 23rd 11:27 AM PST: The best update for this specific issue is the new post regarding the scooter file servers emergency OS upgrade:
http://www.dreamhoststatus.com/2008/07/23/emergency-file-server-os-upgrade/ (Canceled)
We had continued issues with the file server used for lovejoy and sanjay so we have decided to get your files moved to new hardware. As explained above the upgrade is expected to take a day, then after that is done we will start on the moves. We will limit downtime as much as possible.
Update - July 23rd 12:58 PM PST: As you may have noticed the plan with OS upgrade above was canceled. We have decided to add a new unused file server to our data center and move the data to that one instead. This will limit the number of people experiencing downtime, but will keep pretty much the same time line for those of you on Sanjay and Lovejoy. So the new hardware does need to be upgraded still, then we will start the data moves.
Update - July 24 7:20 PM PST: Data moves should be starting tomorrow to the new file server. We will do the moves on a user by user basis, so once the data copies over your user will be restored. If we have a change in plans we will update this post again. Sorry for the continued issues for those of you with users on this file server, we will make sure to get you a credit for the downtime.
Update - July 25 5:01 PM PST: The data moves to the new file server have been running since around noon. We are trying to get a good estimate for all of you affected. Unfortunately the estimate for the move is next Friday. If you have email addresses that are vital set up a new username and set the mailbox to that user. The user will not be on the problem file server. Also if you have a domain you need up and you have the files locally, you can switch the user for the domain and upload your files to get your domain running. I know this work around does not work for everyone but some users were happy with that solution while we finish dealing with this issue. Our sincere apologies for the downtime this has caused.
Update - July 31 11:47 AM PST: Unfortunately nothing new to report. Moves are still running successfully at this time. A lot of you have open support tickets and I will be getting to you one at a time to see what options I can give you. In some cases your data has moved and I have to run some manual updates to get things working for you. Hang in there. I’d have to say the estimate for all of the data completely moving seems to be taking longer than expected. Each user on this file server have tremendous disk usage so the moves take hours to complete for one user. If you have an open ticket I will do my best to get to you today.
Update - August 1 10:50 AM PST: So far the move has completed 36 moves with around 59 more users to go. We did unfortunately have a hiccup during the previous move which set up back about 24 hours. Now the moves are running well. Since each user account takes a various amount of time it will be difficult to give you an accurate ETA. What we’ll do is post how many users are left to move at least twice a day.
Update - August 1 6:20 PM PST: 51 users left to go. We can mount up some month old backups for you if you really need your files. If you are in a situation where you just want access to your files we’ll set up a user.backup user for you which will have the same password. The data will not be live, but will be accessible. If you would like this please send a support message to dhstatus@dreamhost.com. You must send from the primary contact email address on your account.
Update - August 2 2:05 PM PST: 50 users left to go still. A really large user is moving right now as well as two others.
Update - August 3 2:20 PM PST: 45 users left to move.
Update - August 5 11:04 AM PST: 36 users left to move. Sorry for the missed updates yesterday.
Update - August 5 7:54 PM PST: 34 users left to move. There are currently 4 moves running.
Update - August 6 1:11 PM PST: 33 users left to move. 4 moves are running.
Update - August 6 7:05 PM PST: 32 users left to move. Some of you have reported that the email work around is still causing a bounce even though you are receiving mail at the new location. You will actually just need to set the address as a forward only. Once the move is completed you can switch back to a full mailbox. We have 4 moves running.
Update - August 7 9:24 AM PST: 27 users left to move.
Update - August 8 9:56 AM PST: 23 users left to move.
Update - August 8 6:07 PM PST: 22 users left to move. Two moves are running currently. We have been able to get the sites working that have moved. Sorry this move is taking so long, we have the machines plugged directly into each other and the moves are going much slower than we typically see. It is possible that the problems that caused the stability issues are somehow causing the performance issues with the moves. We’ll keep going though. Our sincere apologies and credits will be issued once this issue is finally resolved.
Update - August 9 9:42 AM PST: 19 users left.
Update - August 11 11:16 AM PST: 1 user left to move. User actually already moved but there was a problem so we have to remove them in order to ensure that everything is okay. Once that user moves we will need to fix some mail mounts then issue credits. At that time the problem will be resolved. If you do notice problems right now please contact support for assistance.
Update - August 11 6:28 PM PST: Okay we have finished everything and it should all be working. If you notice any further problems please contact ralph@dreamhost.com. Make sure you write in from your accounts contact address. He’ll fix anything left over. Credits will show up on your accounts by this time tomorrow.
Update - August 12 10:36 AM PST: Thank you to those of you who have contacted me directly. We are still having some problems with mounting some user directories for mail only. I am working with our administrators on this issue with the examples you gave me. I will update as soon as we have this resolved or have more information.
Update - August 12 5:36 PM PST: We feel confident that this problem is resolved now. All email is being reported as working now and credits have been issued. You can see the credit from the Manage Account->View Invoices area of the DreamHost Webpanel. It is an Account Service Credit of $119.40 paid on 8-12-2008. Sorry this took so long to resolve and if you have any further problems just let me, ralph@dreamhost.com, know.
.
July 21st, 2008 at 11:49 am
15+ hrs after ‘my’ inquiry into my site down and no response. I now see this post. It sure would be nice to actually get a response to support tickets.
Thanks for looking at the issue guys/gals.
July 21st, 2008 at 12:24 pm
i noticed my site down last night. It’s a little more complicated than just a reboot though and i know dreamhost is working hard to resolve our issues. Thanks guys/gals from me too : )
Although, the only client screaming at me is the one not on this server, but whose site i am developing on the server. have moved it elsewhere for now.
long live lovejoy!!!
July 21st, 2008 at 1:00 pm
my support request went in last night and I got a response in the AM today. The email problems resulted in mails bouncing
I’m glad they are working on it now but it would have been nice to have seen this sooner..
July 21st, 2008 at 2:40 pm
It would be nice to get some sort of report back from support about these two servers, which unfortunately for me host ALL my site. I’ve yet to receive any response from support.
July 21st, 2008 at 3:31 pm
I’m in the same situtation. My email and all my sites are down. I’ve had a support request open for 12+ hours so far but still no response at all from dreamhost.
July 21st, 2008 at 5:07 pm
Every couple of months, I’ve had something go wrong to take my site down. Usually, it’s been quick to repair, but it’s always been the result of a mistake on Dreamhost’s part. A full day (or more; they don’t sound terribly confident in their 16-hour estimate) is simply unacceptable when I need the site up to conduct business. I’m looking for a new host that can handle Miva Merchant.
July 21st, 2008 at 8:38 pm
I am so sick and tired of this…Starting to regret that I signed up with Dreamhost. Who knew they where the same crap as Sewage (servage..) ??!!
July 22nd, 2008 at 4:36 am
so, my two sites will be down for 2 days or more? and you will issue credits when…oh you never do that you just humbly apologize for running a crappy hosting operation.
July 22nd, 2008 at 5:32 am
Not sure why anyone is bothering to post here– they don’t consider customer service a priority– why would they read forums?
Will make a great case study for my new book….
July 22nd, 2008 at 6:15 am
To be honest randy, the key to customer service is communication. If your website does not work and you sent an email and got no response on their emails/support ticket/phone/website. You have more things running through Your head to figure out what is going on. is the owner on holiday, did they go bust, mpaa riaa fbi cia mi5/6, h4x? is your data safe?
Communication is key to customer service, even though they have not being able to respond to your email about your account not working, they have at least published information in their status blog stating that there is a problem specific to what area and who is affected and whats going on.
They have let their customers know about this webpage and encouraged people to look here first then submit a ticket if the problem has not been stated. At least Dreamhost gives out more details about the problem, most likely the cause of the problem and so on, unlike smaller companies that try to keep it underwraps.
July 22nd, 2008 at 8:04 am
it took almost a day for this issue to even show up on dreamhoststatus. now that it has shown up, it’s going to be another entire day.. can’t get to my mail, can’t get to any of my sites, can’t get to any of my files. it’s really true, dreamhost is just kind of a crappy hosting operation… they clearly don’t have sufficient redundancy or failover mechanisms in place. sure a lot of the neat features are cool, but why is it that once a week there is some screw up that causes some of my features to go down? seriously guys, it’s like once a week! many hosts guarantee a certain amount of uptime.. dreamhost falls far behind..
July 22nd, 2008 at 9:04 am
This is turning out to be a VERY expensive run of downtime for me… Can I send an old 486 with about 5 years of uptime to replace the server they have me on? I promise i’ll put it in a rack mount!
I’d gladly trade fast hardware and blingy features for less downtime. Of course, I respect any efforts on the behalf to repair issues such as these.
However, it is the wonderful hosts that do the maintenance, and hire the competent people, that can eliminate these issues before they manifest themselves in downtime.
-Aaron
July 22nd, 2008 at 10:04 am
just thought i’d pipe in again
- they consider sites of several of their customers being down for more than 24 hours to be severity…..MEDIUM…yep as soon as I can find a new hosting service and getting it up and running I am gone for good.
July 22nd, 2008 at 12:22 pm
Gonna have to agree with you Jim. I’ve already lined up a great host, and since my DH update is coming up soon, they won’t be getting another penny from me. Bye Bye…
July 22nd, 2008 at 2:10 pm
Now I get “Your website does not appear in the Apache configuration file.” errors for all of my domains now that the Lovejoy problem is in “resolved” status.
Also, I initiated a P.S. reboot, and it’s been “pending” for over 20 minutes. I’m just posting this so that anyone else having the same problem can know that they’re not the only ones.
July 22nd, 2008 at 4:32 pm
yup, still having problems with both email and website
July 22nd, 2008 at 4:44 pm
Same here, email and websites still down.
July 23rd, 2008 at 2:09 am
Why don’t these arseholes restore the two servers on a different machine from their backups, and put them live while they fuckwitts figure out whats wrong with their cheapass servers?
July 23rd, 2008 at 2:22 am
Krang, I guess the only backup dreamhost keeps is in the form of netapp filesystem snapshots. So as long as the filer is down they don’t have access to it either. We just have to hope that our files are still there when they manage to get the filer back to life.
July 23rd, 2008 at 7:51 am
This is getting ridiculous. Three days of down time is not acceptable. I like Dreamhost, I’ve hosted with them since ‘99, but this is unacceptable. Time to start shopping hosts. Ugh…
July 23rd, 2008 at 8:34 am
yeah .. like.. i dunno if you guys know this but.. professional web hosts don’t resolve a problem like by making their customers wait for a file check.. the normal way is to be able to restore the server onto a different machine while you fix the original one. how come dreamhost doesn’t follow this type of methodology?
July 23rd, 2008 at 8:34 am
what the fuck is wrong with you people? fucking 2 god damned days without being able to bring your shit back online? (ie all my email and sites) anyone have any good idea for alternate web hosts?
July 23rd, 2008 at 9:01 am
Look at hostmonster.com. Cheaper and unlimited bandwidth and capacity. Pretty highly rated too. Dreamhost is gonna have to pay through the ass to recover from this fuckup.
July 23rd, 2008 at 11:23 am
i agree — this is extremely frustrating, and I’m completely mystefied as to what could be going on. a little more communication from support would definitely be appreciated… oh well.
here’s my question — my issues have flip flopped back and forth - first one problem, then a partial fix, then a new problem, then back to the first problem. anyone else experiencing this same kind of thing? I can’t possibly understand how some of my files can reappear, but not others… sometimes the directory structure will reappear, but with no files (except for in the ~/logs dir). anyone?
July 23rd, 2008 at 11:37 am
this is silly. a host with so many filer problems has to think things over.
July 23rd, 2008 at 1:25 pm
I’ve seen the DH server’s in Alchemy’s data center in LAX, and am fairly impressed with their operation. I trust that there’s a stack of people in the NOC right now attending to this issue. Though, I find it discouraging that this is a “medium” issue. All of my sites are on lovejoy. One user account is up 100%, and one user account is down 100%. Not terribly happy about this and wish their was a quicker resolution, but I’ll just wait it out.
Josh, I’m not leaving, but do you have any assurance that shit like this isn’t gonna happen again?
July 23rd, 2008 at 1:48 pm
the best way to make up for it is to upgrade their shitty disaster recovery operations (a little fucking redundancy and decent backups would help). and of course giving all customers affected free hosting for a couple months…and an upgrade wouldnt hurt either…barring that im fucking gone
July 23rd, 2008 at 7:59 pm
Guys — i gave up waiting and created forwards for all the mail addresses that are dead ending right now. That way, I at least know who is trying to communicate with me.
In the odd chance that some of you haven’t done that yet — or don’t know how to do it, edit the e-mail user and add an address in the “also forwards to” box so you don’t loose business like I did the first two days things were down.
-Aaron
July 23rd, 2008 at 9:26 pm
how da hell is this a medium issue… im loosing my clients fate because of this BS which is categorized as MEDIUM.. faaaaaaaaaaak!!! come on.. please lovejoy rulez..
July 23rd, 2008 at 10:14 pm
One of my clients waited for ftp access and gave up. Few days ago, the MySQL issue frustrated him for a whole day as well. 3rd strike, he’s out =(.
July 24th, 2008 at 4:15 am
and what fucking timeline may that be, jackasses?
July 24th, 2008 at 6:08 am
I got a response to my support ticket within a few hours and pointed me to this issue. Just a data point.
July 24th, 2008 at 6:50 am
Can we please have an update on whether the file transfers to the new hardware started yet? What is the ETA?
July 24th, 2008 at 7:12 am
5 Days down now.
July 24th, 2008 at 11:29 am
this is ridiculous. at least give an ETA. when the hell will these servers be back up so that i can transfer everything off of them and off of your fucked up hosting company. i mean, ive been with you guys for almost 10 fucking years, and its come to this?
July 24th, 2008 at 12:31 pm
i have to aggree i have been here aswell for almost 4 years now , and now this.. come on , dont fuck it up.. give at least info on wtf is going on if you are working or not.. weee demaand ETA!!
GALD DAMIT!!
July 24th, 2008 at 12:50 pm
No email, no site access. How would you guys feel if your email was broken for more than 4 days? We should be getting updates every couple hours, not every 24. I’m now in the “find a new hosting service” camp.
July 24th, 2008 at 2:43 pm
I can see why they recommended we all move to Gmail.
July 24th, 2008 at 10:14 pm
The Simpsons servers are cursed! My last server was Homer. It died. Now Sanjay and Lovejoy are chomping gravel.
I give DreamHost credit for two things
1) Massive storage
2) Allowing unmoderated comments on their status blog.
Look at the vitriol posted! It is brave to allow anonymous and unmoderated feedback from people under stress on a company website.
I bounce back and forth on DreamHost. The reliability is abysmal, truly awful. But the control panel, customer service, and storage & bandwidth are solid. I agree they don’t post issues fast or often enough on dreamhoststatus.com and that drains everyone, but they almost always reply to support inquiries quickly and with an answer (maybe not a solution, but an explanation).
DreamHost is like that one old friend: Great People. Totally Unreliable.
July 24th, 2008 at 11:35 pm
Come on, we signed up Dreamhost because it’s dirt cheap. This level of reliability is what the market offers at this price point. They can go down for 4 days while still remain in business because the supply/demand allows for it. I doubt you’ll find a more reliable hosting with guaranteed uptime at similar prices.
At least they’re willing to give us back credit for the downtime.
July 25th, 2008 at 5:42 am
Cicada speaks the truth. But I’ll note that even in their $395/month plan, “email and databases are still hosted in a shared environment.”
July 25th, 2008 at 7:33 am
Agreed, for the price it’s a deal.
However the better question to ask is, is it worth it? Is it worth sacrificing reliability to save half the cost? Is a full business week of downtime (and counting) worth skimping on the price? Or, would it be worth paying an extra $100-$200/year to know there some redundancy in place that will protect you from a hardware failure.
For a personal site Dreamhost might be great, and I might be able to weather the downtime, but if I have paying clients relying on me to have their sites up, saving a couple hundred bucks (when much, much, more is at stake) is no excuse.
July 25th, 2008 at 7:49 am
I left DH a long time ago, but I thought I’d stop in and see if things have improved. Clearly they have not - still the same shit that they’ve been pulling for a couple of years, and they’re not getting any better. My advice to you all is to get off DH servers because it isn’t going to get any better than what you have now. Personally I would not recommend Bluehost / HostMonster though (same company) - been there, done that. Media Temple are good.
Good luck - all your bitching and moaning here will have absolutely zero effect. Either leave, or shut up and accept that this is the type of ’service’ you will always get from DH.
July 25th, 2008 at 7:50 am
Oh and if you’re running a web-based business on DH, you’re a friggin idiot and are destined to fail.
July 25th, 2008 at 9:30 am
I just wanted to take a quick break from all of my usual vitriol directed towards dreamhost for not having email for 5 days and wish the dreamhost gang a happy sysadmin day - http://www.sysadminday.com/
July 25th, 2008 at 10:24 am
As previously noted, it is a bit unbelievable at their size they don’t have better back up.
Like others here, our business is growing– we are about to launch a program for 3,000 sites. And we will strike our agreement with 1and1 or Powersurge, not Dreamhost– even though we’ve been here for 5 years.
July 25th, 2008 at 1:45 pm
If anyone at dreamhost is reading this….can we get an update?
July 25th, 2008 at 3:03 pm
Oy! twats, can you bring my sites back online so I CAN BLOODY MOVE IT TO ANOTHER HOST!!
July 25th, 2008 at 3:57 pm
so, have they sent nasty responses to anyone else yet, indicating that they dont appreciate your tone and will cancel your service within 30 days? or just me? thats nice — piss off long-time reliable customers and then tell them to fuck off and go to a competitor…
July 25th, 2008 at 8:33 pm
Seriously? One full week to transfer information to another filer? Pathetic. What are you doing, transferring via 56k? How can you even be satisfied with yourselves? If it took me a week to transfer to a new filer, I’d kill myself.
July 25th, 2008 at 8:52 pm
Here’s you an update:
All of your sites are being smited by The Lord for being shitty. None of them glorify Him (either directly or non, either of which is fine).
VOO
July 25th, 2008 at 8:56 pm
DH really sent you a response threatening to cancel your service because they didn’t like your tone when you complained about 5 days (and ongoing) with no service that they were paid to deliver???
ROFLMAO like how is that going to make any fucking difference if they cancel it - their ’service’ ISN’T FUCKING WORKING ANYWAY! OMG these DH asshats really have got their heads stuck firmly up their rectums;
“How dare you peasants complain about paying for hosting and not receiving any hosting! Complain once more and I shall smite you from above! Fear me, for I am the omnipotent DH!”
….well, I heard it’s hard to work in a group when you’re omnipotent (Q), now I see DH is proving that to be fact.
DREAMHOST YOU’RE A BUNCH OF COCKSUCKING FUCKWITS WHO COULDN’T FIND A FUCKING COATHANGER IN A MACY’S DEPARTMENT STORE, EVEN IF 10 OF THEM HAD BEEN SHOVED UP YOUR ASS! GO FUCK YOURSELVES YOU USELESS BUNCH OF FUCKHEADS.
There, that’s more like it.
July 25th, 2008 at 8:57 pm
Oh hello VOO, nice to see you.
July 25th, 2008 at 8:59 pm
almost forgot - WOOF!
July 25th, 2008 at 9:23 pm
Hello AD, nice to see you too… been a while ain’t it? I’m glad to see that you’re still around cause even though we are working different sides of the Yin Yang it just wouldn’t feel like old times without you!
July 26th, 2008 at 3:55 am
even their workaround for email isnt working. sure, i can create a new email box and its on a different server. pity its not receiving any messages though. even email forwards arent working half the time.
July 26th, 2008 at 5:43 am
DH will be lucky if anyone is left once they final active the new server– who in their right mind will stay for a two week outage.
What are we in– the dark ages of the net?
July 26th, 2008 at 8:46 am
Blingy again. Same problem o the other cluster/file server
Expect solution withi 6 months!
July 26th, 2008 at 8:53 am
Blingy again. Same problem o the other cluster/file server
Expect solution withi 6 months!
5 months - problems arised end of Feb. Now is end of July
Former Very Happy DH customer
http://www.dreamhoststatus.com/2008/03/27/filer-problems-with-blingy-cluster/
http://www.dreamhoststatus.com/2008/05/21/blingy-problems/
July 26th, 2008 at 9:08 am
My question is why is this issue still a yellow, not a red? If sites and emails being down for 2 weeks doesn’t constitute a red, then what does?
And yes, if you have “clients” that your paycheck depends on and picked Dreamhost as your host, then you’ve made a pissed poor business decision and your clients can rightfully blame you. Dreamhost can afford to lose half of their clients. Can you with half of yours?
July 26th, 2008 at 6:35 pm
Because it’s only affecting those with shitty sites to begin with. If any of His servants sites were down then it would indeed be red, and it would also be fixed within about an hour tops. So instead of your constant bitching at Dreamhost, why not bitch at the ones who are truly at fault… your parents for not raising you right, and yourselves for not overcoming your raising?
Also, The Lord frowns upon those who use the word “clients”… He finds that shit to be pretentious and would prefer you to humble yourself back down to Earth and just call them customers.
July 27th, 2008 at 4:59 am
Been with Dreamhost for 5 years and despite the constant ups and downs in terms of reliability, I’ve stuck around but this week’s outage is going slightly overboard, despite the price we pay. I was looking at iWeb (not Apple’s iWeb) who seem to promise 100% uptime (don’t know how they can promise that, sounds like a bunch of crap). Does anyone know about them? Any experiences?
July 27th, 2008 at 10:36 am
VOO - do you really fancy yourself as funny?
What I have learned is a very simply - DH is not a place to host serious business sites of any kind. Fun little sites that real businesses do not depend upon sure, anything that someone expects to be up most of the time no way.
I work with other hosting companies and yes downtime is a reality. But with the others (similar pricing to DH so let’s take that silly argument of the equation) being down for an hour or two at a time is considered severe. They don’t apologize for it, they fix it and get us back up and running and then explain what happened after the fact.
I just don’t see any reasonable argument for 2 weeks of downtime. I’ve been in the IT game for over 20 years and I just really can’t come up with a technology scenario that would require this much downtime to fix a problem. Something other than technology (use your imagination I don’t want to be accused of starting rumors) becoming a very likely possibility in my opinion.
The the DH (if you read this) please just keep using your best efforts to get us back up and running….PLEASE we are begging you.
July 27th, 2008 at 11:07 am
OPEN MESSAGE TO DREAMHOST:
I’ve been with you for more than 5 years too and I don’t want to leave, but please give me a reason not to leave, pleeeeaaaase!!!
July 27th, 2008 at 1:38 pm
There is nothing funny about The Lord’s business… when He gets pissed things can really get shook up good.
The Lord has some of His sites here, and I can assure you that they are very, very serious. Do you know the lyrics to RUN DMC’s King Of Rock? Well, THAT’S shit that only The Lord can back up, brother.
July 27th, 2008 at 7:16 pm
Geez! First the payment issues and now this? I hope you get your act together DH.
July 28th, 2008 at 6:50 am
Dreamhost should really get their file server act together.
Whatever technology they use the really need to get failures sorted quickly. This situation is gettings really ridiculous.
We really need to know what technology they use and why they can’t fix problems quickly. If it is inherent in the technology then it might be better to use other providers whose systems allow better recovery from problems.
What is it about their filing systems which makes them so problematic?
I can login via ssh and see my files, so why I aren’t the web servers working?
July 28th, 2008 at 7:13 am
…log in and see your files…cant even do that…
July 28th, 2008 at 10:01 am
Can it be true?? Is Jim’s blind unfailing zealous devotion to DH finally starting to crack?? Say it isn’t so!!
’bout time you woke up Jim, you big dork. How long did it take you - 6 months? Dumbass.
July 28th, 2008 at 1:16 pm
Issue pending since July 21st?
I’m starting migration to MediaTemple, now!
July 28th, 2008 at 3:59 pm
“Also if you have a domain you need up and you have the files locally, you can switch the user for the domain and upload your files to get your domain running. I know this work around does not work for everyone but some users were happy with that solution while we finish dealing with this issue.”
the above doesn’t work…
I requested to move one of my sites to another webid this morning and haven’t heard back from support. Can we get an update?
thanks
July 28th, 2008 at 4:16 pm
-> (mt)
July 28th, 2008 at 4:35 pm
barbara - i think what they mean that if you have your website stored on your home computer you can run the domain as a different user since the problem is local to a select number of users on these servers. If you don’t have your files stored locally then you are out of luck until friday? who knows, right.
July 28th, 2008 at 4:45 pm
Hi John, I do have it stored locally. I created a new user in the panel yesterday with ftp access (the user is shown created without the clock : )), then I went to domains > manage, set the domain to use the new user…i have the files to upload. but, when I attempt to login via ftp (macintosh, cyberduck) I get a connection refused error.
July 28th, 2008 at 5:39 pm
@ Voo : the Lord is clearly verily pissed at DH and has cast down the servers of the pretenders as punishment for their impunity. This soil is sour and forever cursed. You are however welcome at the promised land - Media Temple (heck, even the name has religious connotations). Come join the just and leave the morally depraved to wallow in their inequity.
July 28th, 2008 at 5:50 pm
Maybe in a homoish, Cali graphic arts / surfer wishtheywere kinda way it’s religious. Also, The Lord is not pissed at Dreamhost, not in the least… He is pissed at select users with shitty sites. I would say that there is not one among the site downers who can truly say that there sites praise Him either directly / indirectly, and are W3C compliant.
July 28th, 2008 at 8:27 pm
Media Temple is too wonky for my needs and interests. I need reliable, unlimited, functional IMAP e-mail, and after that, I use my Web account for experimentation and a personal portfolio. Bluehost is a better match for my interests. Here is a list of their pricing:
http://helpdesk.bluehost.com/kb/index.php?x=&mod_id=2&id=162
And here is a list of Bluehost’s features, which I believe are comparable with DreamHost’s:
http://www.bluehost.com/tell_me_more.html
Does anyone have any experience with Bluehost, good or bad? Can you suggest any independent reviews that compare DreamHost and Bluehost?
July 28th, 2008 at 10:08 pm
I want my files back please.
July 29th, 2008 at 4:02 am
cant they even post progress updates?
July 29th, 2008 at 6:09 pm
have anybody’s files been restored yet?!?! mine still aren’t, although it changed from the sites being hosted but blank (no files), to the domains not responding or refusing connections to a bad_http_conf error today (but still no homedir or files, so hupping the daemon doesn’t fix the problem…). anyone? anyone?
of course anyone whose problems are fixed by now is probably not reading the comments…
john
July 29th, 2008 at 6:56 pm
After sending in an email, Ralph (of DH) contacted me and let me know what’s going on. They had a massive hardware failure, as well as a massive backup failure on the system itself. It’s still been unacceptably long, but at least they are being responsive to emails. He was very apologetic, and mentioned that all who were affected were going to be getting a major credit on their account. It looks like the data is being recovered from an offsite backup (over a slow link, it would seem). At least the data was not lost … but I guess we’ll find out for sure by Friday.
(For the record, Ralph said: “I will be pushing for a full year of service credit for users affected by this issue and I do truly apologize for the length of time your service will be down.”)
July 29th, 2008 at 7:19 pm
I just with they’d tell us how to get a domain established on a new server. I tried to set up a new one, but it put it back on lovejoy, and it’s down again!
July 30th, 2008 at 12:49 am
All my domains are hosted on Lovejoy. Having the same problems as the rest of you -started Saturday night, the 26th, when I got a 404 on one of my front pages, that was scary - I ssh’ed in, and was absolutely terrified to discover empty directory trees for all of my domains! As of right now, I can ssh in, and as best I can tell, everything that’s supposed to be there is there. Can’t ftp, though, and via http, all domains are now giving the bad_httpd_conf error. That’s nine domains offline for over three days now.
Severity medium my ass.
July 30th, 2008 at 8:46 am
my sites are accessible now.
I hope it stays that way.
July 30th, 2008 at 8:56 am
Don’t count on it jbloggs.
July 30th, 2008 at 6:26 pm
Wonderful… now our issue is on page 2. I would say “out of sight, out of mind”, but I could we really be on someone’s mind after this amount of time?
July 30th, 2008 at 8:55 pm
it’s on mine : )
so, if my sites are now getting the http error…should I try the fix…or still wait?
July 31st, 2008 at 6:57 am
For fucks sake. The last update was 6 days ago. GIVE US AN UPDATE YOU.
July 31st, 2008 at 8:56 am
This is truly ridiculars.
July 31st, 2008 at 10:25 am
My websites have been giving the bad http conf for days as well. This really pisses me off. Severity WTF level. I’m thinking about dropping DH now. Support is unresponsive.
July 31st, 2008 at 8:12 pm
This shit’s gettin serious now… Bobby Bare’s site is affected.
http://bobbybare.net/
Thank The Lord this is still up though!
http://www.bertc.com/dropkick.htm
August 1st, 2008 at 1:17 am
jonathan, what the hell is wrong with you? why are you still thinking about dropping dreamhost? i mean, honestly, is there any reason to stay? unless you have this completely for fucktarded experimentation and they give it to you for free for the rest of your life, then there is no reason to stay…
August 1st, 2008 at 7:45 am
I think my move compelted on Wednesday; I only had bad_http_conf on one site, and after a support ticket and direct e-mail to Ralph it was solved within a few hours (the manage domains fix didn’t do it). I like the text in the latest update that “Each user on this file server have tremendous disk usage” as I know I’m one of the culprits! haha.
I’m not leaving. I like my Dreamhost too much. Oh, and those bruises? I fell down the stairs..
August 1st, 2008 at 7:52 am
Josh Kirby:
re: bluehost, why should I need to fax them a copy of a picture ID for ssh access? That makes no sense.
http://helpdesk.bluehost.com/kb/index.php?x=&mod_id=2&root=15&id=203
I think I use dreamhost more for the ssh end of things than the web stuff…
August 1st, 2008 at 10:04 am
Word of advice - do NOT use BlueHost (also known as HostMonster) - been there, done that, and they aren’t any better than DreamHost. And yeah, the ID thing is ridiculous.
Site5.com were ok if you want really cheap prices, but for high quality value for money and reliability, go for Media Temple.
@ Ray “I’m not leaving. I like my Dreamhost too much.” May I ask WHY? You like paying for month after month after month of service issues? Man, some people are just too stupid.
August 1st, 2008 at 12:20 pm
@angrydog: Maybe I’m the abused spouse (see: fell down the stairs) that keeps coming back because they really do love me?
Why do I stay? Maybe it’s because being in the computer industry myself, I know that shit breaks all the time and no matter how much redundancy you have, there is always a weak point in the system. Completion times for data transfer always are limited to bus speed and data quantity.
There is a correlation between how much stuff costs and how long it’s gonna be down when (not if) it breaks. If I needed 100% uptime for my shit, I would either be paying top dollar for dedicated hosting or I would build my own machine to drop in a co-lo.
I’m comfortable with the level of downtime I’ve had for what I pay (almost nothing with referrals, since July 2004), and if it’s a real emergency and my sites go down, it’s five minutes for me to change the nameservers to somewhere else to get my sites back up right away, even if it’s at my house.
In my four years here, this is really the only major downtime that I experienced. That’s okay with me. Little things with mail or routing happen every once in a while, but I usually don’t find out until after they’ve been resolved. Maybe I’m just too lazy to go look for someone else, but at the same time, with any problems, I still feel well taken care of here.
August 2nd, 2008 at 5:01 pm
Good non-rabid-DH-supporter answer, thanks Ray - it is appreciated.
While I agree that no system is infallible, I don’t believe this is should serve as an open-ended excuse either. DH have had an issue with this server for many many months (since early 2007), and continue to do so. The need for comprehensive redundancy and it’s cost-effectiveness is one thing at this hosting price level, however when you have paying clients who are being adversely affected for months on end, then one should seriously consider that there are other issues going on here. Whether it be incompetency, lack of funds to replace equipment, or just sheer arrogance, it is hard to say for certain, however one thing is certain - it has gone on for far too long. I don’t know how long you have been with DH, but if you have been here for a significant time with little to no issues, then count yourself extremely fortunate, because that would make you the exception rather than the rule.
August 4th, 2008 at 12:40 am
I just wanted to let you know that my files have finally been restored, yay!
August 4th, 2008 at 8:49 am
I second Ray on his level headed response.
1st, of course My Site is not up yet. And 2nd, I will probably be the 44th of the 45 users remaining to be restored! (as of last update)
I have been with DH for many years, and this is by far the longest and largest issue I have seen. Considering the service provided for such a low cost, I will stay on.
Shit happens, get over it.
August 4th, 2008 at 12:01 pm
my files have been restored, but still throwing bad_httpd_conf errors, even after hupping the http daemon. anyone else having this problem?
August 4th, 2008 at 5:23 pm
All of my sites were down for about six days, then came back up two days ago. Now they’re all down once again.
I am a freelance photographer, plus I run a record label. Two seperate businesses of mine have been greatly fux0red due to this mess. It seems that this has been happening more and more, with an average of my websites being down every other month for the last two years.
This is garbage
August 5th, 2008 at 4:16 am
I’ve been patient, Dreamhost. Very patient. My site is still not up. It hasn’t been up since the initial incident.
What I want to know is; are you guys going to compensate for this? This is utterly ridiculous and a lot of us aren’t paying for almost two weeks of downtime (though some, particularly not your customers, would argue otherwise).
This downtime, if not compensated properly, is a huge huge dent in Dreamhost’s already well-dented record.
August 5th, 2008 at 5:50 am
I have go agree with Kaehyu, i also have been very patient and this downtime has to be compensated very well not only thinking in us as your clients but think what this has cost to US with our clients i have been telling them all kind of lies for them not to leave me so hopefully the compensation will be a good one…
My sites still down btw so im guessing that im still on those 45 users that haven’t been moved.
August 5th, 2008 at 6:00 am
what happened to august 4th’s update…
August 5th, 2008 at 6:06 am
“What we’ll do is post how many users are left to move at least twice a day.”
As of August 5, we don’t have any updates posted since the early afternoon of August 3. Good work Dreamhost. It only takes 1 day for you to forget your promise!!!
August 5th, 2008 at 6:45 am
It seems like apache isn’t even running on sanjay today. All my sites have been returning “Connection Refused” for at least 10h now and I can’t seem to get any attention from the support staff.
August 5th, 2008 at 11:06 am
Still nothing.
Not even a form letter from support. Just being completely ignored.
I photographed a band on Sunday night, to be shown to the band and the label and the managers on Monday. It’s now tuesday and I can’t even use my own websites, which I pay for, to conduct my business. I now have to use a yousendit/rapidshare website so I don’t look like an ignorant n00b to a paying client. Which, in turn, makes me look like an ignorant n00b.
This is a label who I wanted to establish a longterm business relationship with, but if I were them, I’d tell me “no thanks”.
August 5th, 2008 at 4:10 pm
POS webhost. You get what you pay for. In this case, you don’t even get that.
August 5th, 2008 at 6:46 pm
still waiting……
Dear nightterror, what if it was your MOM’s site that was down? (hotmilf… .com)
Would she be as patient as I??
Number 35? anyone have number 35?
August 5th, 2008 at 8:48 pm
Ohmmade:
IF you’re still with DREAMHOST you deserve to look like an ignorant n00b. Running your “important” business on a webhost for pennies is not a good investment. Invest in a decent webhost and you’ll notice the difference.
August 5th, 2008 at 11:29 pm
Are you guys using a USB 1.1 port for your data transfer here? USB 2.0 is much faster, I hear. haha
Part of me wants to buy Josh and co a drink, but then I fear they’ll spill it on lovejoy.
August 6th, 2008 at 7:45 am
(originally written by Jim somehow somwhere, but fully applies to me)
I loved and used DH for years. They have excellent webpanel and *had* excellent support. Fast download speeds, just awesome for $15 to $30 depends on year and plan I subscribed. For me unlimited and fast databases, transfer, SSH etc. !!! I recommended and bring over 10 customers (no provisions, just recommandations to friends) to DH.
What I have learned from 2007/11 is a very simply - DH is not a place to host serious business sites of any kind. Fun little sites that real businesses do not depend upon sure, anything that someone expects to be up most of the time no way.
I work with other hosting companies and yes downtime is a reality. But with the others (similar or lower pricing compared (for same transfer/capacity/SSH/DBs …) to DH so let’s take that silly argument of the equation) being down for an hour or two at a time is considered severe. They don’t apologize for it, they fix it and get us back up and running and then explain what happened after the fact. Or sometime they make a dead bug, no apologize, no explanation, nothing bad happened, no money return, but !! fixed service within few hours!!! (Not fair, but better than NO SERVICE FOR 7 MONTHS !!! and many extra large smile pics huge super fun blog mail apologisies from owner of the company)
(check links)
{5 months - problems arised end of Feb. Now is August}
http://www.dreamhoststatus.com/2008/03/27/filer-problems-with-blingy-cluster/
http://www.dreamhoststatus.com/2008/05/21/blingy-problems/
http://www.dreamhoststatus.com/2008/07/21/lovejoy-and-sanjay-file-server-issues/
I just don’t see any reasonable argument for 2 weeks of downtime. I’ve been in the IT game for over 12 years and I just really can’t come up with a technology scenario that would require this much downtime to fix a problem. Something other than technology (use your imagination I don’t want to be accused of starting rumors) becoming a very likely possibility in my opinion.
The the DH (if you read this) please just keep using your best efforts to get us back up and running….PLEASE we are begging you.
August 6th, 2008 at 10:57 am
My primary mailbox is held hostage by lovejoy, so I had all messages copy to a new mailbox. Now I’m receiving notes from friends saying that (after 5 days) they are receiving bounce notices about e-mail that they know I’ve already received and replied to. If I turn the hostage mailbox into a forwarding only address, (1) will the mailbox that was associated with that e-mail address be deleted, along with all of the hostage messages, and (2) will my friends stop receiving bounce notices?
This situation is so disgusting. We were promised recovery in a week. Now we’re promised nothing at all, and our issue is now on page 3 of their status blog. I don’t have a business, and nothing related to DreamHost earns me money directly, but I still feel like I’ve been shat upon. Someone could have FedEx’d a HDD containing all of the needed backup data by now.
August 6th, 2008 at 10:27 pm
i am still awaiting. i can receive and send email…so that’s not a problem. my account filer status says “sidestep lathe.” I am not sure if my account moved yet or not (i still cannot get through using http://ftp.) All, but one of my sites, still has the httpd error. My main account has two files in it, and I have no clue as to what they are, but certainly not my full site. I don’t know if I moved and none of my part of the server can be restored, or anything. Most of my clients can restore from their own backups (really none of what i have at this point can’t be restored from my own backups except my old emails.)
If there is a way I can just get back online, i will restore myself. I have already started re-building the client’s data i lost. So, if there is a way you can sidestep my sites and KEEP my old emails (YES, I NEED MY OLD EMAILS) then that would be great. but just knowing if i am one that has not moved yet would be great, cause i am somewhat scared now.) Some clue as to what is going on per account (at least for those of us who have the httpd errors) would be great. Tell me if my email is lost forever so i can cry now. the irs might be a pain if so. : (
please let us know, even via email, what status we are at the moment. if i am restored and i have nothing left, i would at least like to know.
August 6th, 2008 at 10:32 pm
At this rate, it seems like you guys are only able to move 1 user per day.
What’s your target date, Dreamhost?
You update this post everyday, but it’s immensely dissatisfying to know that your move rate is 1 user per day with 33 left to move.
I don’t think the users that you do have left to move want to have over a month of downtime.
August 7th, 2008 at 5:11 am
In May 2008, DreamHost’s official web log alluded to their preference for everyone to move their mail to Gmail so that they can focus more on Web hosting. This Chris Brogan blog post, and the succeeding links in the story, demonstrate why I pay (paid?) DreamHost for private, self-controlled, mailboxes and mail management: http://www.chrisbrogan.com/when-google-owns-you/.
I will not have Google claiming control of my digital life. If DreamHost is getting themselves out of the mail hosting/management business (and this ridiculous downtime of lovejoy and sanjay is their de facto exit from that scene), then I will need to find a commercial host that focuses on e-mail with some degree of dedication and reliability. Google, however, will not receive my business. I want to pay for my privacy. If DreamHost won’t take my money to support my needs, who will? Suggestions welcome.
August 7th, 2008 at 12:40 pm
ok if you move user by user , it means all the sites from that user will be copied and restored at once.. or is the copy been made by site.. cause one of my sites doesnt work.. i see all the data when i log to my ftp client, but when i put the html i want to see in browser it says that the file doesnt exist… like the files are there but they arent :-o…
August 7th, 2008 at 5:59 pm
@Josh Kirby:
If you think you have security with shared hosting, think again. Also, if you think e-mail is secure, think again. I don’t think I need to go into details within the context of this status blog, but your transmissions are really only fully safe with something like GPG encryption on both ends, and even then there are issues of private key storage and passphrases.
August 8th, 2008 at 9:09 am
Have no fear, Lovejoy / Sanjay people, I have not forgotten you, and as a show of solidarity I have been leaving my own website down as well. It is a terrible plague that you have faced… to those of you who have made it through and never wavered in your faith, I am sure that He (Josh Jones, maybe, if this little incident makes it as high as His desk, in this case) will reward you well. If you are willing to take the humbling that you have been given by Him (talkin about The Lord now), and rebuild your sites to glorify Him (either directly or indirectly is fine… it need not be obvious and embarrass you… The Lord does not want you to feel ashamed of your love!) and make them W3C compliant, then I am sure that He (again, The Lord now) will also reward you well… He MIGHT even let you into Heaven if you impress Him with your humbleness, but I must be honest, I don’t thank some of you’s gonna git picked fur the teem.
And remember, Media Temple is a temple all right… a temple of The Devil. You KNOW you deelin with money changers when you get charged TWICE as much.
August 10th, 2008 at 11:30 am
The August 9 status update contains the quote, “It is possible that the problems that caused the stability issues are somehow causing the performance issues with the moves. We’ll keep going though.”
Does this mean that the fiasco that we’ve experienced may actually be repeated because the systemic failures may, in fact, not have been resolved?
August 10th, 2008 at 4:24 pm
@Josh Kirby: I would venture to say it probably means the problem will be solved by this move completing, since the hardware everything is moving off of (and thus being vacated) is the one with the problems. That’s what it reads like anyway.
August 10th, 2008 at 9:34 pm
Update?
I seriously feel like I’m going to be the last user to be moved. Ugh.
August 11th, 2008 at 9:48 am
lol @ Kaehyu
I, too, feel i may be the last person. maybe the last person gets free hosting for life : )
August 12th, 2008 at 6:44 am
Anyone else still having continued mail access troubles? I’ve submitted a trouble ticket to Ralph, but I want to know if it is just me given that “it should all be working”. I can not access the particular mail accounts by IMAP or webmail.
I’ve also experienced a number of troubles with the execution of scripts within Web apps hosted on lovejoy–Gallery scripts are encountering permissions trouble, a recent manual install of ZenPhoto is also broken.
It is probably just me. I’ll stab myself with a RAM board now.
August 12th, 2008 at 7:03 am
hoootey hoooo. I am back up : )
@Josh, I never had any mail trouble during all of this. Not really sure why, but didn’t question it. Good luck neighbor.
August 12th, 2008 at 1:58 pm
How do we check to see if a credit was applied to our account?
August 12th, 2008 at 3:39 pm
My mail is now accessible after a few “DH Test” e-mail popped into the inbox over the course of a few hours–Ralph and company seemed to have worked on the residual issues for some time. All things dreamhost seem to be working well for the time being.
August 20th, 2008 at 12:52 pm
5 months - problems arised end of Feb. Now is end of July
Former Very Happy DH customer
http://www.dreamhoststatus.com/2008/03/27/filer-problems-with-blingy-cluster/
http://www.dreamhoststatus.com/2008/05/21/blingy-problems/