Central Services and Spunky Cluster Move
Due to continued space and power constraints in our primary data center, we will be moving all central DreamHost functions as well as the “spunky” cluster to one of our newer data centers. This move will begin Friday, March 21st, at 9PM PDT, and is expected to last up to 12 hours, until Saturday, March 22nd, 9AM PDT.
Central services include panel.dreamhost.com, dreamhost.com, dreambook.com, files.dreamhost.com, media.dreamhost.com, wiki.dreamhost.com, discussion.dreamhost.com, basically everything but secondary DNS and this status blog. Primary and tertiary DNS will be going down as well, but will have a more limited downtime window of only 2-3 hours. For spunky, all web servers, mail servers, file servers, and MySQL servers in the spunky cluster will be unreachable. The list of affected spunky web servers is here:
abbazaba absinthe achilles adams admiralty aero alameda almalibre alondra altoid alvarado archibald arrow atlantic avalon aviation babyruth badillo balu bandini barranca barry baseline bazooka beaudry bellflower beverly bixel blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bubbleyum bundy burton butler butterfinger cahuenga caramello casamento catalina centinela century cerritos cheeto cheezit chex chiclet cienega cognac colbert compton cookie cordial crackerjack crenshaw crunch cupcake dingdong doheny dorito drumstick dumdum edendale euclid fairfax fc23 federal fig fireballs firestone fletcher flower foothill formosa frito frontseat fruitland fudge fundip garland garth garvey gobstopper goober good gower grand granola grappa gumdrop gummi haven heath heaven highland hillhurst hillmont hoho holt hoover hope hops hyperion jalapa jawbreaker jefferson jerky juniormint jurupa kidsgames kiki kitkat krackel lankershim larchmont laurel lemonhead licorice lifesaver limbo-spunky3 limbo-spunky4 locust look lucas malt manchester mariposa mayo melrose mento milliken mills monroe moorpark mounds munchies nash necco nerds nilla nougat nutrageous oakdale ocean olive olympic ord oreo ouzo overland oxnard paramount payday pernod pez pico planters plenty pocky popcorn poprock powerbar pretzel price pringle rampart razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rossmore rum runts sake saltine sanborn sawtelle schnapps scotch selma sepulveda shocktart skittle skor skybar slauson smores snicker snocap soto sourpatch spree starburst sugardaddy sunset taffy tampa temple tequila toblerone toffee tootsie topanga torrance truffle twinkie twix twizzler union vanowen vermouth vine virgil warhead washington western whittier whopper wilmington wilshire york zagnut zuma
We realize this is a very large move and we will have a dozen techs onhand to supervise. My apologies for not posting this earlier.
Happy DreamHost Mover Team!
.
March 21st, 2008 at 8:18 pm
with your lack of planning, you just costed me the release of a quite expensive site.
that’s not HAPPY, and surely not TEAMY, that’s – to say the least – is CRAPPY.
March 21st, 2008 at 8:27 pm
It’s too bad that nobody has invented some sort of electronic computing device that could be used to automatically send planned-downtime notifications to customers on affected servers. 1970 is going to be so awesome!
I shouldn’t find out about moves like this by going “wtf, time to check DreamhostStatus again, my site is unavailable for the eighth time this month.” And no, telling me that I should routinely check DreamhostStatus is not an acceptable answer.
March 21st, 2008 at 8:33 pm
Is the following available in the dh control panel?
a. A list of where all my services are hosted.
b. A place to sign up for emails or rss for planned service outages that will email me prior to the outage?
rss here doesn’t count - I just want outages that might affect me.
Just a thought.
C
March 21st, 2008 at 8:36 pm
So you take down my entire account, including ALL EMAIL FUNCTIONS, for TWELVE HOURS, with NOT EVEN 48 hours notice?
A move like this should have been announced WEEKS ahead of time.
And better planning using processes like mirroring servers across the network first could have greatly reduced downtime.
I am utterly shocked that you would take out people’s email for TWELVE HOURS with less than two days notice. There is no excuse for this, because you chose to do this; it’s not like your building collapsed.
Why do I even bother with DreamHost for email? I could just use Google Apps for Domains for free, and then I’d never have to worry about an outage.
Pathetic. Truly pathetic! And if I am wrong about the patheticness of this decision and the way it was carried out, then by all means, DreamHost, explain to me why you didn’t do a more professional job here!
March 21st, 2008 at 8:39 pm
i remember getting an email about this last week
March 21st, 2008 at 8:40 pm
So spacey and sunnyd are not in this list yet I can’t seem to reach my mail server? Is that because the DNSes are down? … if so, there was a breakdown in communication: until I started having this problem and checking this blog, I had not idea you were doing another cluster move.
March 21st, 2008 at 8:42 pm
No wonder my site is down. blah! Oh well off to watch a movie i guess.
March 21st, 2008 at 8:49 pm
hey shawn..
Who made you the new jello sheriff?
AFAIC, a heads-up email in addition to the rss feed (need not be an either/or approach) would have been suitable in this case.. or even just a quick fyi in the ‘news’lettery.
Meanwhile, since it was planned for over a week in advance, a mirror ‘plumber page’ for the affected domains (now returning 404 not found for the next 12-hours, at least) would have been thoughtful too.
Have fun roasting the easter bunny.. Llama.
March 21st, 2008 at 8:51 pm
First of all I’ve been hosting multiple websites with you guys for six years but recently I’ve been growing more and more dissatisfied with your reliability. I took off my Saturday to work on my business in privacy of empty office and guess what? You guys are having 12 HOURS downtime! All my websites hosted by you are down for 12 HOURS which means I can’t work AND my customers cannot find my websites for 12 HOURS.
This is on top of the billing mistake that, despite the refunds, I lost money because of the currency differences.
I’ve retained one website at another much more pricier host but they never have these issues you are having. I guess you get what you pay for.
March 21st, 2008 at 8:56 pm
Kaboom!
March 21st, 2008 at 8:58 pm
Do you offer refunds on CPC and other advertisings that are running while we are experiencing your Happy downtime.
March 21st, 2008 at 9:00 pm
12 hours on a busy weekend
I know dreamhost is cheap but giving no notice and on a holiday weekend
man that’s bogus
March 21st, 2008 at 9:03 pm
I’m a newbie to this whole website thing and have a question. Hopefully someone will be kind enough to answer. Since my email is down for this move, what happens to the email that people are sending me? Will it bounce back to the sender or somehow be kept floating in cyberspace until our servers are back online?
I am expecting a couple of important things tonight (yes, even this late!) so I would really like an explanation.
Thanks to every,
Sarah
March 21st, 2008 at 9:05 pm
Any recommendations for reliable hosting service? I don’t care about the price anymore if the reliability is on level that satisfies business needs. Can’t host business websites on unreliable host like this.
I’m serious not just venting my anger.
March 21st, 2008 at 9:06 pm
We’re moving a site over to a sweedish server. It’s not because of this down-time. It makes me excited. We’re going to keep our dreamhost account cause that’s what we use to manage subversion and what not, and I have to admit, the panel was made pretty well. I use linux a lot, so I get a little pissed off when I do something via the shell and it doesn’t work for whatever reason and I’m forced to use the panel, but whatever. Anyways, I’m happy with the move. We had a really popular video a while back and got 200k hits and dreamhost went down like a drunk girl at the after-prom, but we stuck with ‘em cause it’s cheap.
P.S. To anybody who read my entire post, I just wasted a good 30 seconds of your life.
March 21st, 2008 at 9:08 pm
Wow, I just came over to say that it’s a good thing I have a Yahoo and Gmail email account, so my obsessive/compulsive email checking will survive the 12 hour downtime. I didn’t know everyone was going to be on here complaining about excessive down time and yelling at each other.
I’ve been a happy customer for 2 years, and haven’t really noticed too much downtime. The biggest problem I have had was the recent double billing which got fixed sufficiently fast for my needs.
I just miss my email right now.
March 21st, 2008 at 9:10 pm
I am pretty annoyed. Middle of the night for PDT is great for those of you who live in the US. For those of us in Australia, this happens to be the afternoon… I was not sent an email, and I honestly do not check the panel on a regular basis to check out upcoming outages. I tried to go to dreamhost.com to see what is going on, and that site is unreachable too… God bless Google.
March 21st, 2008 at 9:12 pm
@feeling lucky - I love you. I’d been wanting to migrate to Gmail forever, but really wanted my custom domains for my e-mail address. Great to find out they’ve now got it set up so you can have the best of both worlds. They even have it set up with IMAP. Thanks so much for the tip!
March 21st, 2008 at 9:13 pm
4 out of 5 websites down. All emails down. 12 h downtime on a weekend. This is f****** unbelievable. No, better yet this is f******** priceless. As soon as I find another hosting service I’m off.
You just lost customer of six years.
March 21st, 2008 at 9:18 pm
Wouldn’t directly emailing users affected be better? I understand there was a bit of advanced notice, but I don’t check this status page until somethings actually down.
March 21st, 2008 at 9:22 pm
To all of you complaining:
I am not a customer, nor would ever be; I need reliability, and gladly pay the extra $20/mo. extra for private colocation right down the road from my office (in Austin).
This is one of the cheapest hosting companies around; what do you expect? If you have your vision exam done at Wal-Mart, don’t expect your glasses to not be a little blurry.
If you are making money with your website, why on earth would you look for the the host as no-i-dont-work-here-and-use-this-alias-to-vent Shawn points out contractually obligates you to agree that they “make no guarantee that any given reader shall be able to access DreamHost Web Hosting’s server at any given time”?
For those of you complaining about the $$$ you are losing, you are doing it to yourself.
For those of you that need reliability, you cannot go with the lowest bidder. In terms of Total Cost of Ownership, the lowest bidder is always the worst choice…
March 21st, 2008 at 9:25 pm
Would be nice if when they did stuff like this all websites would redirect to a Dreamhost is currently moving stuff. This website will be back up shortly.
March 21st, 2008 at 9:27 pm
um, yea. um. this site is always down. i just finished something and shouldnt have to deal with this right? especially when i just sent out emails to a bunch of people to go to the site. thank you. really. now please put up a nice message for EVERYONE affected that says, “The server is being upgraded and will be back shortly.” Because right now my business is getting affected by your business and I think this isn’t a bit wadoo to do.
March 21st, 2008 at 9:29 pm
and another thing. If you “HAVE” to have your webpage up 24/7 then get your own private server in a colo. I can live with a little down time for the price I pay for the service.
March 21st, 2008 at 9:31 pm
Sorry. Let me rephrase, Your business is impacting my business in a negative manner. Especially now. This is costly. Please put up a nice message for EVERYONE affected that says, “The server is being upgraded and will be back shortly.” This is cost-free and helps us know that something is up.
March 21st, 2008 at 9:32 pm
and even another thing. I have about 10 or so sites on Dreamhost and have only had about 24 hours down time on them in the past few years. Those aren’t bad stats for such cheap hosting.
March 21st, 2008 at 9:32 pm
Sucks to have my sites down. My Mom and spam robots might not be able to read my site for a while. I guess I’ll have to call them to tell them to check back later.
Maybe more people will check out my stuff on flickr.
http://www.flickr.com/photos/mohdi
March 21st, 2008 at 9:33 pm
The message would be a solid though for sure. Would probably help out with a lot of angry people having to answer why the webpage is down.
March 21st, 2008 at 9:40 pm
Well, I don’t see the server “thesis” here. But it is also down
March 21st, 2008 at 9:44 pm
um. right. that’s why i said it. see, i’m an average joe user. i have 7 sites. most are ’safe’ if down. but i dont sit there frequenting ‘hey, is this site going down’ this is why load balancing servers are required. if one site is down, push em to another one for the meantime until ‘admin’ fixes the issue. now in my case, i did work up until 5am this morning, then at 8am i went to work, now i sent the owner an email stating, ‘hey ur site is ready send out the notices and lets get traffic.’ see, we were just starting up, but nonetheless when every hit counts it kinda hurts in this scenario. yea you can say whatever you want, but at the end a message and proper load balancing is what we as consumers pay dreamhost for. so before you think of fucking with me, send me your address so i can come over and kick y your sorry ass because i din’t post for shit and dont fucking suck on my balls after you try and talk shit to me. that pisses me off fucker expecially if ur not offering solutions and i am. so go flikr fuckers. all i care right now is that the admins do their jobs for doing what they say their site does - be a webhost. regular users only expect that, hosting. so when u talk about private servers and crap, just stop suckin on your own dick and do me the favor of just bending over cuz u love to pay to get fucked.
March 21st, 2008 at 9:47 pm
I site is down
March 21st, 2008 at 9:48 pm
Damn, that blows. I just realized the Adwords campaign I started has been giving customers a page not available. Wasted cash. Yay! I still love Dreamhost. Screw it.
March 21st, 2008 at 9:48 pm
well great gobs of crap
March 21st, 2008 at 9:52 pm
what man ! It is huge downtime ….
March 21st, 2008 at 10:08 pm
Well, I didn’t find out about the move until I tried to get to work. But I only use my site for sandbox-type messing around. I consider maintaining even my own dev server sufficient pain in the butt (I mess with lots of different stuff) that I’m happy to pay somebody else to do it. In my +2 years with DreamHost, my site’s always been there when I wanted it. It’s also handy, convenient place to stash stuff when I want to rebuild one of my computers. Yeah, I got the “billing error” too. I didn’t care, I was planning on buying another year anyway and I said so. ^_^
The DreamHost guys are always quick and responsive. I can ask them “Hey, can I get blah” and they give me an answer that is either yes or a totally reasonable “nope.” I say “Hey, I did some bonehead thing trying to install blah, can you clear it” and they say “Sure.”
So they’re moving to bigger, better place. Superduper. So it means I don’t get to spend tonight hunched over my computer goofing off building castles in my sandbox, whoop de doo. Oh noes, DreamHost, you’re forcing me to spend the holiday hanging out with my family and talking to people’s faces! Mean, mean DreamHost!
If I had a -business need- to rely on my site? I sure as hell wouldn’t use -shared- hosting, I definately wouldn’t rely on a single shared site, and I most assuredly would not be doing dev on production no matter what.
But if I used remote hosting at all- I’d probably still use DreamHost.
March 21st, 2008 at 10:15 pm
Hope the techs have an easy night, I’m up all night as well with hardy beta and coffee. I’ve been a happy customer for about 2 years now, and this is really the first time I can recall when I discovered it being down. I’m no sycophant, but they really stay on top of features on these machines and I believe they have my long term best interests in mind with this move.
March 21st, 2008 at 10:30 pm
Please back up soooooooooooon ….
my ass is on the line with my clients ….
are we back up now?
March 21st, 2008 at 10:38 pm
Yea:
I didn’t know about the outage until tonight; namely, after it started. I tried to remote into my site and couldn’t- network connection error. I pinged my site, nothing. Pinged IP, nothing. Pinged DreamHost, voila. My site is down because DreamHost is down. Oh noes. If my site were important to me in any way financially, I’d do my own legwork and double hosting elsewhere and maintain dev serving locally. I would use DreamHosts -business plan- for my business needs. Let’s face it, most of us came here because DreamHost is cost-effective and we bought in for >10 bucks a month.
Would you also use your ISP’s residential-grade plan for your business connection, then call and yell when you exceed bandwidth limitations because you thought you could be clever and save a buck? DreamHost’s business-priced plans are business-grade hosting and every remote hosting company on the planet is going to have “scheduled outages” just like this one. Even with a business hosting plan promising business-grade reliability, how do you know somebody won’t fly a plane into the server bays or a hurricane won’t wash your site into the Pacific? If your site/s are that important to you, take responsibility for your own “due diligence” and try some standard mitigation practices.
Somebody call a wahmbulance, we’ve got an emo. ^_^
March 21st, 2008 at 10:41 pm
Been a DH customer for 2 years or so, expect some kind of notice for major events, yes. Didn’t get it, also didn’t get a suggestion that I should subscribe to an RSS feed for such notifications. Sorry, but expecting me to figure out where you’re sticking major server down notifications isn’t exactly professional, and it’s definitely going to cause problems for me this weekend.
Don’t think I’m being a baby or a moron or a whiner or whatever to expect some kind of “HEY WE’RE EVEN TAKING DOWN YOUR CONTROL PANEL” notification, instead of a late “apologies for not telling you sooner” note on a blog/RSS feed that is so badly promoted by DH that I had to find it on a Google search. For the most part your service has been great, but this is definitely a black mark. Please rethink your customer service policies–stuff like this is why I moved to DH in the first place. It can definitely be why I leave, too.
March 21st, 2008 at 10:42 pm
I’m not a Spunkian and my sites are also down. My server is Electra which isn’t on the list. What the heck!?! I’m lost.
March 21st, 2008 at 10:43 pm
I don’t understand why Dreamhost doesn’t use replication, so that this sort of thing doesn’t happen. Can you say VMWare?
March 21st, 2008 at 10:45 pm
Server named:”srsorgyu” is not on “blacklist” but it’s also down… May I ask WHY?
March 21st, 2008 at 10:49 pm
ftp OMG
March 21st, 2008 at 11:10 pm
I HAVE TEN LEGS!!!!!!!!
March 21st, 2008 at 11:12 pm
tyche is also down (or nearly down). Not a spunkian either.
Is it a normal practice for people to monitor their ISP’s unofficial RSS feeds for service outage notifications?
I’m not complaining, noone will notice outage of my $9/mo plan.
I’m curious - people above suggest that they are comfortable with this arrangement - is it really the norm?
March 21st, 2008 at 11:19 pm
Whenever things are done “moving” am I going to have to change any zone records, since my DNS stuff is all at DNSME instead of DreamHost nameservers?
John a/k/a HarpShot
March 21st, 2008 at 11:21 pm
OK - with ALL my sites and email down and a TON of work that I now can’t finish on time, I had nothing better to do than read the posts here, but Shawn made that just TOO boring so I stopped!
Everyone talks about better hosts, but can anyone out there point me in a good direction?
Who do you recommend for afordability and reliability?
March 21st, 2008 at 11:54 pm
Does anyone know about the service at Hostgator?
March 22nd, 2008 at 12:19 am
Well, we might as well put put our enforced leisure time to good use…
HostGator looks quite good. Guaranteed 99.9% uptime and in their larger (but still affordable) packages umlimited domains and subdomains.
March 22nd, 2008 at 12:22 am
http://nouptime.com probably has a lot more uptime than dreamhost does…