Spacey Cluster Move!
Due to space and power constraints in our primary datacenter we are moving the entire shared cluster “spacey” to our LAX facility at 10:00PM PST (GMT - 8), two weeks from today, Friday, November 30th 2007. All webservers, mail servers, file servers, and MySQL servers in the spacey cluster will be unreachable during the move, which we expect to take approximately 8 hours.
The spacey cluster contains the following web servers:
7up, amp, aquafina, arizona, barqs, bawls, boba, brisk, cactuscooler, calpico, cappuccino, caprisun, chai, clamato, codered, coffee, coke, crush, crystallight, culligan, dads, dasani, dietrite, drpepper, eggnog, endless, espresso, evian, fanta, fetish1, fiji, frappe, fresca, fuze, gatorade, geyser, gingerale, hansens, hi-c, hiball, horchata, ibc, inko, izze, jolt, koolaid, lipton, martinellis, milk, minutemaid, monster, mountaindew, mrpibb, mug, musclemilk, nehi, nesbitt, nestea, niagara, odwalla, oj, optimizer, orangebang, ovaltine, pellegrino, pepsi, perrier, plentymag, pom, powerade, propel, punch, redbull, refresco, riptide, rockstar, rootbeer, schweppes, seagull, seltzer, shasta, silk, simpledollar, slice, slim, sly, smoothie, snapple, sobe, sparkletts, sprite, squirt, stewarts, sunkist, sunnyd, tab, tampico, tang, tea, tejava, tizer, tonic, tropicana, v8, voss, welchs, wishserver, worldtown, yoohoo
If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”. We are also sending out an email announcement about this downtime window to affected customers.
We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.
If you have any questions regarding the server move, please contact our support team.
.
November 16th, 2007 at 11:28 am
tat’s a lot of downtime! will we be compensated?
November 16th, 2007 at 11:34 am
Dude, you pay a low price you have to expect downtime, and they are warning you ahead of time.
November 16th, 2007 at 11:49 am
I will never understand how people can complain with scheduled downtimes … Never.
November 16th, 2007 at 11:52 am
Just forgot this: How much of the few $ he paid for it per month, he would like back?
November 16th, 2007 at 11:58 am
Not again, no!!!!!!!
November 16th, 2007 at 12:10 pm
Can’t imagine all the work that is gonna take. I hope everything works out ok.
November 16th, 2007 at 12:23 pm
Thanks for the notice. Shouldn’t this be set as a “high” priority rather than medium?
And I do understand why some people are complaining even though it’s scheduled. DreamHost seems to have been down a lot lately, so scheduled or not it affects some.
November 16th, 2007 at 12:39 pm
Just got the e-mail. Funny thing, when the pop for the e-mail came, another window was blocking it and all I saw was spacey, so I was actually a bit confused as to what e-mail I had just gotten.
November 16th, 2007 at 12:41 pm
it’s a fucked up thing, but it has to be done.
good luck
November 16th, 2007 at 12:47 pm
Will we get any better performance in the other data center? Or is this so the cluster can be expanded?
November 16th, 2007 at 12:51 pm
“you pay a low price you have to expect downtime”
I have had more downtime (not warned “ahead of time”) with Dreamhost than with any other shared hosting provider. Average downtime: 2 hours. Monitorized with Montastic. You guys really suck. 8 hours downtime, that’s service! I’m transferring my account to Bluehost ASAP.
November 16th, 2007 at 1:32 pm
good luck
November 16th, 2007 at 1:36 pm
so 10.95 a month for shared hosting is considered a low price? What do you guys think this is? That shouldnt be considered a low price, but isnt a high price either. Its fair. If dreamhost said 99.9% uptime then i would understand people complaining but they didn’t say that
November 16th, 2007 at 1:44 pm
“I’m transferring my account to Bluehost ASAP.”
Do it already for G**’s sake. Quit talking about it and just do it. Take your negative mojo elsewhere and pollute another host. Thanks.
Appreciate the head’s up DH. 2 Weeks is enough fair warning for even those who have businesses on a shared hosting account to plan for the downtime , notify their customers of maintenance, or make alternate plans.
I use DH as my primary host, but I have failover to a several other hosts that I will not mention here…I don’t get payed anything for advertising their names…plus it has no relevance to DH. I am not a host fanboy, there’s just too many other things to worry about besides promoting someone else’s business for free. They only pick up slack when my DH server is down…which isn’t often for me. Ever since moving to failover to run my business, I have had the luxury of no downtime. I still love DH the most…I have been with them from the beginning and I greatly appreciate their transparent status policy. I wouldn’t trust my business to run on only 1 system, even my own closet-mounted server that people like to boast about being more stable :)…using 1 host for business is inviting downtime.
Remember people, downtime is GUARANTEED no matter who your host is!
November 16th, 2007 at 1:52 pm
8 HOURS??????? I CAN’T BELIEVE IT! My website will be down all day (I’m on GMT+2).
How say low price? If you pay for 2 years you can find webhosting at 5.95$/month for 300GB storage and 3TB traffic. So, this isn’t low price.
November 16th, 2007 at 2:02 pm
“so 10.95 a month for shared hosting is considered a low price?”
Well, 500GB of storage and 5TB of bandwidth to start that grows weekly for a monthly rate of $10.95 month-to-month with no annual pre-pay is pretty darn good in my opinion.
Are you ever gonna use that space? Who knows. Can you get cheaper elsewhere? Sure…and get less space and bandwidth. You can go for 350GB and 3TB bandwidth for $4.95 elsewhere…but do you get all of the options DH has to offer? Probably not. Are they transparent? Nope. Month to month? Nope.
November 16th, 2007 at 2:52 pm
Oh, btw: Take some pictures for the BLOG! I want to see tea being moved!
November 16th, 2007 at 3:14 pm
“Take your negative mojo elsewhere and pollute another host. Thanks.”
Why is everyone so punchy here? People are complaining because their experience has not been good so far. I wouldn’t complain about the scheduled downtime, but it seems my site has been down QUITE often lately. So this is just one more thing.
DreamHost is a cool company, they just might not be the best host. I pay the same price at another host for almost (not exactly) the same space and bandwidth. My site has gone down once and it was at 4AM PST. I wouldn’t have even have known it except that they wrote me to let me know and apologize.
DreamHost isn’t the only host that lets you know what is happening. Every host I’ve ever had does that. But they go down the most by far.
I hope the move goes well, I know things like this are a hassle!
November 16th, 2007 at 3:18 pm
Can I get that with a promise that you’ll never come back? Please?
November 16th, 2007 at 3:59 pm
@Red - Any pointers on how you hooked up for failover?
November 16th, 2007 at 5:17 pm
It’s funny how the people that are stupid enough to whine here just happen to be the ones that think moving to Bluehost is a smart move.
If they want to sit around crying with being called whining retards, then they should stick with crying to their mommies. Their mommies have to pretend to love them, whereas we can just be honest and call them losers.
November 16th, 2007 at 5:21 pm
Rofl Bluehost sucked. I started with them and yes they were 99.9% uptime but max download from a site was 200k. With dreamhost I can get 2500k!
November 16th, 2007 at 6:31 pm
hey, why can i not complaining, it is not about the price issue man.
but 8 hours of downtime is serious, when there is no way i can do a failover automatically! or come on, those who suggested failed over, explain how you do it!
i m on gmt +8 and it is the holiday seasons! so 8 hours of downtime is serious disruption to my webservices.
i do understand the need for dreamhost to do it, but then why didn’t they chose a less peak season?
of cos, i do hope that the recent freq disruption issues will be resolved by this physical move.
November 16th, 2007 at 6:58 pm
I predict failure on an epic scale!
- Roach
November 16th, 2007 at 7:01 pm
What will happen to any email sent to the “spacey” server during this time? I understand I won’t be getting the emails. But will they be held elsewhere, or just rejected?
Thanks for the heads-up though.
November 16th, 2007 at 7:25 pm
@Chris - that might be worth asking support directly. It might also be worthy of an update from DH staff here.
November 16th, 2007 at 7:36 pm
Yeah, good idea.
November 16th, 2007 at 8:11 pm
* Response from DreamHost regarding email sent to the server:
Any emails sent to domains hosted in the spacey cluster will either
bounce or be deferred on the sending server. Which of these will happen
is something that the sending server will decide when it attempts to
contact the receiving server and finds it can’t, it’s out of our control.
November 16th, 2007 at 8:17 pm
My site works fine since about 2 weeks, and i´m on sunnyd.
I hope still this way.
Many downtime hours, but maybe this fix the recent problems with all the people, so good luck guys and everyone here.
Regards.
November 17th, 2007 at 2:10 am
I did not notice much downtime on my account recently. Anyway this is planned and we are being given notice way ahead.
So seems pretty fine to me.
November 17th, 2007 at 5:21 am
Perfect! I will be on the road (and not online) on the 30th and 1st anyway, timing couldn’t be more perfect!
November 17th, 2007 at 7:38 am
Umm, i dont know if this issue is affecting me but my blog is down. (im in kratos server) …
November 17th, 2007 at 7:57 am
You guys really ought to look into setting up a home server for yourself, and then adding to it, taking away, having hardware go bad, get attacked, exploited, and compromised, and see what it really takes to run one of these centers. If you haven’t done that, I think you need to relax, shut up, and let your hosting provider do their job, as that’s WHAT YOU’RE PAYING THEM FOR! They are trying to better your experience, and you’re all bitching about it… You guys are worse than the guys I work with in this union company….you bitch more than they and their wives do. If you don’t like this place, go out, buy a T1, lease a /24 of addresses off ICANN, ARIN, whoever, install your own email, ftp, http, pop3, smtp, nntp, ntp, dns, vpn, LAMP (WAMP), firewalls on your own connections……I bet your shit wouldn’t stay alive a week before China got into it.
November 17th, 2007 at 8:14 am
Its funny, your going to move to a new host, but your “business” is still on a shared server… How is that smart? I have said it once and I will say it again… “If your going to run a business get a dedicated server, and run it your self”. If you want no down time, you do that, and pay:
~ $150.00/month to your ISP
~ $1,000.00 for the server
~ $50.00 for the DNS
~ $10.00/month for the name
aka: $1,810/yr. ($3620/2yrs.)
and an extra $1,050 for everything else.
so that means in 2 years you will have to pay: $4,670
or you can use dreamhost and pay $214 in 2 years.
With that being said, will your site gain you money or set you back with $1,810/yr?
QUIT ACTING LIKE DREAMHOST IS RETARTED THAT IS ONLY ONE SERVER TRY PAYING FOR 500+ AND MANAGING THEM.
This one cluster has over 100 servers, and I don’t know how many clusters they have but I do know the wiki says 500 serves.
November 17th, 2007 at 9:00 am
@Woz - it’s a FUTURE EVENT. Unless you’re Einstein or Hawking (and you’re not, you’re Woz) it’s not capable of conceiving a way for a future event to affect the present.
Sucker’s busted, write support.
November 17th, 2007 at 9:03 am
@Ryan/Just - That’s why I’m here. I don’t have time to babysit systems, I need to get work done. I do the work, they do the hosting. Once in a while I get bit, but so what - MS pushes fixes on Tuesdays and knock my desktop(s) over “Your system has been rebooted…” “But I was WORKING!!!” (I’ve now turned off the automatic application so I don’t get surprised) - BUT it’s still downtime.
November 17th, 2007 at 2:55 pm
@omg, what i wrote wasn’t directed at you, it was directed at the multitudes of others out there who seem to have nothing more to do than bitch about their hosting provider, rather than just shut up and deal with the company trying to expand. If you want a site that has no downtime, run your own, buy the lines, as I stated previously, otherwise, I don’t see the great audacity in bitching, other than making themselves look like complete jackasses.
November 18th, 2007 at 1:02 am
Let’s see… 8 hours is 1/3rd of a day, in turn about 1/90th of a month. All the services I pay DreamHost per month amount to about $12. $12/90 = $0.1333… $0.14 if you round up. I expect to see that credit in my account immediately. Waaah! How else am I going to make 1/10th of my bus fare? How much is your monthly steven? Are you saving up for candy?
November 18th, 2007 at 8:05 am
So, will this improve performance on the cluster?
November 18th, 2007 at 8:09 am
My whole presidential campaign is riding on Dreamhost’s shared server uptime.
November 18th, 2007 at 10:53 pm
Thanks for the heads up…
Keifer
November 19th, 2007 at 3:39 am
MySql on Spacey is gone… down..
Error: Can’t create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
November 19th, 2007 at 3:42 am
…. to clarify….
MySql on server catdog
November 19th, 2007 at 7:20 am
si el sitio sigue cortado mi jefe me va a cortar las pelotas por favor solucionen esto !!
If the site continues down, my boss is going to cut my balls please fix this!
November 19th, 2007 at 12:21 pm
If they compensated everybody, everyone would get $0.13 — 13 cents. According to my calculator.
I love DH. Thanks for the heads up.
November 19th, 2007 at 7:15 pm
I don’t think the issue is that they are having scheduled downtime. I don’t think it’s an issue of cost or reliability.
I think this issue is competence. Let’s just think about the number of ways this particular problem could be perhaps better handled:
1) Don’t put clusters in a full environment. If you were moving all your servers because your data center was shutting down, or because you were getting your own data center, maybe those are moves that “have to be done” and “could not be foreseen”. But a single cluster needing to be moved from one environment to another because of “over-crowding” of your own equipment sounds like someone didn’t run their numbers correctly before it was put in.
2) Dreamhost should be taking every consideration to try to limit the damage of the downtime to it’s customers. Ways to help limit the harm of the downtime:
- Setup a backup MX server (or leave the one you have) running at the old location to have it trap the email until you get to the new data center. Dreamhost should NOT have to rely on other people’s server configurations to insure deliverability. Now, given this won’t solve the problem for the people who maintain their own DNS enteries, it will for all of the customers who are using the DH nameservers. The only trouble to make it happen: a backup mail server with a slightly different configuration from your normal ones.
- Put up a “this site is moving page” on all of the affected IP addresses. I’m assuming that the IP’s will not be moving with cluster to the new location, but even if they are, this is still doable. This will make sure that customers who are running into the pages are getting error messages that don’t make any sense to the majority of the public.
- Assuming the IP addresses again aren’t moving, it would be a very good idea to switch the DNS records before “hitting the road” than after… Fewer delays in getting the site back up (You may already be doing this, but your exact plan of attack isn’t overly public).
November 20th, 2007 at 10:35 am
For all those people who think because it is shared that means downtime is ok. Well it isn’t and I pay a very low price for my shared with another host.I have had the same amount of downtime in 2 years with them that i have had with dreamhost in the past 3 months. I will be moving all my domains to that host before the downtime. Dreamhost can keep their money i don’t want it. By the way they are not the cheapest. I HATE DREAMHOST their site sucks their service sucks stop complaining and move like I am. by the way bluehost is just as bad
a little hint about hosting stay with the little guys they still give a shit.
November 20th, 2007 at 12:39 pm
Hopefully someone reads this and it helps out.
Someone asked how to implement failover. Here is the “jist” without mentioning companies’ names (they don’t pay me to advertise / not a fanboy - just want to keep my business running proactively).
This is not an easy task, it requires planning and learning. Don’t hold me responsible if you mess anything up…this requires knowledge of how to control your DNS settings. These are not step-by-step instructions, they are the idea of what you should be doing to set it up.
No host will recommend this method because it gives the perception their systems are not reliable, plus it pushes business to the competition.
Also, I will not support any questions if you get stuck here, I just don’t have the time. If you get stuck, research it, Google is very valuable as a research tool.
Use common sense if you implement this. Plan it out first before you make the first settings change or you could be sorry. get the right hosting accounts for you…price, services, make sure they have all of the same programs you run at dreamhost or you may have to manually install…that said, make sure your alternate hosts allow you to install if you need to! If you are just doing straight, non-interactivewebsite, it’ll be easy. mail is not failed over unless you fully research this and plan for it. I’d recommend hosting mail wherever you feel is the most reliable…your own server, another host…whatever you decide. Being a business, I host my own with several layers of backup/redundancy in place. If my server goes down, I can still get new mail and send mail due to my failover plan as well as keep all my existing email accessible.
————
Instead of paying around $100/mo or more for dedicated hosting, failover to several shared hosts is cheaper and should work just fine. (just make sure they are on different ISP backbones if you can in case they have nothing in place to protect against an ISP outage).
-You must setup multiple hosting accounts. The failover company I use allows up to 5 failover hosts…it’s your choice how much failover you want.
-You must setup an account at a DNS failover company. Use Google to find one that suits you. You’re looking at approx $15/yr to $60/yr depending on how many hosted domains you want to control.
-You must setup your domains and all of your DNS settings at the DNS failover company prior to switching over so you don’t have downtime when you switch.
-Turn on the failover option for each domain (dns name) you want failover for and point to the IP of your new hosted account. Setup the failover IP address order to however many failover hosts you have purchased (my failover allows up to 5 hosts). Do this per domain you are controlling.
-You must make sure all of your new hosting accounts mirror your primary, so that when failover occurs, your sites are the same. This means backing up your databases on a regular basis and transferring your databases to the new hosts…either do it manually or figure out an automated way to do this. If you run forums or whatever requires a database, you will want the most up to date versions all the time. Basically, when your site goes down, you will be notified via email addresses you program in…I have text messages sent to me also. Unless you host your mail on a different server, don’t use the email address from the domain you are failing over…setup a free account somewhere or use an alternate email you already have. When you get the alert, upload the latest version of your database you have. I won’t get into that anymore, research how if you do not know how.
-Make sure all htaccess passwords are set and the same, you don’t want to give unfettered access to protected areas of your primary site, and you want to keep passwords and everything working in case you have users who use passwords.
-***To reiterate, use common sense…whatever modifications you have made on your original site must be reproduced on your alternate hosts when you upload your sites.
-Upload all site contents to each alternate host. Make sure everything is setup and in place prior to changing your nameservers over.
-Then, when all your DNS settings are setup and all your sites are uploaded and working as they should with the same data, you must pass DNS control over your domain name to this DNS failover company (you will have full control over everything from there). If your DNS is hosted at Dreamhost, you must pass DNS control from there…same with Network Solutions or whoever you have (meaning, point your nameservers to the DNS Failover company’s nameservers).
-Wait for DNS propagation…1-72 hours for complete propagation across the internet.
-Use dnsstuff.com or dnsgoodies.com…or whoever you like for checking on your DNS settings taking effect.
-Probably good idea to wait at least 24 hours befor testing failover.
-You can test failover by changing the monitored website to an html page that does not exist…wait for the alert and check the site.
-Failover scans cycle every few minutes, so you could have a couple of minutes downtime until the next scan cycle completes and detects your outage. Failover will automatically route traffic to the new host you setup…and as long as all your DNS settings are in place and databases are up to date, you should experience downtime only during the failover time period. I notice a maximum of about 6 minutes and a minimum of 2 mins downtime during failover.
-Again, use common sense, setup all your failover options to the way you want it to work…they have automatic failover back to original site or not, so you can copy your latest database back to the original site prior to switching back.
-reasearch, plan, and research…then plan some more prior to implementing.
Good luck!
November 20th, 2007 at 12:57 pm
I would appreciate if they could put some page which would say, “Under construction/moving, etc.,” something like that.
Just showing 404 or not allowing us to put a page is not good strategy.
November 20th, 2007 at 1:23 pm
Sweet post Red. Exactly what I was looking for and the right level of detail. If you can’t figure it out from that post you PROBABLY shouldn’t be trying it.
Thanks!