Spunky and Central Move
We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.
Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.
For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.
UPDATE — Sat Mar 22 02:45:26 PDT 2008 –
Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.
UPDATE — Sat Mar 22 03:40:27 PDT 2008 –
Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.
UPDATE — Sat Mar 22 06:01:28 PDT 2008 –
Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!
UPDATE — Sat Mar 22 07:04:35 PDT 2008 –
We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.
UPDATE — Sat Mar 22 09:55:54 PDT 2008 –
The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.
.
March 22nd, 2008 at 12:34 am
Shawn, who are you? You must not be in business. You don’t argue with customers. Sure, it’s not Dreamhost’s policy to e-mail people about downtime. That means they’re free of any legal issues — good. In terms of business, they are pissing people off. I sometimes complain about cold fries at McDonalds’, and they give me a fresh bag of fries. And so I complain about Dreamhost’s downtime policy, and I expect them to listen. As has been said in prior comments, it’s not much of an effort to e-mail your customers about downtime. I’m not a novice web developer. I’ve been doing this for many, many years, and I’ve always used dreamhost for my clients’ sites. I’ve never experienced a situation like this.
I highly suggest that dreamhost e-mail affected customers in the future.
-Joshua
March 22nd, 2008 at 12:34 am
Shawn, who cares what the policy has been. If enough customers request it, the company should respond by sending out email alerts. Nobody reads the fine print upon signing up that they’ll never get a notice by email and by golly they better use RSS or else. WE WANT OUR EMAIL ALERTS!!
(Quick, somebody start a petition!)
March 22nd, 2008 at 12:35 am
Businesses in the US, with a majority of their customers being the US, are likely going to use their own time. It doesn’t make sense to alienate the majority of your clients.
Notice they’re not fluent in every language either, even though there are customers all over the globe that don’t speak English. I’d guess that if someone is capable of converting English to their own language, they can probably figure out the time.
March 22nd, 2008 at 12:36 am
@Shawn: My mail server (spunky) should be affected, but that would mean that everything else I’m running should be fine. It isn’t. It went down at 9:00 when the outage started. The list is obviously incomplete -or- only includes the names of the physical hosts and not the virtualized hosts that they use for sharing and PS.
They gave us a week’s notice, which isn’t too bad except that the info is incomplete. I looked at the list of affected hosts and thought at first that my websites would be unaffected. Then realized that because it’s my “personalized” feed, I must be affected otherwise they wouldn’t have sent it. Anyone subscribed to the status blog as a generic feed wouldn’t have been able to make that realization.
For updates to the blog when unexpected problems happen, the RSS feed is a great resource. It’s timely and allows DH to push information out for us to pick up in a timely manner.
The only time I’m advocating email is for long-lasting _PLANNED_ outages. There’s been a lot made about the fact that it’s a discount service, $10 a month or whatever. I’m on PS and pay about $40 / month in PS fees. For that kind of money, an email for a 12-hour PLANNED outage isn’t outside the realm of feasibility. Is the current policy to do blog-only? Yes. I’m advocating an amendment.
March 22nd, 2008 at 12:36 am
Shawn have you really been on here for 2 hours posting on this thing? Sheesh, get some sleep.
March 22nd, 2008 at 12:38 am
Well that’s nice for you all, I’ve scanned all my mailboxes and junk..high and low, and not found a single email warning about this downtime..
It’s an extremely LONG time to be down…I have no mail, no websites, nothing.
March 22nd, 2008 at 12:38 am
Shawn hasn’t just been posting tonight, he’s been arguing with people here about this downtime ALL WEEK.
March 22nd, 2008 at 12:39 am
Agreed that an email should have been sent. I do subscribe to the Dreamhost Status blog in my feed reader, but when a post starts with “Resolved” I quickly scan past it as it appears it’s not an issue any more. Maybe they should remove the “Resolved” prefix from these feeds, or at least make them reflect some meaningful status instead of confusing us. Of course, an email sent to everyone affected would have been incredibly easy and helped a lot of folks out.
March 22nd, 2008 at 12:40 am
Shawn obviously does not work or represent Dreamhost. I don’t know what his purpose is here besides being an ass.
March 22nd, 2008 at 12:40 am
Why are these things always done at the peek time? The weekend is the most important time for me business wise, and the servers are always off, down at these times
March 22nd, 2008 at 12:41 am
melrose is down. my site is down. if i had gotten an email i wouldn’t have pushed traffic to my site tonite. and i would have put up a notice on my site to expect downtime for maintenance.
but no notice of a 12 hour down?
this is really bush league.
March 22nd, 2008 at 12:41 am
@Peter - Heh… yea. Ya think?
March 22nd, 2008 at 12:41 am
Just a point that I see raised here a lot…
Not everyone pays only $10 a month for hosting. Some of us are paying more than that and are on older plans.
I was actually planning on migrating a whole heap of DH sites over this weekend to my new DV Hosting and I can’t because everything is down. I am laughing at the situation, and luckily my clients have a healthy sense of humour as well… but… man
I started on DH because a friend recommended them to me when I first started out, but obviously I made a mistake and now have to migrate. Its annoying, I can live with it, but I am also peeved at the “OMG you pay TEN DOLLA FOR HOSTING!!!111!! You CAN’T COMPLAIN!” — when not everybody is on that plan.
But, $10 a month is still $10 a month. Sure, it isn’t going to get you rock solid uptime, but 12 hours downtime (especially around the time a lot of webbies catch up on work) isn’t really acceptable either.
Ah well… have another chocolate bunny I guess
March 22nd, 2008 at 12:43 am
well my site is down and it IS affected. this sucks. if i could easily move a 200GB site I would.
March 22nd, 2008 at 12:43 am
For example, even this post is listed as “Resolved” in the feed, when it is obviously not “Resolved.”
March 22nd, 2008 at 12:43 am
Down too long time!
Maybe I will change my host to other company.
March 22nd, 2008 at 12:46 am
Kuma, I’m not a webhost professional but I did do some sysadmin back in the day and I’ll answer and say that I think this is taking 12 hours because DH is being lazy. It’s just easier and cheaper for them to physically move the whole spunky cluster than purchase new boxes and duplicate the sites over onto these boxes and then flip a switch when it’s ready.
PS Panel is NOT back up, like they say…I’m betting this downtime goes for much longer than 12 hours.
March 22nd, 2008 at 12:46 am
You should because you agreed to it. They don’t force you to host here.
Notice the majority aren’t complaining about it. They did change it to what it is and only people that can’t follow directions have a problem with it.
Speak for yourself. Anyone that doesn’t read and understand something before agreeing to be bound by it is a dumbass.
Then go to a host that uses them. Seems simple enough to me.
The whiners ALWAYS overestimate their numbers. The majority of their customers are smart enough to follow directions and not cry in blog comments about not getting enough babysitting for their hosting payment.
March 22nd, 2008 at 12:48 am
interesting shhhhyawn. so that’s how professionals do it. so is there an easy way to migrate a 200GB site?
March 22nd, 2008 at 12:48 am
The majority aren’t complaining HERE Shawn because they don’t even know where this status page is…OR they have lives and are asleep. Unlike you. That doesn’t mean they are satisfied customers.
March 22nd, 2008 at 12:50 am
if only i had known 1 year ago………what i know now.
March 22nd, 2008 at 12:50 am
Gosh Pete, what are you hosting on there?
I have 15 clients on my account and they only come in at 10GB
I feel your pain with moving though, its why I have been procrastinating about moving too… and then I finally set aside time to do it this weekend, and …voila 
March 22nd, 2008 at 12:50 am
“interesting shhhhyawn. so that’s how professionals do it. so is there an easy way to migrate a 200GB site?”
Pete, FTP the site down to your computer and back up again on your new host. If you are using MySQL then just export your database code to an SQL file and import it on your new database. It’s really just the time it takes to download and upload but it shouldn’t be that hard if it’s just one domain.
March 22nd, 2008 at 12:51 am
@KumaZatheef and @shhhhyawn:
I agree, a very possible reason for such a move during a long downtime window is due to lack of infrastructure or time to do a gradual migration of servers.
March 22nd, 2008 at 12:51 am
My websites on Krusty are down. Is this a complete outage for Dreamhost? I thought it was only going to affect central services and the Spunky cluster?
March 22nd, 2008 at 12:51 am
> UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
>
> Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still
> underway. We’ll post more here later.
Still no sign of warhead being available. My website is still gone, and of course, no email. Isn’t a big part of the point of distributed servers so you can move/change a few at a time, keeping the whole of the group up so any system-wide downtime is largely invisible?
March 22nd, 2008 at 12:53 am
I’m gonna need a LOT of coffee, to stay up enjoying the fireworks here.. folks slugging it out.. doors not hitting ya on the way out.. &c.
March 22nd, 2008 at 12:55 am
Kuma, my impression (and I could be wrong) is that the staff here is young, idealistic, and a bit inexperienced. As in, the 50 year old conservative IT guy who has some sense and a long resume probably wouldn’t want to be working at Dreamhost.
March 22nd, 2008 at 12:55 am
Any time the good outweighs the bad, it’s worth it. The 12 hours is probably nothing compared to problems they might have seen coming. They sure didn’t do it because they like working better than sleeping.
A physical move cannot be done without downtime by any host.
There are also hosts that take days to move and don’t announce it because they don’t want people to leave. They just apologize when they’re back up and hope they’re too lazy to move at that point.
In fact, hosts have sold out (including hardware) and just cut everything off, put the servers on planes and shipped them across the country to the buyer. That always goes over well. The buyers are silenced by contracts and get to clean up the mess when things are back up and running. You just calculate the losses ahead of time and work it into the price.
There are much worse things out there than what Dreamhost customers see here.
March 22nd, 2008 at 12:59 am
A physical move CAN be done without downtime.
It’s simply accomplished by having the secondary location up and running (synced with the data from the original).
You then load balance the incoming request to both locations.
When you’re ready to move, just take the old location out of the load balance rotation.
March 22nd, 2008 at 1:00 am
@Alex
See, any admin worth their salt would send out a shutdown notice 5 minutes beforehand that way everyone knows to save their work and log out.
So what if the outage starts at some whacky US time. My clocks are set to a different time, and I’d be nice to know that the server is going down in 5 minutes and not discover it’s gone down now.
and it’s not that fucking hard to do either: “shutdown -h +5″ and perhaps “shutdown -h +5 See planned outage on DH Status” if you’re a nice person.
March 22nd, 2008 at 1:01 am
Young, yes, but wrong about the rest. They have more experience and have been in business a lot longer than most hosts out there have been. A host is lucky to survive a year. Make that an hour, if they try to compete on price with the big boys.
Dreamhost wrote their own control panel before a lot of these guys even paid for their first CPanel license.
Off the top of my head, I doubt I could name 5 other hosts that have been around as long or longer. There aren’t even many that are bigger–and that’s worldwide, not just in the US. I don’t count on them passing biggies like Godaddy/Wildwest, but they certainly haven’t stopped growing.
March 22nd, 2008 at 1:01 am
follow up…
But I’ll state again, DH probably does not have the kind of infrastructure or time to do a move without a long downtime due to lack of resources.
March 22nd, 2008 at 1:01 am
oh, it’s only 60GB. I think of it in terms of DVD storage media, so that’s about 12 DVDs. i guess i could move that over a week or two……
March 22nd, 2008 at 1:02 am
shawn runs dh.
March 22nd, 2008 at 1:03 am
Alex, that’s exactly what I was thinking, and why this 12 hour downtime is just bad business practice and laziness on the part of DH.
March 22nd, 2008 at 1:04 am
if I got a T1 connection maybe 48 hours and $200 of co-lo time
March 22nd, 2008 at 1:05 am
That’s not simply a physical move and you’re not going to find any budget hosts that are going to buy two of everything to do that, when the point is to move hardware–not just accounts.
March 22nd, 2008 at 1:05 am
http://www.dreamhost.com/ website was down but is back online
maybe a sign that recovering is in their way
Please sort it out ASAP
March 22nd, 2008 at 1:08 am
if they can screw all my cloaking last week (without an email letting me know, one of my colleagues emailed me about it) then they can do a 404 redirect to a “sorry we’re down for awhile” page.
March 22nd, 2008 at 1:09 am
You just proved yourself wrong by posting that. If I ran it, I’d close the comments, or at least require that some sort of registration was required to post, that linked to an actually customer account.
Plus, I’d be too busy rolling around in a huge pile of money to post here, so I’d probably just have my butler fill in for me.
March 22nd, 2008 at 1:10 am
Seconding you Kuma…and here’s the thing. It’s a lame excuse to say that DH can’t afford to buy new boxes for this move. If they are growing in leaps and bounds as Shawn seems to think, then they’ll need extra boxes soon enough. So there would be no extra cost in getting new servers to mirror everything, since the old servers would be re-used for new clusters. Right?
Let’s see Shawn talk his way out of this one…only, I’m going to bed. Have fun!
March 22nd, 2008 at 1:13 am
hey guys, what’s the hell is going on here,. dreamhost is rubbish. will definitely move to other place. this is completely not fair. how come 12 hours.
they fucking saying green host. this is Red with full of bullshit.
March 22nd, 2008 at 1:13 am
@Shawn
>That’s not simply a physical move and you’re not going to find any budget hosts that are going to buy two of everything to do that, when the point is to move hardware–not just accounts.
They didn’t have to “buy two of everything” if they did a gradual relocation and repurposing of the servers (which would have taken more time, yes… but at least it would be less downtime).
Further, the You Get What You Paid For excuse is a very poor excuse. Only an idiot would attempt to run any business with that model. Yes, DH is a budget host, but that doesn’t mean it can be excused for not being attentive to the needs and requests of it’s customers.
March 22nd, 2008 at 1:14 am
1 BBB complaint lodged.
March 22nd, 2008 at 1:16 am
We’re the customers who are worth 100 customers to DH. We tell the newbies and others where to get their hosting.
March 22nd, 2008 at 1:17 am
Hi!!
Come on, next time send us a status mail so we can made other things, just send us a mail so we can stop working
Ciao, Tristan
March 22nd, 2008 at 1:17 am
Can someone explain why melrose is down too, Shawn?
March 22nd, 2008 at 1:18 am
@Pete
Agreed. A good customer will complain in order for the business to know what it’s doing wrong. A bad customer will just leave without saying anything.
March 22nd, 2008 at 1:18 am
Comparison is part of shopping and choosing. If you want to go somewhere better, go. No one is forced to be here. Everyone that whines about Dreamhost and how they do things chooses to be here over every other host out there, so either it’s not as bad as they pretend it is, or they’re trying to take stupidity to a whole new level.
You basically said the same thing as the other guy… and that’s not going to happen with a budget host, expecting them to buy two of everything and way more labor.
You are not going to get that in this price range anywhere, unless you go with a host that has 1 server and doesn’t mind buying a single spare.
12 hours might be super-fast for what they’re doing. It’s hard to say without seeing just how much they have to move, how far they have to move, etc…
What they’re doing is a lot more complicated than helping a neighbor move a couch and a few lamps two houses down. I’m sure they want to get it done and go home, so they probably haven’t picked the slowest way to do it.