Spunky and Central Move
We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.
Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.
For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.
UPDATE — Sat Mar 22 02:45:26 PDT 2008 –
Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.
UPDATE — Sat Mar 22 03:40:27 PDT 2008 –
Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.
UPDATE — Sat Mar 22 06:01:28 PDT 2008 –
Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!
UPDATE — Sat Mar 22 07:04:35 PDT 2008 –
We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.
UPDATE — Sat Mar 22 09:55:54 PDT 2008 –
The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.
.
March 22nd, 2008 at 2:17 am
@ Not Me - You could do it, technically… but the network traffic would be a killer… the servers talk in a cluster, independent to the other clusters… so when you tell them to talk across the network, into the internet, to the other network, and into it’s partner cluster, you’re creating a lot of traffic… and a security hole, since that traffic is not normally exposed to the outside world.
Matt
March 22nd, 2008 at 2:18 am
@darcy
My IQ is just fine thank you. Unlike some, I’m pointing out that it’s not hard for Dreamhost to implement these features. But hey, keep taking pot shots at people.
@Not Me
Clusters don’t have to be in the same room, or even in the same state. A local University’s cluster is spread out between two cities.
March 22nd, 2008 at 2:19 am
Please For GOD SAKE solve this problem
March 22nd, 2008 at 2:19 am
blah, blah, blah, blah……
For 20 bucks a month Dremhost gives you:
557.23 GB Disk (Grows 2 GB / week)
10.82 TB BW per Cycle (Grows 40 GB / week)
Just were the hell do you get such a good deal from any toher host. You can do ANYTHING you want with your account… rewrite the php file, cron job it, stack it, sub domain it till the cows come home, Mysql db’s till you overdose….. etc… etc… etc….
And you n00bs still complain. I hope ALL your sites goto hell.
One week notice and you’re blaming some else for your ignorance.
Classic.
Good luck Dreamhost dudes…. keep up the good work…..
March 22nd, 2008 at 2:20 am
My sites are on a webserver, janky, that isn’t listed as being in the spunky cluster. So why is it affected?
Also, it seems DNS is somewhat screwed up. Is this to be expected with this move? Does it include DNS servers as well? One of my sites is resolving to one ip, which currently works, while the others are resolving to a different ip, which hangs.
March 22nd, 2008 at 2:21 am
Yeah, you all think you have problems. My server’s never listed in their “these are the servers we’re fucking with” listings, yet it’s always fucking affected by every little upgrade Dreamhost decides to do.
In the past, it’s been just a few minutes here and there, maybe an hour or so of downtime, no big deal.
This time, my site has been dead ALL FUCKING DAY. Now, I realize my site’s not your ordinary “grandma’s cooking website” garbage… it’s a high traffic file hosting site that I’ve shoehorned into a cheap Dreamhost web hosting plan. But now this is ridiculous. I don’t make near enough from that site to warrant any better service, and hell’s I know that the bandwidth my site uses isn’t that much (although I’d love to get some insight into the CPU performance of my scripts).
Fact of the matter is, the site’s been dead to the world ALL DAY LONG. Not even so much as a “host is down for maintenance” post linking to the blog or something. That is not hard to do, as I’ve done it before while moving my site to a new host.
What the up is?
March 22nd, 2008 at 2:22 am
Tandem, are you a negative-thinker? Everything is doubleplusgood!
March 22nd, 2008 at 2:23 am
@ Tandem - Yea, seems to me the DNS is a little flaky atm… I’m sure it’ll be fixed. I don’t think Abazaba is up yet (wish they’d have done this in Alphabetical order
) They moved 2 of the 3 DNS servers, and the DNS issues would cause any site to be down, even if they aren’t in Spunky… including the VPS’s.
Matt
March 22nd, 2008 at 2:24 am
Haha - I knew about this a week now and STILL forgot to warn the members on my site.
I’m never going to hear the end of this.
March 22nd, 2008 at 2:25 am
Darcy
[quote]Dreamhost notified us of this[/quote]
When..? I can’t remember reading anything about downtime..?
[quote]http://fuckdreamhost.com[/quote]
hahaha…
March 22nd, 2008 at 2:26 am
Well done guys!
March 22nd, 2008 at 2:27 am
They warned us here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
And if you have an RSS feed (as you should) you knew… refer to my previous comment about that :).
Matt
March 22nd, 2008 at 2:27 am
And, after reviewing the list of servers in the cluster, I find that my server (dionysus) is also dead but not shown in the list.
Nor did I receive any emails from Dreamhost about this outage. The last email I got from Dreamhost was the Feb ‘08 newsletter, which, while entertaining and refreshing, made no mention of an upgrade or outage. Probably because my server wasn’t even LISTED in the affected cluster!
Fail… fail… fail.
March 22nd, 2008 at 2:28 am
@Daniel - Well yes I’m sure you could, the same way the information makes it to clients for one, but I’d imagine that would really slow things down if say, the file server, web server, and SQL server were all hundreds of miles apart, for example. You could also hook the DVD player in the analogy up to transmit its signal however far you wished, but there’s a difference between “possible” and “practical” IMHO, and I would definitely expect better performance and more convenience in the long run if both devices were say, in my living room.
March 22nd, 2008 at 2:31 am
I’d like to suggest people read at least the same page of comments they are posting into before posting… because people are saying the same things over and over again…
@ Falcon… as I’ve explained, they moved some DNS servers.. and they are being flaky atm… DNS errors will bring down sites, no matter what machine they reside.
PLEASE read a little before posting. Seeing the same complaints over and over again is getting tiring.
Matt
March 22nd, 2008 at 2:32 am
Why would I look at the status page unless there was something wrong, do you scratch your ass before you get an itch..?
This should have been emailed, I don’t agree with all the bitching and ‘I’m gonna leave’ crap.. what an arse it would be to relocate all my sites and domains.. christ, shit happens… but it’s nice to know anyway possible if it’s going to affect so many people.
March 22nd, 2008 at 2:36 am
@ Darren -
Do you wait for your computer network at work to catch a virus before you see what viruses are a threat? No! Am I the only network administrator here?
You are trained to be proactive… to know what might take down your network BEFORE it happens. It only take a few minutes of my day to sit at my computer go to my Google News Reader (and I can even read it on my Blackberry) and see if their are any notices that apply to me. If not, cool… if so, then I act on them… after that, I read the latest security vulnerabilities, then I go to lunch. I don’t understand any good network administrator that would want to stay on top of things that might affect their customers.
Matt
March 22nd, 2008 at 2:36 am
Shawn, it’s way past your bedtime! Go to bed now - You can play with your friends in the morning!
Shawn’s Mom
March 22nd, 2008 at 2:37 am
I don’t understand all that flames around here.
I’ve moved servers between datacenters some time ago, and it need lots of *careful* planning just to keep things up and running.
BTW - They’ve started their migration on their friday night in holidays. That should leave most of their small business clients fine.
March 22nd, 2008 at 2:38 am
“I don’t understand any good network administrator that would want to stay on top of things that might affect their customers.”
– Er… I meant wouldn’t.
March 22nd, 2008 at 2:38 am
My main server “hotdog” has been {my site} has been Been down for over 8 hours now, bot 2-3 hours, saturday is our busiest day, can someone advise, we are on spacey, which is not mentioned above
March 22nd, 2008 at 2:40 am
I dont carry a torch for DH. But they really should realize that not everyone does RSS but everyone has email. It works for the newsletter and billing. It should work for a 12 hour down.
So to repeat……email. That’s all it would have taken.
And now back to the foul mouthed defenders of the faith.
March 22nd, 2008 at 2:40 am
@ Ark -
#
322
Matthew Says:
March 22nd, 2008 at 2:31 am
I’d like to suggest people read at least the same page of comments they are posting into before posting… because people are saying the same things over and over again…
@ Falcon… as I’ve explained, they moved some DNS servers.. and they are being flaky atm… DNS errors will bring down sites, no matter what machine they reside.
PLEASE read a little before posting. Seeing the same complaints over and over again is getting tiring.
Matt
March 22nd, 2008 at 2:41 am
@Shawn’s Mom - You should see what I found out your son is up to while I was doing some snooping on the web! http://shawn-is-pathetic.com/ It’s no wonder he’s so bitter, pretty sad stuff.
March 22nd, 2008 at 2:42 am
No matter how loud you complain, NOBODY will listen to you. In case you didn’t get it, this a just release valve so people say their own and then forget about it.
But no matter if you’re a Dreamhost fan this is business man and taking out of business thousands of sites for at least 12 hours shows how serious us customers are taken.
Just expect apologies and wise ass comments from DH stuff and everything will be as before.
Nikos
March 22nd, 2008 at 2:42 am
I’m trying to run a business in France - busy Easter weekend hoping to get lots of bookings for summer season, get up this morning and I can’t get into my webmail - remember you have customers this side of the World as well
March 22nd, 2008 at 2:44 am
@sm -
You can’t please everyone in the global market… and the truth is, they are a US company, and most of their customers are US customers… so you go with the bigger portion. That, is business.
March 22nd, 2008 at 2:45 am
NO NOTICE, THANKS!
8+ HOURS DOWNTIME SO FAR, THIS IS RIDICULUS!
March 22nd, 2008 at 2:45 am
Nikos is right.
this proves DH doesn’t give a damm about its customers.
March 22nd, 2008 at 2:46 am
@ JAMES -
This is the last time I’m going to tell people to jump into Web 2.0 and get a damn RSS feed… then you’da had notice.
On an unrelated note, these CAPTCHA’s are getting anoying..
Matt
March 22nd, 2008 at 2:48 am
Well, my site is back. But the DNS needs a swift kick.
March 22nd, 2008 at 2:49 am
Just logout and back in James. And my email works just fine for that silly newsletter. it’s obvious Dreamhost counts on most of its customer never knowing this ever happened.
C’mon, do you take us for idiots?
March 22nd, 2008 at 2:51 am
Location - DREAMHOST HQ : Time - 5 minutes before Server move
Employee #1:
A hat is held out and 3 tickets are pulled by employee #2, the first is read out
John:
Ticket number 2 is called..
Dave:
Ticket number 3 is called
Bill:
5 minutes later John is curled up in his exec chair watching the monitor, Dave is counting to 10 at John’s request and pressing F5.. Bill is already out the door to retrieve the next hours stock of coffee and Dough’s.
March 22nd, 2008 at 2:51 am
Spacey is down as well and it is not in the list of effected machines.
And before everyone shouts me down with DNS rubbish, IP addresses are resolving fine and DNS is resolving fine - just webserver at the end of it.
I was aware of the down time but was not warned it would effect my site as well.
March 22nd, 2008 at 2:54 am
It’s humorous to read all these whines.
Looking at this status page, it’s clear that they gave us about a week’s warning about this maintenance. If you don’t subscribe to the feed here, you get what’s coming to you.
I have had no problems with DH since I joined them over 2 years ago. They are a great webhost with some really great features. This outage is just regular maintenance. No matter what host you sign up with, you are going to have maintenance performed and an outage that will probably be inconvenient.
Just suck it up and wait it out. I’m sure they will be back before you know it.
March 22nd, 2008 at 2:55 am
@ Matthew: Here I am talking to myself, but do us all a favor: SHUT UP.
If they were to be screwing around with DNS, they should have told EVERYONE via EMAIL, not (as Darren says) scratching your ass before it itches and watching a blog for outage shit. When there are about a thousand downtime posts on this blog a day, and only one in a million actually applies to you, do you REALLY think it’s worth keeping in my tabs just to watch for never-coming failure? NO. IT’S CALLED EMAIL. USE IT.
As it is, I use a Squid proxy that seems to cache DNS entries to fucking infinity. And thanks to the fucking money-grab asshats at DNSStuff that took out the VERY BASIC free DNS lookup tool, I can’t even try a second testing point to see if I’m getting a stale DNS result. Man, fuck the internet.
March 22nd, 2008 at 2:58 am
In the interest of fairness, there is one thing I’d like to point out. When people have complained about the timing and the time zone, the same people who have been telling those who prefer email - which have been the overwhelming majority here - that they shouldn’t expect anything except for what DH prefers, suddenly turn around and say that DH has to go with the majority of their customers on these other issues. The point is still moot I suppose because the only place for people to actually address that is through the Suggestions section of the panel, but still, I couldn’t resist pointing out the double standard.
March 22nd, 2008 at 2:59 am
Falcon
You need to learn how to read, I was the one that said why the hell should I keep tabs on a page, I actually said I would have prefered an email..
Christ, I’m outta here this is pointless drivvle…
March 22nd, 2008 at 2:59 am
@Falcon Says. on March 22nd, 2008 at 2:55 am : ” Man, fuck the internet.”
Yes. Fuck it, Fuck it good!
http://fuckdreamhost.com
March 22nd, 2008 at 2:59 am
“Indeed, an e-mail for planned outages like this should be sent out as a courtesy to your customers.”
What he said. I’m pretty miffed by now, I was affected by the last move too with no warning - and I always seem to be in the last 1% of accounts that are not working properly when work is over.
Seriously, a heads up is REQUIRED in when offering uptime as a service.
March 22nd, 2008 at 3:01 am
All you mother fuckers, licking up dreamhost ass, 500GB HDD and BW is mother fucking nothing if your website is down for fucking 4 hours, did you listen? FUCKING 4 HOURS!!!!! And it happens almost every week, either fucking wrong router password or something like that, but dream fucking host does it all the time. what kind of fucking business is it. It’s not about 20 or 200 bucks, some sites don’t need dedicated server but yet very important.
March 22nd, 2008 at 3:02 am
I hope this quoting thing works…
@ Darren:
[quote]You need to learn how to read, I was the one that said why the hell should I keep tabs on a page, I actually said I would have prefered an email..
Christ, I’m outta here this is pointless drivvle…[/quote]
Uh, I don’t think I’m the one that needs English classes… I don’t know where you got that idea but that is EXACTLY WHAT I AM AGREEING WITH YOU ON. Wow, you know the internet has sunk to a NEW LOW when you’re arguing about AGREEING with each other. Fucking hell.
And I just confirmed that DNSStuff DID, in fact, strip ALL free tools out of their system. Not even a DNS lookup tool anymore. FUCK THEM.
March 22nd, 2008 at 3:04 am
Get a fucking life.
Get a fucking life.
GET A FUCKING LIFE!!!
March 22nd, 2008 at 3:04 am
price web server still down
Good luck DreamHost Team!
March 22nd, 2008 at 3:07 am
Uh, I don’t think I’m the one that needs English classes… I don’t know where you got that idea but that is EXACTLY WHAT I AM AGREEING WITH YOU ON. Wow, you know the internet has sunk to a NEW LOW when you’re arguing about AGREEING with each other. Fucking hell.
Hahaha.. I see, but I fear it’s your grammar my good man. (as Darren says) should have come before ‘not’ as it gives it a whole new meaning.
March 22nd, 2008 at 3:07 am
I migrated a client’s website on Friday, no how fucking silly do I look, still good thing the email isn’t working either.
I still can’t believe you chose to do these things a peek surfing time and a holiday weekend to, I losing business guys. Have you never hear of Parallel running? It makes me wonder if you where ever seriously hit with a problem, we would all have an outage of services get your self some mirror servers, that way when you are working, all our services will still run.
No doubt you are using what servers you have to get more customers on board, look after the one you have now.
I’m with out any service until 5pm (GMT) that a whole days of business gone for me.
March 22nd, 2008 at 3:09 am
Voila!.. Everything is back to normal…..
March 22nd, 2008 at 3:11 am
@Shawn
I’m here because I’ve gotten messages from clients all night about sites being down. Why are you here? I beg you, PLEASE, humor me, tell me why you are here right now. I’d really, really love to know what reason you have to tell OTHERS to get a life.
March 22nd, 2008 at 3:13 am
i haven’t had any problem with dreamhost in the past a part of the slowness sometime, I was also aware of the maintenance schedule that we are discussing now,
but my only concern is to know when my 3 websites we’ll back online…
March 22nd, 2008 at 3:15 am
[Quote]
When there are about a thousand downtime posts on this blog a day, and only one in a million actually applies to you, do you REALLY think it’s worth keeping in my tabs just to watch for never-coming failure? NO. IT’S CALLED EMAIL. USE IT.
[/Quote]
Well, when you get the chance, go to the webpanel, bring down the System Status Drop from the right hand side at the top, and you’ll see that their is an RSS link at the bottom there. Add THAT to your reader, and it’ll only show you things that APPLY to you… I get maybe 2 daily… usually none. This was there though.
Matt