Spunky and Central Move
We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.
Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.
For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.
UPDATE — Sat Mar 22 02:45:26 PDT 2008 –
Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.
UPDATE — Sat Mar 22 03:40:27 PDT 2008 –
Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.
UPDATE — Sat Mar 22 06:01:28 PDT 2008 –
Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!
UPDATE — Sat Mar 22 07:04:35 PDT 2008 –
We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.
UPDATE — Sat Mar 22 09:55:54 PDT 2008 –
The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.
.
March 21st, 2008 at 11:17 pm
@45 Grey Hodge,
No, they did not tell us a week ago! It is not the CUSTOMER’S responsibility to check every day for 12 hour maintenances. Anything this long should have been communicated directly to us or had huge warnings put on our control panels.
March 21st, 2008 at 11:18 pm
FUCKING MOTHER FUCKING DREAMHOST!!! I DONT KNOW WHAT FUCKING JEWS OWN THIS COMPANY
March 21st, 2008 at 11:20 pm
Yes, they did tell you a week ago. And yes again, it is YOUR responsibility to stay up to date.
Do you need them to do all of your thinking for you, or just when it comes to something super hard like subscribing to an RSS feed.
Maybe they should also send smoke signals out, in case they have any Indian customers that can’t follow simple directions?
March 21st, 2008 at 11:21 pm
Why all of my websites hosted at Dreamhost are up? You can check http://www.kojakdirectory.com its up here
March 21st, 2008 at 11:21 pm
my site still down…. why
March 21st, 2008 at 11:21 pm
Ditto for all the comments on getting an email about something like this - I subscribe to the feed but somehow when I see “spunky cluster” it doesn’t ring any bells - I have to dig down through a big long list to see if it applies to me. Plus the post title should emphasize “12 hour outage coming!” that should get more people’s attention.
I think an option to get an email or text for any planned outage more than 1 hour should be provided.. It wouldn’t really cost them that much. Lets face it 12 hours is an instant downtime hit of 0.14% over a year. So for a huge number of customers DreamHost just blew any chance of 99.9% uptime for the next year - let alone 99.99% - that by anyones books is a major outage.
Oh well…
Even so I’m still a satisfied DreamHost customer.
March 21st, 2008 at 11:22 pm
damn you amusing comments
i need to sleep
March 21st, 2008 at 11:23 pm
Moschops: I’ll fuck your mother in her skull if you say you’re a fucking SATISFIED customer. Is that a fucking hosting which is always down EVERY NIGHT?
March 21st, 2008 at 11:23 pm
Shawn, get over your faux smartass RSS promo spiels. Not everyone cares to use RSS, let alone subscribe to any feed Dreamhost may supply. This fact ought to have been realised by DH and emails should have been sent out accordingly.
March 21st, 2008 at 11:23 pm
So, anyone got any recommendations for a good host?
March 21st, 2008 at 11:26 pm
Hi my name is Shawn and I’m a huge dumbass. Who wants to do me up the pooper?
March 21st, 2008 at 11:29 pm
Shawn and other fucking assholes, take your fight somewhere else and suck each other cocks or spoon each other, you fucking fags.
March 21st, 2008 at 11:30 pm
Is the site up yet?
March 21st, 2008 at 11:31 pm
This kind of sucks.
March 21st, 2008 at 11:33 pm
Seriously… I’m a busy guy, as many of you folks are. It shouldn’t be up to me to repeatedly check an RSS feed to see when my sites are going to be down. If I know a client’s site is going to be down, I need to be able to let them know… and I should be TOLD, not have to hunt for the Easter egg in RSS. How hard is a mass email? Man…
March 21st, 2008 at 11:35 pm
It is nice, when you compare it to your hobby, which is being one of those retards.
I can’t say that I’m surprised that nothing you said makes any sense.
You forgot to point out your useful contributions. I actually did contribute by telling some of the dumbasses to subscribe to the feed if they ever get tired of being clueless.
You’re just another moron that’s crying because you can’t be a little baby here and not get called out for it. Go cry on your mommy’s shoulder if you need to be comforted. It’s not going to happen here.
March 21st, 2008 at 11:37 pm
If you can’t be bothered with something that simple, then don’t pretend keeping your clients informed is even the least bit important. Ask your clients if they’re smart enough to subscribe to a feed, then let them open up their own accounts here and bypass your incompetence.
March 21st, 2008 at 11:39 pm
I agree about the email thing too…would have been nice.
@tara (#184): I’m on PS and my servers aren’t listed in the massive server list. Way more than “a handful” are affected by this… but with DH’s track record I fully expected it for my stuff.
@hex (#165): class action? hardly. try reading the TOS before you decide to talk lawsuit and accuse them of false advertising. It’s budget hosting, what do you expect. Their colo customers aren’t affected by this outage.
@Mike (#135) It would help if you actually read the note and understood DNS yourself. DH has (at least) three sets of DNS servers (primary, secondary, tertiary). Primary and Tertiary are being moved now, secondary is handling all the DNS duties. This is why your host will still resolve to an IP address, and again, non-shared-hosting customers’ sites still work.
March 21st, 2008 at 11:40 pm
OK… the numbers are messed up on that now… some comments have gone missing.
March 21st, 2008 at 11:41 pm
Dreamhost! I was performing cardiac surgery on the Vice President using my webpage and now it’s down and he’s dead. Thanks for not warning us at all! I’m starting an anti-dreamhost blog because that will certainly show you!
March 21st, 2008 at 11:42 pm
Well, it’s time to sit back and watch, “I Am Legend”. Overall, this is one of the few times that I’ve experienced any issues with DreamHost, so keep up the good work!!! I’ll most likely be answering complaints in the morning from my website members, but that’s part of running a business.
March 21st, 2008 at 11:44 pm
Shawn, you obvously need more business if you have the time to constantly check RSS feeds. What is easier, a push email or having to check RSS? Email is a constant for most, if not all of us. The volume of complaints in this forum suggest that you and your attitude are by far in the minority.
March 21st, 2008 at 11:44 pm
Anyone here seen “Shutter” ?
March 21st, 2008 at 11:45 pm
Guys anyone on Bazooka? My sites are still down, can somone on Bazooka reply?
March 21st, 2008 at 11:45 pm
@TTR the vast majority of the web servers are still down. Probably be several more hours.
March 21st, 2008 at 11:46 pm
Right on the money — and just to expand on that, read the TOS at ANY host. It’s not just a Dreamhost thing.
March 21st, 2008 at 11:48 pm
Thanks Keith for the update
March 21st, 2008 at 11:48 pm
@Shawn True. Again, this is budget service. If you’re expecting 99.9% uptime, it’s unlikely to happen over the course of a year.
March 21st, 2008 at 11:52 pm
I agree with many of the “posters” here. An e-mail would have been a nice courtesy. If I can receive an e-mail from Dreamhost regarding up coming “bill dues” - the least I can expect is an e-mail regarding up coming “server shut-downs”.
The only thing I really want to know is - when is the server coming back on? I have committments that I am required to uphold. I will be unable to follow through with these committments unless the server comes back on. Maybe this does not effect Dreamhost…but it effects MY BUSINESS!
If you cannot protect my business then how can you expect me to protect your business? I am paying you MONEY for a service that supplies my business. Tonight my business is at a stand still because of Dreamhost.
I am VERY displeased with your service right now.
Sincerely,
Jeri-Lyn McCrea
March 21st, 2008 at 11:53 pm
Today is absolutely a big day!!! In Taiwan, we have the 12th president election today. In US, my blog is moving to a new home now. Hope the cluster moving can be completed as soon as possible. I just voted for my ideal candidate this morning. And I’ll update the latest news and election result on my blog if the services is back.
Keep going and good luck!!!
March 21st, 2008 at 11:57 pm
@Jeri-Lyn: The original update (linked above) did say that it would be up to 12 hours.
March 21st, 2008 at 11:59 pm
FYI Dreamhost, if anyone is listening…it’s not just spunky that’s down, MYSQL is down on sites that are on different clusters.
As for the email requests - yes, that would be smart business. It’s really ridiculous to expect the average web user to even understand RSS much less have to subscribe to it to get a critical update. Just because someone has a website doesn’t mean they even known what RSS is. And no, that’s NOT the fault of the customer.
DH needs to respond to the needs of its customers, not remain in a techie fantasy utopia where people should use RSS. Most people DON’T use RSS. They use email. Send critical downtime alerts IN EMAIL.
It’s not like DH sends out a big warning email when you sign up that you MUST subscribe to RSS to know what’s going on. I found my way here by accident. I can guarantee you that my friends who use DH have no idea what’s going on right now.
March 22nd, 2008 at 12:04 am
Yep. Which isn’t even a really bad thing. If your site isn’t worth more than $5 - $10/month, the world probably won’t notice when it’s down.
The funny thing is that you also don’t always get that at the more expensive hosts that “guarantee” it either. LOTS of ways around it. Uptime guarantees are a bigger gimmick than overselling.
Any host can guarantee 100% uptime, even Dreamhost if they wanted to stoop to that level. All you need is a page of disclaimers that redefine uptime. Then, you make them jump through hoops to get reimbursed. One of those hoops is often making the customer prove the downtime with logs, which pretty much screws anyone that can’t even figure out RSS.
Try finding a host that logs all your downtime for you, requires nothing on your part, then automatically sends out a refund check. They’re easy to find… just look for the ones that go out of business during their first big downtime, because they actually offered a true guarantee instead of a marketing gimmick.
I’ve always been glad Dreamhost never bought into that crap. I believe they’ve even publicly addressed it before, either on the blog, forum, newsletter, wiki… in other words, I forget where.
On the other hand, it’s not hard or expensive for the customer (NOT the host) to achieve pretty close to 100% uptime by themselves, with failover, 2 cheap shared accounts and preferably offsite mail hosting. Much easier than expecting a host to build twice as many DCs, double the server count and do it all.
March 22nd, 2008 at 12:07 am
Wouldn’t someone that can’t do something as simple as subscribing to an RSS feed be an idiot for choosing a host that uses one?
Again, this is like complaining about no phone support, then choosing a host that doesn’t offer it.
This is not new at all. This is how it’s been for a very long time. And it only affects a tiny handful of people that can’t follow directions, so it’s far from a catastrophe or breakdown in communication.
March 22nd, 2008 at 12:08 am
This really, really sucks. No email to inform about this, and I’m on GMT +8 and I need to work, guys. Thank goodness for Gmail.
March 22nd, 2008 at 12:11 am
@shawn
But just because it isn’t new doesn’t mean they shouldn’t email the affected customers. Policies can change.
I _did_ read the RSS feed. Looked for my server in the extensive list they put in the email, and guess what? It isn’t there. If they can email for things like billing, they can email for long pre-planned outages like this. I’m not suggesting they bog down mail servers for every little outage, but a DC move is pretty significant and they had the time and resources to send something out for this one.
March 22nd, 2008 at 12:14 am
gosh darn it, you guys are killing me. i don’t voice out that much… believe me if i opened a support ticket every time i notice the server was down, you’d be busy.
what’s crazy is that for every time i notice the server is down… IMAGINE just how many times i didn’t notice.
March 22nd, 2008 at 12:16 am
Yeah, I have 10,000 pissed off people mailing nonstop about their missing information. >:(
March 22nd, 2008 at 12:18 am
Nice work guys. I only check this blog when shit fucks up, but surprise surprise, this would be the first time I’ve heard about a 12 hour outage. Posting to a blog is not a way to inform people of a 12 hour outage.
While DH may host 600,000 domains, there may only be 200,000 accounts behind those domains. 200,000 e-mails, mostly to their own internal servers is no big deal.
This is why I use google to host all my e-mail.
March 22nd, 2008 at 12:19 am
Does this outage include hypnos? I’m on hypnos and my sites down too,
March 22nd, 2008 at 12:21 am
If I may…
@KumaZatheef If you log into the panel (which is back up now) and click on the ‘Account Status’ link in the top-right corner it will pop up a list of status information. The link to the RSS is at the bottom of that list.
For feed readers, check out Google reader (reader.google.com); or FeedGator (feedgator.com) for options.
For info on RSS itself check out wikipedia (en.wikipedia.org/wiki/RSS)
March 22nd, 2008 at 12:22 am
@John (and anyone else): If your stuff is down, chances are it’s part of the outage.
March 22nd, 2008 at 12:24 am
The policy DID change… to what it is now. You accept that by being a customer. That’s how it works.
You may have noticed that you can’t order lobster & steak at Wendy’s. If you want to eat there, you are going to accept their menu. It’s that simple.
If your server isn’t part of the move, then any problems you’re having aren’t related to this, which means you wouldn’t have gotten the email anyway.
This is only a big deal to those that are affected. Notice that the list was only a tiny fraction of their 1500+ servers.
March 22nd, 2008 at 12:25 am
It’s worth repeating: a planned service outage of more than a few minutes needs a notification email. Moreover, such urgen notifications need to be sent a couple of times. A 12-hour outage is more than an inconvenience, it is a genuine problem.
I was badly burned by the billing debacle. This experience is leaving a very bad taste in my mouth.
March 22nd, 2008 at 12:26 am
The only comfort I receive from this downtime is reading other’s comments about how crappy this situation is for everybody.
Right in the middle of coding…
—–
Broadcast message from root (Fri Mar 21 20:58:26 2008):
The system is going down for system halt NOW!
—–
Gee… thanks for the heads up!
March 22nd, 2008 at 12:28 am
It wouldn’t matter if it was only 5 accounts, it’s still their policy.
And it shouldn’t be going mostly to their own servers because people should be using email addresses that aren’t hosted here for that.
March 22nd, 2008 at 12:30 am
And will you guys please specify the time in either UTC or something like EST ? What is PDT? Ok. I know it, but I’d rather prefer you guys use UTC. For all us people who are not in USA.
March 22nd, 2008 at 12:30 am
plz make our website online ASAP, it is down for so long time!
March 22nd, 2008 at 12:31 am
So, you get comfort by spending time here reading other posts complaining about downtime, but can’t be bothered to stay informed and realize that you had a weeks notice?
March 22nd, 2008 at 12:32 am
Shawn is obviously not on Oprah’s book club reading Eckart Tolle’s A New Earth because if he was, he’d know that his ego was wayyyyy out of control here.