Spunky and Central Move
We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.
Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.
For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.
UPDATE — Sat Mar 22 02:45:26 PDT 2008 –
Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.
UPDATE — Sat Mar 22 03:40:27 PDT 2008 –
Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.
UPDATE — Sat Mar 22 06:01:28 PDT 2008 –
Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!
UPDATE — Sat Mar 22 07:04:35 PDT 2008 –
We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.
UPDATE — Sat Mar 22 09:55:54 PDT 2008 –
The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.
.
March 21st, 2008 at 7:58 pm
Ahhh man… and I was trying to do a blog post…. oh well, time for beer
March 21st, 2008 at 7:59 pm
What? Again??
Was there some sort of heads-up given for this that I missed?
March 21st, 2008 at 8:00 pm
I wold LOVE to know where dreamhoststatus.com is hosted.
March 21st, 2008 at 8:01 pm
damn, guess I’ll stop working tonight then
March 21st, 2008 at 8:03 pm
12 hours begin now…
Flood this with hate comments!!
It gonna be a long time for sure I tell you.
But I only hope that things improve for better after all this pain.
March 21st, 2008 at 8:03 pm
I agree, it’s time to stop working and go to bed.
Does Dreamhost have folks working 24/7?
J
March 21st, 2008 at 8:04 pm
Best of luck to everyone working on this move tonight!!!!
March 21st, 2008 at 8:04 pm
I was wondering why a cb_login module was taking so long to get installed. Now i know why. I feel stupid, i shoulded knew that if i was doing something important the server was going to go down :).
March 21st, 2008 at 8:05 pm
Why the hell don’t you give us some notice before you just shutdown the server?
I was in the middle of a commit, and then needed to do an svn up on my home system - I’m working here.
[snicker]$
Broadcast message from root (Fri Mar 21 20:58:25 2008):
The system is going down for system halt NOW!
svn commit -m “favicon”
Sending .htaccess
Sending data.txt
Adding (bin) favicon.ico
Transmitting file data .Connection to *****.com closed by remote host.
Connection to ***.com closed.
this is the second time you guys have shut the server down without warning while I was working, don’t you have any respect for your customers?
All I needed is to know 10 minutes before hand that it was imminent, and I’d have been able to continue working.
Pissed as hell
March 21st, 2008 at 8:06 pm
and just to point out to some of the posters - its 3pm here.DH has customers that are _not_ from that tiny country USA.
March 21st, 2008 at 8:06 pm
bob cobb– same here….
March 21st, 2008 at 8:07 pm
We should all play Super Smash Bros. Brawl while our sites are down. Friend codes, anyone?
March 21st, 2008 at 8:08 pm
Man, sure picked the wrong (or right) time to backup stuff I had on sake.
on to 12 hours of waiting
March 21st, 2008 at 8:08 pm
There was a post here a week ago warning everyone about this downtime. Everyone doesnt read this page, but atleast they did that. I forgot to tell my users about it though
March 21st, 2008 at 8:11 pm
@”Ahhhhhhhh Says:
March 21st, 2008 at 8:00 pm
I wold LOVE to know where dreamhoststatus.com is hosted.”
It appears that dreamhoststatus.com is hosted at http://www.servepath.com/
March 21st, 2008 at 8:11 pm
Are you guys drunk? This is the 8th time i’ve had some issues this company!
My grandmother has her cooking site on here and she is so pissed right now. I’m literally not allowed to go out anymore!! She’s grounded me! For christ sakes please can you get the servers up and running right now.
March 21st, 2008 at 8:11 pm
They should email everyone a week in advance with an expected downtime of this duration. They should also start the downtime when they say they will instead of 10 minutes early. Are we supposed to check this blog out each day to see if maybe they’re going to take things down for 12 hours someday? 1000’s of web admins are all getting paged right now that their site is down. At first, many will ignore it, because they’re used to short downtimes on occasion. LOL Then they will figure out that this isn’t the normal outage. Dreamhosties, please shoot us all an email next time you plan something extreme like this okay?
March 21st, 2008 at 8:13 pm
Indeed, an e-mail for planned outages like this should be sent out as a courtesy to your customers.
March 21st, 2008 at 8:15 pm
oh god I thought my site was down because I didnt pay my bill yet :gonk:
March 21st, 2008 at 8:15 pm
Aff my site off
March 21st, 2008 at 8:15 pm
Aww man, just my luck…half way through upgrading my wordpress. I hope this is resolved quickly as it’s past 4am in the UK and I am ready for bed, but can’t leave things as they are now…eep.
March 21st, 2008 at 8:16 pm
Pardon a stupid question, but why can’t this be done by mirroring an old server onto a new server at the new data center and then doing a network switch? If you don’t want to commit to that many new machines, then do it a few at a time. Switch 10 to new machines, move the 10 old machines to the new data center, do another switch, continue until done.
I find it difficult to believe that considering how much you use network storage that you couldn’t have done this move with only minimal downtime (e.g., seconds).
I understand you need to upgrade your infrastructure on occasion, but how about doing a bit more thinking about how to do these things without impacting customers? Much less impacting them for HOURS of downtime! I really like you guys, but most hosting companies would be embarrassed by this much continuous downtime, planned or not.
March 21st, 2008 at 8:16 pm
Hum, I agree, they should notify everyone a few night in advance…
That whta email is for. seriously…
Need some thing better then this stupid blog once it down…
Come on how much can a mass emailer cost?…
anyway guess i got to bed…
Bob
March 21st, 2008 at 8:18 pm
Good luck, hope you guys get it done soon, I am working on some scripts for my game. So hopefully it will be back and running by the time I have to upload them.
March 21st, 2008 at 8:18 pm
Ah, just read it is set to last 12+ hours…ouch, that is a crazy amount of downtime
but at least it means I can head to bed now.
March 21st, 2008 at 8:18 pm
Aff my site is off.
Hosting bad, very bad
March 21st, 2008 at 8:19 pm
Yes, emails should be posted to the address of each user account on relevant machines!
I hate this status site. I hate having an RSS feed. I want email alerts for downtime warnings.
March 21st, 2008 at 8:20 pm
If anyone was interested about being notified of such things, they could have subscribed to the RSS feed … over 600,000 domain hosted does *NOT* make email a good way to do this …. srsly!
March 21st, 2008 at 8:21 pm
I have no other choice, I have to change the dreamhost
March 21st, 2008 at 8:22 pm
An alert on the control panel or something would have been nice.. 12 hours on a friday night/saturday? Are you shitting me?
March 21st, 2008 at 8:22 pm
Nice service, Dreamhost. Of course I chose to try to make a blog post tonight of all nights. It was working fine one minute, then everything’s broken, even panel.dreamhost.com is down. Makes you guys look real unprofessional. I just got my renewal bill yesterday — looks like I’ll be doing some shopping in the next few weeks…
March 21st, 2008 at 8:23 pm
Oh and I absolutely LOVE how people can still sign up for service but no one can contact Dreamhost..
March 21st, 2008 at 8:23 pm
And another thing… this point is moot since I didn’t know this was going to happen anyway, but I’m on a PS, and my hostname was NOT listed in your list of hosts going down. I assume one of the names was probably the “real” hostname, but that doesn’t do me any good since I don’t know what that is. If it had been important for me to know that it was going down, I’d be screwed. So if you’re going to give us a list, make sure it’s an accurate list including the virtual servers.
(Does anybody from Dreamhost actually read these rant sessions when things go down?)
March 21st, 2008 at 8:23 pm
The Hosting this very bad
March 21st, 2008 at 8:24 pm
Lol thank god that’s what it is. We’re having some MMO DRAMALLAMAS on my forums and I thought the jackass who started it tried something.
But if it’s just downtime, that makes me feel better. :3
March 21st, 2008 at 8:25 pm
Furthermore.. couldn’t they at LEAST have redirected all traffic for those servers to a generic page saying what is going on? All of our users have NO idea why we are down and it is freaking annoying.
March 21st, 2008 at 8:27 pm
@Tim (#34) - “(Does anybody from Dreamhost actually read these rant sessions when things go down?)”
Only for comic relief! The *REAL* action is at http://fuckdreamhost.com, which DreamHost staff visits regularly!
March 21st, 2008 at 8:28 pm
Hmm that domain isn’t registered.. yet.
March 21st, 2008 at 8:28 pm
I agree with others who have stated that Dreamhost NEEDED to proactively notify customers of this. I was working on finishing up a client site tonight and now I can’t, even though I promised my client delivery of the new changes by tomorrow.
If I had known of this migration, I would have planned around it. An email would have been nice.
March 21st, 2008 at 8:30 pm
The list of servers affected in this move is not accurate. My sites are not on the list, but are all down right now. I can live with the down time, but please, please make the list accurate.
March 21st, 2008 at 8:31 pm
I wish they would have given the heads up >.
March 21st, 2008 at 8:31 pm
Nice warning!! Man, DH is pissing me off. I think I’m switching to http://www.hostmysite.com/. Time to get out of DH. Been using DH for years, but this is getting too bad. Thanks DH for your years of service, but good things have to come to an end sometime.
March 21st, 2008 at 8:33 pm
And, of course, since the panel is down I can’t submit a trouble ticket… I’m not getting any response from spacey at the moment. Sigh.
March 21st, 2008 at 8:35 pm
A) There was a warning put out A WEEK AGO, people. A WEEK AGO. WAKE UP.
B) THEY TOLD YOU ALL A WEEK AGO.
C) AN ENTIRE DAMN WEEK.
Also, my site is down, whaaa!
March 21st, 2008 at 8:37 pm
I think of Dreamhost as a pretty professional and polished web hosting provider. I really like the fact that you guys realize what the internet is all about, communication. Your blogs and news letters are awesome, and engender a lot of forgiveness.
However, you guys have made a few mistakes recently. Such as the credit card charge debacle, and now this. You made a huge mistake not thinking of sending out a warning email. You managed to blog about it back on the 14th, however, not everyone bothers following the blog. Emails were absolutely necessary in this situation. Please get your act together! If your blog post has a ’severity’ of ‘high’, then why not automatically email everyone regarding that?
Thanks for listening.
March 21st, 2008 at 8:40 pm
Nobody with a brain would ever recommend hostmysite.com except for spammers that work there.
March 21st, 2008 at 8:42 pm
Oh the agony of a misplaced comma - real satisfaction during such times can be found at:
http://fuckdreamhost.com
March 21st, 2008 at 8:43 pm
Waaaaaaaaaaaaa!
I’m sure the internet will survive 12 hours with one less meaningless blog that only exists for the sake of saying “I blog.”
And it doesn’t make them look unprofessional since they gave you a week of notice. You could have bored the world with your pointless blogging before now. That way, your readers would all still be asleep and wouldn’t even see this downtime.
March 21st, 2008 at 8:44 pm
Well I agree about not receiving an email, as I don’t read this blog unless my site is down.
I’ve been really happy with the service, so don’t let things like this slip up cos bitterness spreads fast around the net.
Anyway, 12 hours to go, lucky that I was going to update tomorrow anyway.
March 21st, 2008 at 8:44 pm
Is it up yet? Is it up yet? Is it up yet?