Spacey Cluster Move! (Reminder) UPDATE
Posted (November 30th, 2007 at 9:47 am PST) by micahThis is just a reminder that the Spacey move announced a couple weeks ago will take place in approximately 12 hours from now.
– UPDATE — Sat Dec 1 06:12:59 PST 2007
Most servers are back up and running normally. A few webservers are still having issues due to a petulant file server which we’re currently rebooting (hopefully for the last time) so we’re not giving the all-clear just yet. Most services were restored within the 8 hour mark but we wanted to have everything 100% up before we resolved this post. Unfortunately it’s taking a bit longer to get these last stragglers online. Please bear with us, we’re very, very close.
– UPDATE — Sat Dec 1 06:37:54 PST 2007
All services are back online.
If you have any questions regarding the server move, please contact our support team.
143 Responses to “Spacey Cluster Move! (Reminder) UPDATE”
Good Luck!
Oh, and please take pictures and don’t drop the filers! 
wrong thread.
your sites are not down because of the spacey move. the spacey move wont happen for a bunch more hours. you are experiencing some other difficulty.
try dreamhost statusing your server, otherwise its probably just a hiccup. another annoying dreamhost hiccup.
that last post sounded really bitchy on my part. I actually have been very lucky, my sites & servers have only had like 2 outages and one was planned. this will make the third in over a year now…
They are up again. Probably coincidense it was the same time this post was placed, of which I thought it was initiated now, but its in 12 hours. Anyway… the support team acted very fast, so no problemo anymore. Sorry for posting wrong section, thought it was related (you never know).
Good luck.
S.
Just don’t lose my emails and files dreamhost. Rock on.
Best of Luck! Wish the move could get over soon!
DG…
Down time of about 8 hours would hurt some business. good luck with the move.
Yeah! Take pictures please!!!
My main concern is email, I can live with my web site being offline as it’s just a personal site. I hope sending mail servers all queue mail for delivery when Spacey comes back up.
There was no mention of whether or not emails will bounce or will simply be delayed in a queue. Anyone know?
I second that comment, I hope the do not lose and files or email data stored on the servers.
@Steve Davis: Word is that the mail delivery depends on the others hosts mail server. They’ll either return and error message to the mail sender or queue the mail for later submission (as far as I know this is what should happen). It’s mentioned in the comments below the original post concerning the moves.
yikes! that is horrible news for the mail
if this really does go on for more then an hour or 2 this will really cause some bad headaches and bad impressions on our own clients
could not a mirror or something better than nothing have been setup to handle this while they moved?
my web server is bourbon and email spunky … will i be down?
Broadcast message from root (pts/3) (Fri Nov 30 21:49:33 2007):
The system is going down for system halt NOW!
71.192.253.11 - - [30/Nov/2007:21:49:32 -0800]
Here we go!
8 hours, gosh! fingers crossed!
not even a little message for site visitors or anything? returned emails? ugh.
hope everything goes well.
I am very sad
is this done? because 208.97.167.154 is down
208.97.167.154 is currently down. is this because of the spacey move? if yes, when are you going to have our sites up?
Jolt is down.
I just wanted to say thanks for the emailed heads-up two weeks ago. Because of that, I was able to give notice, and I really appreciate your efforts
REDBULL is down !
Millionaires can’t read.
Just want to make a post here for a time stamp so I can judge how near we are to the 8 hours which you can bet will take 16. You always seem to get an unforseen problem message why it takes longer….
Good Luck!
You guys just keep up your great service.
let’s see…should we trust them with something as complex as a move when they can’t seem to get the correct time posted? HEY DAYLIGHT SAVINGS TIME ENDED A FEW WEEKS AGO…DUHOOOH!
Best of luck with the move. Not really had many problems with Dreamhost and the two week heads-up gave me a chance to sort out anything I might need to sort. Besides, it’s the weekend - not being able to check my email for a few hours might actually *help* me to relax!!
I agree with happy, some pictures would be fun.
Eight hours of downtime for a hundred servers is “Medium” severity? What’s “High”, “Planet has just been blown up by Vogons, total data loss”?
good luck
wow, such bad timing for my business. and i had no idea til i got customer complaints just now. i hope it goes smooth so it really does get back up in 8 hours.
I hoped it wouldn’t take so long time but my hopings are falling down … I think it’ll take more than 8 hours and unfortunately in the weekend, the only time I have to work on my websites and now I cannot.
please back it up as soon as possible
——————————
5 Hours left to estimated time needed
People cheer up. It all for the better.
They know what they are doing.
They posted that this would be down a LONG time ago.
If your customers are complaning perhaps you should have warned them 
I am able to create new databases and users, but I cannot access them. I can access the old ones, but not the new ones. Does this have anything to do with the current issue?
@Xora K Joken
Something would have changed ? I don’t think.
I’m ok they do what they need to do … BUT perhaps they could’ve done it piece-by-piece, not an enormous server move, which (the first) I think would’ve needed very less time.
any update on the outtage??? Is it working?? have the tech team gone out for coffee???
Inquiring minds need to know!
Status??
ETA??
Any count? How many websites are down in total?
Either way, there doing it.
Neglecting to tell you customers is nothing to complain about.
5h paased!
3h to go!
GOOOOO DH.
its til 2pm uk time
over 16 hours now and counting….. all my sites are down 
not good.
all my sites are down. F*ck service
I am watching this and have to get a chuckle out of the stress that seems to be hanging on this server move. I applaud the DH team on their ability to stay on top of things and after 4 years with this service I have seen hiccups coe and go. Any business that has grown as quickly as DH has is bound to have hiccups from time to time.
WE WERE ALL WARNED WEEKS AGO ABOUT THE MOVE and if it caught you by surprise then you haven’t been reading your mail.
If your customers are upset, it is YOUR OWN FAULT for not warning them ahead of time and you should be ashamed.
It is 550AM EST and they went down at approximately 1230AM EST so we have some time to kill before things are finished.
Good luck with the move guys and if you do this move as efficiently as you handle the other hiccups that have come your way over the last 4 years then this will be a breeze.
whats goin on here ?
i didnt understand the problem
whats “spacey cluster” ?
why are all the servers down , actually when they will be active again ?
@mmt - that was a joke, right?
So… it will be up in 1 hour from now?
Ok, 2 hours I guess…
I appreciate the warning, as everyone else did, but i am also confused as to why there was nothing temporary put in place.
I am hoping that all possible steps were taken, al avenues investigated, and that this was the only way it could have been done…
I’m on the Coke server - looking forward to being able to log into my Wordpress installation again ^-^
Hoping this is done soon.. I guess the bad side of having hosting over there is that it can be down during busy times over here *cry*
All my subdomains have been removed. Is that due to move and will they be back afterwards?
Ben’s comment, “Eight hours of downtime for a hundred servers is ‘Medium’ severity? What’s ‘High,’ ‘Planet has just been blown up by Vogons, total data loss?’” made me laugh. But seriously, who chose a weekend evening (and Sat. afternoon, for Europe) to do this? Hope it irons out the other downtime/load (300!?) hiccups that have been cropping up more often since Feb. I can deal with a “night off” if it means no mo’ o’ that.
DE NUEVO 80% UPTIME
DREAMHOST IS NOT A SERIOUSLY HOSTING
IS BLOG HOSTING
IS NOT FOR BUSSINESS WEB
DREAMHOST BLOCK SPIDERS
DREAMHOST DOWN MANY TIMES
3times /week aprox
no no no is no good but what i need to do, i dont have much money and then i`m user of dreamhot
thanks dreamhost.
I’ve been trying to get this working for a long time. have some important update to make to the website. is there an ETA for the cluster to come back up?
liaaaarsssssssssssssss
forget websites… but mails… please…
Mail can’t get delivered to DH mail systems becaus the MX records of the domains are not responding correctly, so mail will be queued in the source mail servers to deliver it later.
You won’t lost any mail, but mail will be delivered a bit later… let it a few hours after all mail systems are up.
Don’t worry! Be happy! Just a 8 hours of planned downtime. Not a problem, at all 
Till Down all my sites
It is the first time i have to wait 8 hours for a move
how long will be the down time?
I make it one hour to go?
I FOUND THE REASON FOR THIS LOOONG DELAY!
JUST LOOK AT THE PICTURE
http://dreamhost.com/
WAKE UP F*** LOOSERS!
50 mins and counting…
[[[I FOUND THE REASON FOR THIS LOOONG DELAY!
JUST LOOK AT THE PICTURE
http://dreamhost.com/
WAKE UP F*** LOOSERS!]]]
HAHAHAHAHAHA
All my sites are up again. Juhu! Good work, DH! Thanks!
have ftp to hansens but no http
getting scared now. no sites and it is morning
waiting of the sides too.
but dont worry i hope it will be
Well its past the 8hr mark … and we’re still down ……. hope there they didnt run into big problems!
opps … different time zone! well its close
9am would be the 8 hr mark
They’ll do it
I have ftp to hansens and gingerale
http should be right around the corner
9am est
how much more time we can see the sites up,its been like 7 hours now !
My websites are working now :]
my sites too !
my sites on PEPSI are down.
hansens and gingerale up W00T:)
Knoppe: Your site *is* up!
LIPTON OK
i bet shasta is gona be the last one up xD
Schweppes up. 9am EST
woah minutemaid is back
Fatal error: Allowed memory size of 838808 bytes exhausted (tried to allocate 4864 bytes) in /home/.mendlesohn/username/mywebsite.com/includes/JobQueue.php on line 220
My websites (fresca server) are back up. Good work, DH!
For the last few months, after some problems during the spring/summer, I’m more and more satisfied with DreamHost!
oj is down
my site on tang is still down. …
Thanks DH! Redbull is back online :D.
One of my sites was up now its down.
username@nehi:/$ cd ~
-bash: cd: /home/username: No such file or directory
well thats nice….
where did the files go?
My site on rootbeer still isn’t up… my luck it’ll be last
the first time I went to make changes to my site in a while, and it was down, worried the sh*t out of me haha!
Although I have got an FTP connection
Tonic appears to be up and running again.
(grrr said: One of my sites was up now its down)
Same, and the ftp works but not the http….
retification… it works YAY! Thank you DH
Site on rootbeer is up and running
good work guys!
calpico/spacey is up! all sites working…
Tang’s up, but email is still wonky…
Sobe working
Mail is OK, but webmail don’t work: «ERROR - Unknown user or password incorrect.»
Seeing all the whining and bitching on here makes me glad I don’t have to deal with public customers anymore. The guys at dreamhost gave ample warning and are doing their best to get everything moved in a timely manner. For those of you who think 8 hours or more to do what they are having to do is too much, you have obviously NEVER done anything like this. There are ALWAYS going to be little problems and glitches when doing something like this. Kudos to the folks at dreamhost for their continued hard work and excellent service.
All sites are up and running. I forgot about the move, but wasn’t a problem. I was up early this morning to track the progress, and all was up as was expected. I appreciate all your hard work, and congrats on the move. Thanks for the good service and keep it up.
still down
Whoops, looks like the mail server’s IP address changed. Quick change in ZoneEdit and I’m in business! 
Now to see what happened to my email from last night…
Great! Thanks for expediating soooooo fast!
That was fast work for such a big job, well done guys
How great is this? I was asleep the whole time.
My site is up and operational! Thanks guys! I also appreciate the advanced notice. Kudos!
For those of you who cannot see your site yet, try rebooting your machine and then re-attempt connection. It can be as simple as changed DNS.
Happy Holidays!
Holy Cow!!! 8 hours and 37 minutes to move the whole cluster? That is SMOKIN! I used to be a net admin myself and I really didn’t think you could do it. Hat’s off to you guys!
bloody hell.. one of my FTP account didnt migrate across properly.. its no longer linked to my general account.. i cant login to it no more! ARGHHHHHH
Operation restored at 2007-12-01 14:47:32.
Url was down as a result of:
Http error:Http_client.Bad_message(”Unknown reason (e.g. unexpected eof, timeout)”)
First error was detected at 2007-12-01 07:39:47
Last error was detected at 2007-12-01 14:42:31
Downtime total 7 hour(s) 2 min(s) 44 sec(s).
Operation restored at 2007-12-01 15:24:28.
Url was down as a result of:
Http error:Http_client.Bad_message(”Unknown reason (e.g. unexpected eof, timeout)”)
First error was detected at 2007-12-01 07:32:36
Last error was detected at 2007-12-01 15:14:18
Downtime total 7 hour(s) 41 min(s) 42 sec(s).
cant say that it was too bad!
They did it under the 8h limit!
Great job, DH! Sites are up and running, the one e-mail I got from one of my message board users during the outage got to me, looking good!
For those saying it’s peoples’ own fault they are upset because they didn’t check their email, I just want to point out that I too was agitated with the site owner (I’m the developer) for not warning me as I have enough email of my own to check regularly without doing the same for all her addresses, until I decided to have a peek myself and found that it never arrived. I have no idea whether there was some malfunction that caused the address to be skipped on DreamHost’s end or a hiccup with the mail server that it got sent to it that coincided with when they sent it, but these things do happen, and one would assume there would also be new customers and people who switched servers due to various issues in the time after that email was sent anyway, so don’t forget that it’s quite possible the people you’re insulting simply didn’t receive the warning even if they did check their email.
As for the DreamHost team, congratulations on a job well done, I realize this was no minor task. I do wish you would have sent out additional emails at the one week and one day mark, as being down for eight hours on the weekend during the holiday season is far too severe to count on just a single email not getting lost in transit; perhaps for the long run you may want to consider developing a routine maintenance warning system that will place an alert right in the control panel of affected customers.
You stand to your promise. Thank you for the 2 weeks notice
btw my sites hosted at POSEIDON still on and off…. what tha sky is happening here??? is weekend my god… my sites experience a lot of traffic these days… well done guys
My email still isn’t working…
Same here. I’m on spacey
On tea everything seems fine! Thanks DH!
GREAT JOB AND KUDOS to the DH team for the quick move.
All the whining and complaining did nothing cause internal stress for those involved.
This was a daunting task that seems to have been handled very well, I really appreciate that you chose to do this late at night on a weekend instead of during the business day mid week.
Thanks for the good work guys
All working.
Many thanks for keep your promise, 8 hours (a little more) and now my site it’s up and running.
Good work, best regards.
web sites working fine, webmail is working, but pop3 mail is *not* working for me yet.
“Receiving reported 0×800CCC0F error” is what i get from outlook. no response from support for the past six hours since i first reported it.
BTW. - nearly perfect, just 4 minutes and 12 seconds over the estimate.
First error was detected at 2007-12-01 08:07:28
Last error was detected at 2007-12-01 16:11:40
Downtime total 8 hour(s) 4 min(s) 12 sec(s).
So, is anyone else getting SSL errors on IMAP after the move?
pop3 mail is *STILL NOT* working for me.
@xphia: what do you mean “didn’t migrate over”?? This wasn’t data being copied…all of the machines were physically picked up and moved in a big truck to a new location….where do you think it went? Is there an FTP account hanging in space there in the middle of an empty datacenter? If you’re having issues, contact support.
Very impressed. Well done guys!
i’ve contacted support several times, *STILL NO POP3* mail. a little frustrated, since the only response i’ve gotten in the past 24 hours to any tickets is an automated response saying that they’d moved spacey. no actual response saying anyone had looked into the issue. 24 hours is a long time for no response to a ticket. kinda crappy.
thank you
Looks like some daemon didn’t restart? I’m still getting SSL errors.
Looks good on my end. Thanks for keeping us in the loop.
so i’m now at 29 hours since the servers were considered “up” and 28 hours since i first posted a ticket on the issue (no pop3) and only one single automated response. i know it’s a weekend, but shouldn’t 24 hours be a reasonable minimum for someone to even get me a legit response acknowledging the problem?
come on, guys. i wasn’t one of the people bitching about the move. i don’t complain here, i know i’m getting inexpensive hosting. but i’d like to know that if i have a problem, and the only avenue for addressing it is the web panel, that someone is at least monitoring the web panel and would at least acknowledge that you’re aware of the problem. i’m not asking for instant response and immediate action, but 28 hours is too long to wait.
I’m getting an unfathomable 17.2670 sec/795.38 server load.
My problem seemed to stem from the fact that I was using “mail.dreamhost.com” as the server rather than the more specific “spacey.mail.dreamhost.com”. It may be, especially if the IP address changed, that the new spacey address isn’t in the mail.dreamhost.com table. In any event, being specific about the server resolved my problem.
@Pat:
You should never use mail.dreamhost.com, spacey.mail.dreamhost.com, or any other derivative of dreamhost.com for your mail. Always use mail.yourdomain.com for incoming/outgoing.
My $400 per month dedicated server runs like crap since the move and I get no response from tech support…I can’t imagine how you $10 per month guys are being treated Everything was FANTASTIC before the move…Can I pay another couple of hundred per month to move me back to where I was ??
Somthing…any sort of options…anyone…anyone…bueller??
Did the ports maybe change? Maybe another parameter? None of my users can access their email, since last Friday!
thank you
THANKS
Mmm… My sites services are on Buck, Spunky and Moorpark and though none of them are in the list, all sites are down. Already posted a check through the panel… but no results yet. Can this be resolved!?!