Spunky and Central Move
We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.
Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.
For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/
UPDATE — Sat Mar 22 00:12:05 PDT 2008 –
Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.
UPDATE — Sat Mar 22 02:45:26 PDT 2008 –
Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.
UPDATE — Sat Mar 22 03:40:27 PDT 2008 –
Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.
UPDATE — Sat Mar 22 06:01:28 PDT 2008 –
Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!
UPDATE — Sat Mar 22 07:04:35 PDT 2008 –
We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.
UPDATE — Sat Mar 22 09:55:54 PDT 2008 –
The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.
.
March 22nd, 2008 at 3:15 am
I think the next site I’m going to be working on is a DNSStuff.com replacement. I’ve already created http://whatismyrdns.com but that’s child’s play. Now with DNSStuff.com signing their suicide note, I guess it’s up to someone else to come up with a replacement. God, I can’t believe they’d actually get so stupid as to think that ONLY big companies that use DNSStuff.com every day use the site at all. I only use it once every several months, and that in no way justifies the 50 bucks a year price tag on it, that’s applied to all users - daily users or every-few-months users.
Meanwhile, let’s try figuring out that quoting thing. Obviously, [quote] (bbcode) doesn’t work… maybe HTML (blockquote tag, as found in the page source)?
Aye, I did notice that. And I suppose this discussion “thread” is rather volatile… EVERYONE here is fucking pissed.
@ Shawn:
http://xkcd.com/137/
I have a life, thankyou.
March 22nd, 2008 at 3:17 am
@gammo “Voila!.. Everything is back to normal”
ummmm…. no…. it isn’t. electra is still unresponsive. a server that, according to their entry above, shouldn’t have been affected to begin with.
March 22nd, 2008 at 3:19 am
Ah ha! Another XKCD fan. Sweet.
And no, not everyone is pissed. I’m just checking in here every so often between other things I’ve got to do. It breaks the boredom.
Matt
March 22nd, 2008 at 3:21 am
formosa is still down.
March 22nd, 2008 at 3:22 am
@falcon “I’ve already created http://whatismyrdns.com”
ummm…. that site is down too. is it hosted by dh? heh.
March 22nd, 2008 at 3:26 am
Yeah, whatismyrdns.com, as well as falconfour.com, hostfile.org (my main site in DH), and blockbusterforum.net (for sale!11), are all hosted by DH, and are all down. FAIL.
Actually, they may or may not be down. In lack of a friggin’ DNSStuff DNS lookup tool, can anyone verify that the IP for hostfile.org is 208.113.190.37? Strangely, http://blockbusterforum.net (also hosted on the same server) is working fine for me. Maybe it’s a stale DNS?
March 22nd, 2008 at 3:28 am
That’s the IP I get for hostfile.org.
March 22nd, 2008 at 3:30 am
March 22nd, 2008 at 3:30 am
Falcon : On my local machine here hostfile.org is resolving to 208.113.190.37 ok!!!
Like my site the pings are just going in to a black hole!!!
March 22nd, 2008 at 3:32 am
Nate, all my sites on formosa are running, Webmail not, but i think is in another machine for all webmail (IMAP also down)
March 22nd, 2008 at 3:32 am
looks like holt is on the way back up too… thank fuck for that
March 22nd, 2008 at 3:34 am
Wrong. The US majority & timezone thing is real.
But you are calling a tiny group of people whining about email in blog comments a majority, which it is not. The real majority are the thousands of customers that aren’t here whining about email like it’s some new problem, because they aren’t baffled by simple things like RSS. RSS isn’t going away. Quite the opposite. I’d get used to it.
Not a single smart person has trouble keeping up to date and dealing with this stuff. Read the comments and honestly tell me how many smart people you see complaining here. It’s mostly just kids throwing tantrums under multiple names. Half of the posts are just idiots that aren’t even reading and show up just to announce they didn’t get an email, or morons that think I’m a troll, but think they’re brilliant for flooding this place with garbage under my name, or just making other dumb names, etc…
Dreamhost is probably one of the few companies that can post a blog about a multi-million dollar billing error… and still not look half as stupid as their customers replying in the comments section.
March 22nd, 2008 at 3:34 am
Incredible. Well, if dnsstuff were still around (I know, I keep talking about them, but seriously!), I’d be doing a DNS cache test as well, to see that all the ISP DNS caches were all showing the same thing and maybe get an updated address. No such luck.
Damnit, Dreamhost… quit failing.
March 22nd, 2008 at 3:34 am
Crenshaw is STILL down, no mail, no http no ftp….
Come on DH, lets get’er done.
March 22nd, 2008 at 3:35 am
It’s stale DNS. I’ve got the same problem.
My main site works, as it resolves as: 208.97.139.11
All my subdomains however, resolve as: 208.113.134.71
March 22nd, 2008 at 3:35 am
jabber on grand is still down.. can’t IM. v annoying.
March 22nd, 2008 at 3:36 am
I’d imagine most should be up now… and it’s not even 8 hours into the 12 hour window.
March 22nd, 2008 at 3:36 am
Falcon : I think there is a routing problem as your IP is in the same IP block as my site 208.113.*.*
Doing a local trace route seems to be no route to 208.113.*.*
March 22nd, 2008 at 3:41 am
@ shawn “I’d imagine most should be up now… and it’s not even 8 hours into the 12 hour window.”
ummmm… not the case. in-fact, i’m on electra, which has been down all night, and it’s not even on their list of servers to be affected.
March 22nd, 2008 at 3:41 am
ATLANTIC IS BACK UP !!! WOOHOO!
March 22nd, 2008 at 3:42 am
scratch that…. electra FINALLY seems to be back up….. even though it wasn’t on their list to begin with.
March 22nd, 2008 at 3:44 am
It’s Saturday in South Korea. It’s SEOUL FASHION WEEK, and we run Korea’s first online street fashion magazine, and are doing fairly well here. I and my partners, over the last 2 days) gave our business cards to the editors of Vogue Korea, Elle Korea, as well as similar writers and correspondents from Taiwan and Tokyo. I interviewed Hwang Jae-bok, one of Korea’s most famous fashion designers and also the head of the Korean Fashion Designers Association, as well as several other major figures in Korean fashion.
They’re going to have been going to my site anytime today, or possiibly tonight. HOW F*@#ing professional do I look right now? I don’t even get an internal server error or anything — the site just plain doesn’t exit right now.
AN EMAIL WOULD HAVE BEEN NICE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Then I could have said somethng like, “Please check on Sunday. Our server is undergoing maintenance and will be down tonight.”
But no, I might get 30 seconds of some busy Ms. Really Important Someone’s precious time, and if they even think to go into my site, I get nothing.
THANKS FOR THE PROFESSIONALISM, DREAMHOST! I’m SOOOOOOOOOOOOOOO glad you give us monthly little emails detailing everything you do there, along with the silly “homey” style that’s supposed to be disarming yet engaging, but YOU CAN’T EVEN SEND US A WARNING EMAIL so we can warn clients and contacts?
I won’t curse here, since it’ll just get my comments banned. Let me just say you owe a lot of people at LEAST a year’s free service.
I can understand downtime and emergencies. But not even a mass email, when you obviously have the capability of making one!?!?!?!
This is ghetto-ass. Seriously.
I’m sitting in the press room in Seoul, Korea talking about how great our magazine is a fashion writer from Taiwan and I can’t even get the site up, and am surprised as she thinks I’m incompetent.
THANKS, DREAMHOST, FOR A WONDERFUL, WONDERFUL DAY!
I’ll remember it always.
March 22nd, 2008 at 3:44 am
aaw shite.. it was a cached page
March 22nd, 2008 at 3:44 am
Electra is reachable, and not on the list, so if your site has an unrelated problem, you need to take that up with support.
I also said most — not all.
March 22nd, 2008 at 3:46 am
Michael Hurt: Since you apparently can’t read, an email wouldn’t have done you any good anyway.
March 22nd, 2008 at 3:49 am
All of my dreamhost accts are down. So at least 3 boxs have been down for over 8 hours by my watch.
Also these boxes are not listed in
“abbazaba absinthe achilles adams admiralty aero alameda almalibre alondra altoid alvarado archibald arrow atlantic avalon aviation babyruth badillo balu bandini barranca barry baseline bazooka beaudry bellflower beverly bixel blowpop bonbon bottlecap bounty bourbon boyd broadway brownie bubbleyum bundy burton butler butterfinger cahuenga caramello casamento catalina centinela century cerritos cheeto cheezit chex chiclet cienega cognac colbert compton cookie cordial crackerjack crenshaw crunch cupcake dingdong doheny dorito drumstick dumdum edendale euclid fairfax fc23 federal fig fireballs firestone fletcher flower foothill formosa frito frontseat fruitland fudge fundip garland garth garvey gobstopper goober good gower grand granola grappa gumdrop gummi haven heath heaven highland hillhurst hillmont hoho holt hoover hope hops hyperion jalapa jawbreaker jefferson jerky juniormint jurupa kidsgames kiki kitkat krackel lankershim larchmont laurel lemonhead licorice lifesaver limbo-spunky3 limbo-spunky4 locust look lucas malt manchester mariposa mayo melrose mento milliken mills monroe moorpark mounds munchies nash necco nerds nilla nougat nutrageous oakdale ocean olive olympic ord oreo ouzo overland oxnard paramount payday pernod pez pico planters plenty pocky popcorn poprock powerbar pretzel price pringle rampart razzle redhot redondo reeses rexford ritz rockyroad rodeo rollo roscoe rosecrans rossmore rum runts sake saltine sanborn sawtelle schnapps scotch selma sepulveda shocktart skittle skor skybar slauson smores snicker snocap soto sourpatch spree starburst sugardaddy sunset taffy tampa temple tequila toblerone toffee tootsie topanga torrance truffle twinkie twix twizzler union vanowen vermouth vine virgil warhead washington western whittier whopper wilmington wilshire york zagnut zuma”
I think they are understating the issue.
March 22nd, 2008 at 3:51 am
Eric : Your site is on the IP range 208.113.*.*
Is your site on server - pom?
March 22nd, 2008 at 3:54 am
Darn it! Yesterday was not a good day o join dreamhost, i must have tried to do a one-click wordpress install 3 times: and NOTHING appeared.
March 22nd, 2008 at 3:55 am
Still no go, waiting now for more than 3 hours for my PS MySQL server to reboot…
Panel says:
You have a reboot for psmysqlxxx currently pending! Please wait a couple of minutes..
Minutes are now hours!
Asked for an manual reboot 3 hours ago by email: ticket still pending…
Very frustrating.
One very unhappy costomer.
March 22nd, 2008 at 3:55 am
espresso wasn’t on the list but it’s down!
March 22nd, 2008 at 3:57 am
I said “majority here,” so it still holds in that sense, but point taken that all those here are just a very very tiny fraction of the people affected anyway. I’m not so sure that everyone who didn’t jump in here to bawl was subscribed to the feed; they might have been asleep, or Heaven forbid, even had lives that kept them away from the computer long enough to not notice a few hours of downtime in the middle of a Friday night, but who knows. Either way, I’m amazed at far ahead of schedule they are on such a massive undertaking, it is sad that there will still be people complaining about it taking “so long” anyway, it’s no wonder the blood reaches boiling point so quickly around here.
March 22nd, 2008 at 4:00 am
My site is still not up and neither the hosting server(eggnog) nor the mysql server are affected apparently according to that.
March 22nd, 2008 at 4:00 am
My random blog site that has no purpose what so ever and that is most likely a complete waste of time is BACK UP!! Whoooo!!
Still gotta love DreamHost.
March 22nd, 2008 at 4:00 am
@ Michael Hurt
As another disenchanted DH customer I totally agree with you. Your story echos my own.
DH should have emailed about this one.
March 22nd, 2008 at 4:01 am
@ shawn “Electra is reachable, and not on the list, so if your site has an unrelated problem, you need to take that up with support.”
Like I said in the entry after, “scratch that…. electra FINALLY seems to be back up….. even though it wasn’t on their list to begin with.” I corrected my statement once it was back up.
Furthermore…. it’s slightly ironic that Electra went down when Spunky was taken down. Also ironic that Electra came back up when Spunky servers started coming back up.
March 22nd, 2008 at 4:05 am
None of you know anything about how shared web hosting works.
Your sites will be back up soon. If you don’t like your sites being down in the middle of the night on rare occasions, get a dedicated server somewhere with redundant auto-booting backups and pay a few hundred times what you are now — no one is obligated to keep a Dreamhost account. Trust me, DH is much better than most shared hosting providers; and you’ll never, ever find one that offers 100% guaranteed uptime and lives up to it. This is the life of a shared hosting account holder. Deal with it.
March 22nd, 2008 at 4:06 am
Michael FeetmanSeoul is back up again. It looks quite a cool website.
March 22nd, 2008 at 4:08 am
@vinic “If you don’t like your sites being down in the middle of the night on rare occasions”
13:08 here
March 22nd, 2008 at 4:13 am
I’m hosted on tizer and have’nt been able to reach my site since this started. It’s 12pm here in London.
Can someone in NA let me know if it’s reachable.
Thanks
March 22nd, 2008 at 4:13 am
My sites on jawbreaker are still down, as is my email. I got NO email alerting me to this half-day outage. NOT a happy camper.
March 22nd, 2008 at 4:14 am
“Most” site sure doesn’t seem to include mine which is still down. ARGH!
March 22nd, 2008 at 4:15 am
Some of my domains are working, some not. The ones not working are in the 208.113.*.* range that keeps being mentioned. I have to say I don’t understand why those ones aren’t working when my others are, when they’re all supposed to be on the same server (fanta).
March 22nd, 2008 at 4:19 am
Adjusting PS resources still don’t work for webserver (I’m on vilatika) nor MySQL server…
Reboot of the mySQL server still pending….
Critical Announcement! Please Read!
We are finishing the last stages of the spunky cluster and central system move as posted on DreamHostStatus.com:
http://www.dreamhoststatus.com/2008/03/21/spunky-and-central-move/
Some servers are requiring special attention and we are getting the remaining problems fixed up now. Thank you for your patience during this important server maintenance. (posted 18 mins 35 secs ago)
Open Tickets
ID Subject Message Age Actions
# Re: 3 hours 5 mins
well… we can only wait and wait and wait… … …
March 22nd, 2008 at 4:19 am
After the credit card charge nonsense, you’ve made a huge mistake by not thinking of sending out a warning email.
I think you blogged about it a while ago but not everyone follows the blog.
Emails were absolutely necessary in this situation.
Please sort this out asap!
If your blog post has a ’severity’ of ‘high’, then why not automatically email everyone?
March 22nd, 2008 at 4:20 am
Not able to get into email. Keep getting this error message:
ERROR
Error connecting to IMAP server: localhost.
111 : Connection refused
Go to the login page
March 22nd, 2008 at 4:23 am
Yes… My MYSQL server is up and running again!
After more than 3 hours of frustrated waiting… well, at least we are back online…
Next time, plase send us an amail to farn us upfront….
Ibo.
March 22nd, 2008 at 4:24 am
Hmmmm….. If DH could make everyone’s account overdue and take payments and make unpaid sites offline all at once, they could notify all accounts at once of expected outages.
I think DH needs Viagra to get back up.
March 22nd, 2008 at 4:25 am
@Vinic
None of you know anything about how shared web hosting works… etc.
Not the point. It’s about letting your customer know that the service they are paying you for is about to be temporarily discontinued.
Posting it on a blog one week before is not good enough. Direct email notification is the very least one could have expected.
March 22nd, 2008 at 4:25 am
Why are you idiots still making us pay for your service?
March 22nd, 2008 at 4:26 am
Warning: mysql_pconnect() [function.mysql-pconnect]: Can’t connect to MySQL server on 192-.–.–.–’ (4) in /usr/local/squirrelmail-1.4.9a/session_mysql.php on line 61
please hurry up!
Can’t connect to 192——– as squirrelmail
MySQL Error: Can’t connect to MySQL server on ‘192——–’ (4)