Router Upgrade
In addition to the recent router hardware replacement, we will be performing multiple upgrades to our primary core router at 10pm PDT tonight. This will entail the same amount of downtime, roughly 30-60 minutes, to the same services (HTTP, Mail, webpanel, SSH, etc).
For those of you interested, we will be installing 2 2500W power supplies, a Sup720 supervisor, 2 fabric enabled 48 port GigE blades, and a high speed fan tray.
We will post again when the upgrade has been completed and network functionality has been restored.
Happy DreamHost Late Night Router Upgrade Team!
.
September 13th, 2006 at 9:13 pm
Thanks for your hard work!!
I hope it goes well~
September 13th, 2006 at 9:17 pm
Looking forward to once they’re up and going.
Kudos to the DreamHost team for all their work on this long and strenuous problem
September 13th, 2006 at 9:31 pm
It is 6:19 a.m. here.
I woke up … and was alreay thinking of all of you working there.
What do you feel? How do you talk to each other? How are you?
The sun is rising here.
I am sending you my best thoughts.
I have made a little page with photos to you yesterday. As a present. And as good wibe. It is in the /dreamhost folder on my site
I know you do your best.
Thanks you, You all.
Tamas
September 13th, 2006 at 9:40 pm
Good luck guys!
September 13th, 2006 at 9:47 pm
Thank you for keeping us informed and for all the hard work! It is greatly appreciated.
September 13th, 2006 at 9:53 pm
Keeping my fat fingers crossed.
Anyone know of a McDonalds opened late I need my 23 cheese burgers.
September 13th, 2006 at 9:55 pm
Unfortunately 10pm for you is 2 in the afternoon for me. My site development time.
Hope it’s fixed soon!
Helen
September 13th, 2006 at 10:08 pm
Gah, I was working on my blog site, and bam it wacked…
Link: http://rage-network.com/blog
If anyone needs freelancers, contact me.
Aim- x3xRx
September 13th, 2006 at 10:08 pm
And let the games begin!
September 13th, 2006 at 10:09 pm
You couldn’t have picked a worse time. :p
Just kidding. It’s not critical.
September 13th, 2006 at 10:10 pm
I sense idiots approaching this comments thread as we speak.
September 13th, 2006 at 10:11 pm
Hope Router Upgrade successful!
September 13th, 2006 at 10:12 pm
i’m sick of all the dreamhost downtime
get a grip of your business already
September 13th, 2006 at 10:13 pm
Good luck guys… I can tell you just started this within the last 15 mins… everything went out all of a sudden… guess I ought to go to bed anyway.. but I’ll try to read ahead on the status page before I plan a big “all-nigher-design-spree.”
Anyhooo.. good luck with the work, thanks for everything!
Matt
PS: btw.. just an *extra* thanks for the .snapshot folders… it saved me a few hours of extra work today.
-MG
September 13th, 2006 at 10:14 pm
Good luck. May the force…uh…not get in your way.
September 13th, 2006 at 10:15 pm
What the hell is going with the servers? It’s always down! Get your act together!!!!!!!!!!!
September 13th, 2006 at 10:15 pm
That was less than an hour warning. I’m right in the middle of something… :o(
September 13th, 2006 at 10:17 pm
>Unfortunately 10pm for you is 2 in the afternoon for me. My site development time.
Hope it’s fixed soon!
And unfortunately 10pm for DreamHost is 10pm for me, which is *my* site development time. Here’s hoping that this clears things up for a while.
September 13th, 2006 at 10:18 pm
lol, I had very important thing going on rightnow, and its down. hope it get well soon.
Thanks for your hard work
Take care
September 13th, 2006 at 10:18 pm
Ah, so that’s why it didn’t load.
I like being on EST. It’s 1am here so everything is fine with me.
September 13th, 2006 at 10:19 pm
I can’t believe I paid for a year of this
September 13th, 2006 at 10:19 pm
As much as I have complained outwardly about the service over the last few months, you guys are a different (in a good way) kind of hosting company. I do appreciate all of the open communication and the technical explanations. Good luck this evening.
September 13th, 2006 at 10:19 pm
This downtime was planned and announced yesterday. Let’s all just hope this solves all our problems. Changing webhosts is such an ordeal.
September 13th, 2006 at 10:19 pm
Just wondering - why does replacing one router bring everything down? As a long-time network engineer I’ve been trained to design everything with redundancy. Replacing a router should allow traffic to use the backup router and/or backup path.
September 13th, 2006 at 10:20 pm
Get a grip, Paul. The servers are not always down. Hell, I’ll take occasional downtime if it means greater reliability. I just wish I would have read this before. I started uploading my site and everything quit. lol
Good luck, guys!
September 13th, 2006 at 10:22 pm
Keeping my fingers crossed for better days at Dreamhost.
Thanks for your hard work.
September 13th, 2006 at 10:22 pm
Don’t give up on Dreamhost! Yet!
Their downtimes are indeed becoming more frequent, and it’s in the middle of my day, so things really slow down.
mmm
Kenneth
September 13th, 2006 at 10:22 pm
It would be helpful if they said what time zone this down time is scheduled for. It would also be VERY helpful if they identified which servers will be affected by this router replacement. As a dreamhost user, I need to know if my site’s current downtime is due to this router issue or if I need to report it as a seperate incident. We had downtime the other day because the /tmp directory filled, and I don’t know if the site is currently down because of that or something else.
Seth
September 13th, 2006 at 10:22 pm
An hours notice? The notice of downtime was posted on cpanel 11 and a half hours before it was due to start..
September 13th, 2006 at 10:24 pm
Down again. I am close to canceling my account with dreamhost. I love your service, when it works. Which is next to never. I really wish this was not a constant problem.
September 13th, 2006 at 10:24 pm
My Dreamhost sites are working again. Is it all done?
September 13th, 2006 at 10:25 pm
These equipment and “incidental” problems have become chronic in the past several months. C’mon guys, you’re giving us fair to short notice of these fixes, but what is your overall infrastrcuture status that Dreamhost is so prone to thse fubars? Something is not right with your network planning and engineering if you’re so vulnerable to just about any ball thrown at you… these days seems like it doesn’t even have to be a curve ball….
September 13th, 2006 at 10:27 pm
Seth: It says right in the post “10pm PDT”. That’s Pacific time if your in the US, GMT -7 or -8 (depending on Daylight Savings).
Oddly I did something else after posting the first time and forgot about it and tried to check my email. LOL
September 13th, 2006 at 10:27 pm
a Sup720 supervisor? Droppin a lot of scratch!
I hope you didnt buy a refurb!!!!
September 13th, 2006 at 10:28 pm
thanx for keeping us informed,
because knowing is half the battle.
keep at it.
September 13th, 2006 at 10:29 pm
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
Happy DreamHost Late Night Router Upgrade Team!
September 13th, 2006 at 10:30 pm
Good luck fixing this… perhaps I should have got a host in the same time zone so as to have outages at 10pm instead of 3pm.
September 13th, 2006 at 10:31 pm
As much as I like the services here when it’s actually up and running I’m about to the point where I’m ready to switch back to my old host that cost me $100/month. This shit is getting OLD real quick. With the number of customers you guys have I can’t imagine you guys don’t have the cash to build a reliable network and get some competent people to run it. Please take care of this bullshit once and for all!
September 13th, 2006 at 10:34 pm
ditto to the comment about timezones and server names.
With so many outages recently - how am I supposed to know which ones you know about, and which ones you don’t?
PLEEEEEAAASSSE put timezones on EVERYTHING - like this, maybe - “Posted at 5:33 UTC
September 13th, 2006 at 10:37 pm
Ya know, I’ve only been a DH customer for a little under a year. Before that, I ran all my services from my home Linux box. Everything worked. I had no real reason to move to DH, except that it’s easier to deal with when it’s someone else’s box and I don’t feel like “working” when I get home from work.
Does the downtime suck? Yeah. Is it annoying that the same problems have been reoccurring or otherwise remaining problems? Yeah.
But to the people that insist on posting that “DH is teh sux0rz” and all that drivel, a few words:
1) THESE KINDS OF PROBLEMS ARE NOT SIMPLE FLIP A SWITCH AND IT’S FIXED ISSUES! I do system and network administration. It took us months to track down an intermittant problem before. It took us weeks to track down a semi-constant problem, because every time we got close to an answer it disappeared. These things happen. If you don’t do this kind of work, you don’t understand what it’s like. Does Josh and company do this for fun? No, it costs them money. It costs them reputation. And if they’re anything like me, they don’t *like* seeing machines and networks down. They take pride in seeing everything running smoothly. So when they say they’re doing all they can, I believe it.
2) IF YOU THINK YOU CAN DO BETTER ON YOUR OWN, GO AHEAD! I’d love to see you try to manage a network and systems as complex as what goes on there. And for those of you who say you can do better because you’re in this business and you know what you’re doing, I think you’re full of steaming excrement. Why? Because if you were such an expert, you’d be doing it yourself, and/or you’d know how complex it can be. Don’t tell me you’ve got your MCSE and know everything about Cisco hardware. Don’t tell me you installed a hard drive in your k-rad gamerz box, so a NetApp should be a cakewalk. If you really know how all this stuff works, then you also know how difficult it can be, and how challenging network administration can be. When it runs smoothly, it’s a thing of beauty; and when there’s a problem, it’s quite ugly. And lastly, if you were such an expert, you wouldn’t be whining about how these people need to hit the books, learn how to run a network, keep a machine running.. instead you’d be wincing a little, feeling glad you’re not them right now, and probably stepping away from the computer for a little while.
Got encouraging words? Great. Got desparaging remarks? Here, have a nice tall glass of STFU.
September 13th, 2006 at 10:38 pm
OOOH I just got throught eating 23 cheese burgers I almost feel full.
May Dreamhost have many moons of success.
September 13th, 2006 at 10:42 pm
I 100% agree with Steve.. exactly what i wanted to say.
September 13th, 2006 at 10:43 pm
This is starting to get to me… for a week now, I’ve been experiencing downtime with various services, webmail, ftp, ssh, and now even web gallery. Explanations in the emergency page include all sorts of stuff about replacing servers & routers &c.
Well, DH, I’m JUST NOT INTERESTED! I pay for web hosting which means I expect service! I can undestand a scheduled 1-2 hour maintenance break every once in a while, but I cannot undestand a period of ONE WEEK with continuous interruptions in services. This is unprofessional!
The best explanation ever when I asked DH support (admitted: DH support normally works like a charm) about IMAP server dropping connections and timing out all the time: “This may be a cookie handling problem.” Fire that guy.
I’m very disappointed with DH service this time.
September 13th, 2006 at 10:46 pm
Happy DreamHost Late Night Router Upgrade Team!
eh..it’s one hour already, and my site is still down.
September 13th, 2006 at 10:47 pm
Steve,
If every time I did a “core router upgrade” I brought the _entire company_ down for 20-30 minutes, I would be out of a job.
Maybe I can’t fault DH for the network problems (I’ve had my share of crazy Cisco bugs) but I CAN fault them for maintenance that brings everything down. Two core switches and routers with edge switches dual-homed and bam, you can upgrade things one at a time w/o bringing everything down. I don’t get it.
September 13th, 2006 at 10:49 pm
You guys kick @ss. Less than 2 hours down for what appears to be a major overhaul.
September 13th, 2006 at 10:51 pm
Ok so now we should be all good and the network should be all happy. Lets see what happens next
September 13th, 2006 at 10:51 pm
mara .. pay more (more!), for a better service.
September 13th, 2006 at 10:55 pm
I got a error: “failed: Connection refused” after they said “Router Upgrade Complete!”
September 13th, 2006 at 10:55 pm
Ok so now we should be all good
Not all, still down and out.
September 13th, 2006 at 11:01 pm
I agree with the above post about getting a grip on your business. Some of my sites are up..some are down. IS THIS EVER GOING TO END?!?!?!?!?!?!?!
C’mon now!
September 13th, 2006 at 11:05 pm
Common people! Give them a break! you wan’t it fixed, they’re fixing it. Stop complaining already.
September 13th, 2006 at 11:08 pm
I’ve been a DH customer for 6 years:
1) I have never known them to have any excessive downtime (past *and* present).
2) When something crashes, they get right on it and within minutes it’s back up and running.
3) They’ve always given pleanty of notice.
4) Their estimated downtime is usually way higher than the actual downtime (e.g., they said it’d take 40 minutes and it only took 10 minutes).
September 13th, 2006 at 11:16 pm
Steve… have yourself a glass of STFU!
I’m a customer here. I have been for well over a year now. When I got here they were having problems and I rode it out. They admitted at that point that they were having scaling problems. About a year later things seem to be crashing down around their ankles. I’ll repeat… I’M A CUSTOMER! I pay for a service and I expect to get it. I’m not oblivious to the fact that shit does happen, but with the amount of shit happening lately I have to question exactly what the hell is going on over there. I hope the best for them(and me) and I’d love to keep using the service because i really like the simplicity of it all, BUT if this keeps going on I’d rather pay six times the price (as I used to) to have the the 99.9% uptime I expect when I’m paying someone for hosting!
September 14th, 2006 at 3:59 am
My mail is downloading a little too fast now… you guys are dope.
September 14th, 2006 at 9:18 am
“AntiSteve” (Cute :P) : Yes, I agree that there should be redundancy built in. There should be a way to failover to another core router so that one can be upgraded. I’d love to do that at work, but we just don’t have the money for it. So if I have to upgrade a firewall or the NetApp, things have to go down. For some businesses, this is tolerable, for others it is not. My flame was more directed towards those that complain, as I saw in another thread, that their 6-year-old sister could run a network better than this. Ya know what? With unlimited funds, I might be able to do a better job. But that money’s gotta come from somewhere, and if they don’t have it to buy into that kind of redundancy, then some downtime is to be expected (and with what they charge for what you get, I’m not surprised they don’t have the extra cash to have two or more of everything).
Lucky: That’s great, you’re a customer. You admit that you used to pay six times what you pay now, and say you had better service. What do you expect then? I didn’t see anywhere that Dreamhost guarantees 99.9% uptime.
September 14th, 2006 at 9:54 am
Steve, you seem to be the most adamant here.
I stumbled into these new posts again because as I allowed many hours to pass-by, my client’s websites are still down. Bah, humbug!
Now, really, a lot of us here without lots of letters attached to our names can run a better network. It’s unfortunate that some of my clients got enticed into signing up with (nightmare) DH. We, the web guys suffer the day to day of–
> up then down then up then down then slow then down then up then slow, slow, slow then down, down, down then slow (toss some dice) ah we’re UP hoorah! then down
So, yes, I know the DH Late Night Router Upgrade Team is hard at work but from a service-level perspective DH sucks. If these were a free service I’d STFU. And I don’t need to get RSS of some more excuses. I need the damn service UP. Period.
and don’t give us that BS about the non-existent 99.9% uptime guarantee clause. let see if they put that statement in their sign-up ads
What’s odd to me is that while DH seem to alway have these network uproars other hosting services are just silently reliable you can even hear a pin drop.
Of course I wish that would happen and challenge the whole DH team a bit to drop the hype and get the damn network to just work. Bite the excuses and just provide us a good ‘ole rock-solid network complete with redundancies, fail-overs, back-ups, etc.
I guess you’ve got your job cut-out for you, Mr. Josh.
September 14th, 2006 at 10:33 am
Sounds like you guys have snakes in your internets.
September 14th, 2006 at 9:00 pm
I pay for my services and expect a certain level of said service. I am not paying an amateur, apparently your level, for service.
September 15th, 2006 at 8:09 am
hey! you listened to my advise and got 2 big power supplies. I think that is the right thing to do, thanks for listening!