The Network is Still Bad
Aside from early indications that the router hardware swap was successful last night it seems today that things are depressingly close to the way they were before.
While core01 is running on all new hardware (except power cables and power supply) it was not upgraded. Unfortunately while configuring the new chassis with the Sup720 supervisor and improved blades and extra “high speed” fan tray, we ran into a problem.
The router complained that the power supplies we have (1300W, 27amp) are not “compatible” with the new fan tray. This isn’t to say that they can’t power it, just that somewhere a version or model number is incompatible.
We already knew that we’d need 2500W to power the new supervisor and fan tray, and we’d spoken to cisco techs about running our two 1300W power supplies in cumulative mode for now which basically makes them output 2600W worth of power. They said that should work fine. We spoke to our Cisco network engineer yesterday and he said it should work fine. Cisco’s documentation clearly says:
With insertion of equal-wattage power supplies in cumulative mode:
The system power is the combined power capability of both supplies.
So, it definitely seems that it’s a false limitation somewhere that would be solved by getting two 2500W powersupplies. These are already on their way and scheduled to arrive tomorrow.
The upshot of this is we’ll be having another network maintenance window tomorrow, September 13th, at 10pm PDT.
(Just to get everybody on the same page.. essentially ALL performance and stability problems experienced over the last months have been due to these internal network problems. High web and mail server loads are due to the lag they’re getting when NFS mounting the file system from our file servers. This causes slow/flakey performance across the board. It’s actually not a matter of overloaded servers or anything like that, it’s all this very weird network problem that has had us and Cisco stumped. We’ve now completely replaced all the related physical networking gear and it’s still happening. Cisco support and network consultants have been unable to find any problems with our configuration or hardware and have no explanation as to why things are still behaving like they are. So we’re now upgrading the core routers to be 8 times more powerful.)
Also, we’ve decided to start allowing comments on this blog.. remember the note at the top though: Posting in the comments here WILL NOT help you get your problem resolved faster.
We hope it will at least help foster some better communication related to events we post here.
.
September 12th, 2006 at 11:49 am
Cisco guys should’ve known better. Even if there is more than enough power for the system, the HSF tray for the Sup720s requires a 2500W or higher PS. This is something hardcoded into the IOS. You have some good reason to push this up through TAC and get a new engineer assigned. I’d raise heck with your sales rep too.
September 12th, 2006 at 11:50 am
Thanks for the update. I hope this gets sorted soon!
September 12th, 2006 at 11:54 am
We’re all rooting for ya to get these problems fixed once and for all - thanks for keeping us updated.
September 12th, 2006 at 11:55 am
It’s pretty rough when even “THE guys” (CISCO) gives you inaccurate information. Hopefully this last bit of work will finally stabilize the network, and then you all can spend your time yelling at CISCO and posting entertaining blogs about it.
September 12th, 2006 at 11:58 am
Let’s hope all the troubles will be gone soon!
How long do you expect the network to be down tomorrow?
September 12th, 2006 at 11:59 am
Don’t you just love technology when it presents a problem that is difficult to nail down?
I’ve just redesigned my site and it went live a few days ago. Very bad timing for this to be happening as I’m promoting the site and hoping to get visitors… They will no doubt be put off by the slowness.
Also have an event scheduled for the 23rd, where visitors to my site will be able to listen to a live broadcast preview of my new album tracks. While that’s on a separate server entirely, I still need to get my visitors to it.
Let’s hope all goes well with this next bit of maintenance!
September 12th, 2006 at 12:01 pm
Wow, and I thought I had problems with Tech support for my little computers at home. I guess it goes to show that even businesses get crappy advice just like the little guys in the home market.
I hope you guys can get this sorted out soon. Fortunately my websites and my clients’ websites have not suffered too much with all of this mess. Thank you!
September 12th, 2006 at 12:02 pm
Thanks for keeping us updated, I do appreciate it.
September 12th, 2006 at 12:05 pm
Your openness about the problem is appreciated, along with the quick response time to support issues.
Thanks!
September 12th, 2006 at 12:07 pm
Good luck getting that sorted, and I really appreciate you guys giving such detailed updates on server problems; most other hosts would have left me in the dark, until the slowness got fixed, if it ever did. I only signed up last Friday and was a bit worried about the speed, so in a way I’m actually glad that it’s hardware problems
September 12th, 2006 at 12:23 pm
Just here to give my support & thanks as well. I know you guys wouldn’t have wished for any of these problems to ensue in a million years! It’s a shame, but I’m confident you’ll pull through and I for one will stick it out. Thanks for all your late nights and consistent communication. xo
September 12th, 2006 at 12:24 pm
It’s great to have good information about the updates and maintenance and all that, but I think it would be better if you could provide some kind of an announcement list for the critical stuff. That is downtime and that kind of information.
September 12th, 2006 at 12:25 pm
I have to say I’d expect more from CISCO, but nevermind. However technology is all about learning. The chances are you won’t make the same mistake twice! I have to say that although the recent problems have been a bit annoying, it’s countered by the consistent blog updates. I feel better when I know “what” is wrong rather than finding out “something” is broken.
September 12th, 2006 at 12:26 pm
The frequent updates are very very useful and also assume that we have technical knowledge which is good
September 12th, 2006 at 12:29 pm
Really comprehensive explanation. In the few past days I was sending mails to support, at least once a day. They were always very patient explaining me what the problems are. I’m glad you have such good communication with your clients. Just keep up the good work, and of course get the job done
September 12th, 2006 at 12:32 pm
Amazing. More issues with dreamhost service. It’s been months of shit service. Why don’t you winners spend less time writing your blog posts and spend more time fixing the network. This is the worst hosting experience I’ve ever had. These troubles won’t go away, the problem is the DH staff has over sold and is completely incompetent. Thanks for the great service. YEAH - Can’t wait til my contract expires.
September 12th, 2006 at 12:38 pm
Well, contrary to JP’s carping, I don’t have plans to ditch y’all anytime soon. I frankly find your service, in every other way, exemplary and I am actually grateful that you are being so open about these internal netowork issues — transparency like this feels like a luxury.
Good luck with this next network maintenance!
September 12th, 2006 at 12:41 pm
THIS IS 100% DREAMHOST’S FAULT
Passing the bucket is not a professional response to a problem. Like any other project/business/institution, DH should have had a protocol in place should a down-time problem arise. It’s called risk management. Since reasonable problems can occur, I patiently waited after the Monday maintenance to redirect my customers - almost a week delay! The result is that this morning, Tuesday, I lost 3 business opportunities because DH led me to trust that my website would be up and running after their Monday upgrade/maintenance. Today, Tuesday, it took 2 hours for DH to respond to my SITE DOWN notice and I received no accountability. Accountability means to shoulder responsibility with actions from discounts, credits to financial, service offers. I have been told there is no timeframe for my website to be UP. I regret paying 1 year in advance since I have no leverage.
I am deeply disappointed with DH and actively seeking an alternative and requesting a full refund. Hosting.com has a 99.9% Up-time guarantee and I will try and verify this.
September 12th, 2006 at 12:41 pm
This will sound like a cliche, but my brother lives down in Austin, TX. I’m also pretty sure he has almost every Cisco certification available. He’s one of the few guys in the US with enough certifications to route Internet traffic all over the world, That is in fact what he does for a living. He might be able to find the problem if you wish to contact him.
September 12th, 2006 at 12:43 pm
I appreciate you guys and the problems you have had. I have been a customer for over four years and have always been happy with your service. These bumps in the road have not shaken my confidence. I also have total confidence in CISCO and I know they will come through eventually. I have a large amount of CISCO gear here and, while there have been occasional problems that have popped up, the service has been very good. Let me know if you want me to ask some of my CCIE friends to look into things for you.
September 12th, 2006 at 12:44 pm
Evidently some customers don’t actually pay attention to the explanation and just like to gripe with no real foundation in reality.
Here’s hoping you get these things fixed. And configuring the PSs like this will give you a little more redundancy as either one will be able to handle things on its own should the other fail.
September 12th, 2006 at 12:51 pm
Thanks DreamHost for keeping us in the loop. It certainly makes me feel better knowing that you know there is a problem (though I can’t say that it has affected me much)
September 12th, 2006 at 1:02 pm
It would have been nice if Dreamhost had said from the beginning that the router upgrade wasn’t completed as planned. It seems a bit strange to have said that everything was fine and then later say that things weren’t done the way they were supposed to have been.
September 12th, 2006 at 1:03 pm
what’s in a name?
September 12th, 2006 at 1:10 pm
Communication is all. If I see a spinning “ajax loading” symbol I wonder WTF is going on …. if I see an update here, I *shrug* and just know that we all have problems.
Good luck guys. You have my sympathy, and my continued custom. Just remember that the negative feedback is the one that shouts loudest :o)
Please do try and find space to keep us updated … and hey! find a way to help out us old (Brit) geezers who can’t convert PST to GMT to BST ;o)
September 12th, 2006 at 1:12 pm
Come on guys, crack those Cisco routers open and SOLDER!!
You’ll be the happy cisco-router-soldering-crew from then on
Thanks for the updates, and good luck resolving them… Things like this make me feel very happy customer
ROCK ON!
September 12th, 2006 at 1:13 pm
In spite of all the problems, I think you guys are doing one hell of a job. Being fair to the users and not hiding problems is exactly what I was looking for in a hosting company. Good luck with all your work, it’s really appreciated.
September 12th, 2006 at 1:15 pm
Thanks for keeping us in the loop. You guys are 10 times better than Woolnet. It often took days just ot get a responmse or find out what was up. With you guys I at least get a response in an hour or two. Many times quicker. Thanks again for keeping us updated.
September 12th, 2006 at 1:23 pm
I have to agree with cricket on this one. I can understand if you have problems–it happens to the best of us–but you should feel lucky to have the customers who haven’t left you yet. Some kind of discount/rebate/whatever would be one way to express your gratitude for our faith in you. And keep up with the updates! It’s quite clear that the problem you are having now has been going on for awhile, so it would have been helpful to know earlier on that you were aware of it. Probably would have saved you a few support tickets, too.
September 12th, 2006 at 1:33 pm
I understand your frustration and appreciate your hard work at tackling the problem. Having said that though, I can’t wait to much longer to have stable websites. I have lost customers, search engine rankings, link exchanges have been pulled, and lost I dont know how much money. If things are not resolved soon then no matter how much I like dreamhost I will be forced to move to another host.
September 12th, 2006 at 1:34 pm
Anonymous @ 1:13 pm
I like the whole not-hiding-problems thing, too, but where do you see fairness to users? Customer A and customer B pay the same for services, but customer B, whose email is down or uselessly slow for many days per month, or whose website is inaccessible too often, etc. is getting far less for his money. And while it’s true that 99.999% uptime costs (and should cost) more than $9.95/month, I’d guess it’s also true (all evidence here to the contrary - this is a temporary phenomenon) that there’s not really a market for an “only works half the time” web service. I’d consider what’s being delivered to customer B to be worth nothing, rather than five bucks a month.
So, I see the openess, but where’s the fairness? Or did you mean “being fair to the users by not hiding problems”? In which case, um, never mind.
September 12th, 2006 at 1:34 pm
Dreamhost having problems? Wow, this is unusual…..
September 12th, 2006 at 1:40 pm
Ditto to Cricket and Brandy. I’m paying for shared hosting, therefore I expect short downtimes. On the other hand, I do expect fair compensation for extraordinary loss of business. Just as you lose business when your website is down, so do we lose business ($$$) when ours is down. I love the panel admin, extra features, and 24/7 email support (except for the time it takes to respond), but the uptime has really screwed the pooch since I’ve been a member. Fortunately, I haven’t been one for long, so I’m giving you the benefit of the doubt.
I’m just glad I’m not in your situation. Best of luck.
September 12th, 2006 at 1:40 pm
I have to say, I’ve been very tolerant of all the technical issues over the last few months but it’s seeming more and more like dreamhost is simply counting on the good will of their customers. At this point, I’d really like a refund or a credit for a month of service! This has gone on long enoug that I’ve actually been pricing hosting plans at other companies.
September 12th, 2006 at 1:47 pm
Even though it’s depressing to log into bloglines and see 7 new posts from Dreamhost status, I can grit my teeth and say this feed is better than having nothing. I don’t know if every host has the same problems and you’re just transparent or if these issues are unique to you, but this status site (especially now that it’s two way) means that I’ll suck it up for the next two months and believe that improvement is on the way.
That said, I’m with those above - some sort of compensation is definitely due. I’m getting awfully embarassed everytime I send out another client-wide e-mail advising them of downtime. Most are understanding, some… not so much. Rightly or wrongly, I’ve made a bit of my professional reputation dependent on you.
September 12th, 2006 at 1:48 pm
I have to agree with Cricket and Brandy too, although the service offers a great deal to make it attractive, it has been practically useless for me for months, it is terribly terribly unreliable, at almost any given time I can’t access either a website or my email. I feel some sort of extension on my hosting or a partial refund would be fitting to make up for the repeated inconveniences.
September 12th, 2006 at 1:49 pm
I agree with most of the rest:
* I’m hosted by you
* I understand that there will be downtime
* I understand that there will be long streaks of issues and sub-par performance simply because you don’t make everything you run and in fact glue a lot of different things together
* I’m patient even with this long streak of issues because you’re being honest and telling me what’s going on
* When this is over, I would appreciate a small discount as a token of appreciation for staying with the company “through thick and thin” and for accepting performance degradations even though I paid for something different
September 12th, 2006 at 1:50 pm
Thanks for the transparency and keeping us updated, I cannot overstate how much this helps.
I agree the service has been in sucksville over the last month, esp compared to what it was earlier this year, but wonder what plan ‘cricket’ is on? Paying so much for hosting? I have yet to beat the deal DH gives but I guess that is somewhat moot if most of the services are not usable.
Also, my sites are borderline worthless, esp any php/mysql driven stuff so upgrade that sh*t anytime day or night! Cant sell my goods if you cant see the page!
September 12th, 2006 at 1:51 pm
This is killing me. I used to host my own email, but moved it to dreamhost because friends and family wanted to use it too. “No more downtime, no more problems!” I promised. Then the same with my web server, photo server, etc. “No more downtime, faster downloads, no more problems!” I again promised. Sure, I symphatize about the Cisco bullshit. I understand computers, so I understand downtime. The people relying on me, however, don’t - they just see me as not delivering a promised service. Anything is okay for a day or two… less for a week or two… this has been more than a month. arg.
September 12th, 2006 at 1:57 pm
This is one outage too many. Everybody has a bad day now and then, but you guys seem to have one after another. Worse, you seem totally unprepared for perfectly predictable contingencies, like your building power going out. Being “open” and “communicative” doesn’t make up for that.
I had intended to start looking for a new provider in a few months, when my 1-year renewal was almost used up. But I can’t wait any longer — I need a provider who knows what they’re doing.
September 12th, 2006 at 2:05 pm
While yes there have been some issues lately, this has been the only major probelm I have seen from DH in the more then a year I have been with them. There has been some small hour outages here and there, but they are always resolved quickly, or already being worked on by the time I learn of them and bring it to thier attention. I switched to DH after being with a webhost who cost more and even had a 99.9% uptime statement, but that was almost never true. On top of that, if there was any down time, they would never tell me why.
I am happy with what they do, and I haven’t seen a single reason to leave them. If just running because of one problem, well every host will have problem at some point, so good luck finding one that can ever make you happy.
September 12th, 2006 at 2:08 pm
I was with HostRocket for nearly 6 years before I moved over here to Dreamhost, and all the people complaining should just can it. HostRocket, while they started out well, had started to run into problems and were beginning to overcrowd their servers. My website started going down around noon everyday with no explanation. I knew it was down because I couldn’t get any sites on my server to load up. I mailed 5 times asking for help, asking for ANY answer, even a “we looked into it” but every time I did they just sent me a “seems to be working fine to me, we’re sorry for inconvenience” form letter. In fact, in my 6 years there, I never got an explanation of what the problem was. And that might not mean much to the slower folks in the crowd, us developers and techs GREATLY appreciate knowning the why.
Moreover, DH did not at all shrug responsibility, and I’ve been on DH for a few months now. The downtime has been NOWHERE NEAR bad enough to warrant a refund, or even an apology for that matter. Hell, your refund is the tons of money they’re spending to not only fix, but upgrade their routing services.
Thanks for the answers and the stellar service. Even with the routing problems I’ve had better uptimes and far better customer service than with other services from “better” hosting companies, including dedicated services.
DON’T WORRY! BE HAPPY!!!!
September 12th, 2006 at 2:12 pm
I for one think Dreamhost is doing a good job. I’ve searched many hosting services, and none out there give me a better bang for my buck than Dreamhost. If you are hosting a site that you expect to be up 24/7/365 then your best bet is to buy some rackspace and use your own hardware or go through The Planet to rent a dedicated server. However I’m more than happy with some downtime, because it’s not the end of the world. Worse, if you are running a web based business that you make a living on, why the hell are you using shared hosting? If what I hosted provided revenue to me, I would be a lot more careful about where my data was hosted. I’d also ensure redundancy because no hosting providor is safe.
What I respect about Dreamhost is they are being upfront about every issue. They don’t try to hide issues, they don’t try to pretend they don’t exist, and they even admit to screwing up when they do. What happens when your DSL line goes down? You speak to somebody in India who pretends like it’s your fault, and forces you to clear your cache and cookies three times and reboot. Then, you get a trouble ticket. However you have no idea who’s working the ticket or where it went, or even if you will get a response back.
Hey, if you can offer what Dreamhost offers, at the same price, with better uptime and promise faster service, then you got one hell of a business model. Start up a hosting company, and compete with Dreamhost!
September 12th, 2006 at 2:21 pm
My opinion is that DH have a clue, as in cluetrain, and are doing their level best to host as well and as inexpensively as possible. Admittedly, my opinion is in large part influenced by comparing my own experience with their very frank communication.
I’m not sure what all these calls for refunds are about. As I see it (as a DH customer for a year or so) we customers are all in this together. So they give everyone a free year, and then have to raise prices to cover the outlay. In what way are we all better off?
I actually think they will sort this out; and I am still better off letting them worry about it than trying to do this work myself. I’m just hoping the extra hardware cost doesn’t drive them out of business so I need to re-host again.
Presumably they scheduled their down-time to match their own data on overall system load to inconvenience the least number of sites. Not real convenient for me in my timezone, but I don’t know where other sites have their heavy traffic periods and the world doesn’t revolve around me and my sites.
Good luck DH, and DH customers.
September 12th, 2006 at 2:24 pm
Not really interested in the tec-talk, I want a system that functions. FIX IT
September 12th, 2006 at 2:31 pm
those of you threatening to leave, just leave already. You’re threats are not going to make anyone work faster all because you threaten to leave. They have to wait for the part to come in to begin with and if you can’t want then just leave please!!!
Complainging here to get a refund will more then likely be met with laughter, this is not the accounting or sales department blog. It’s a status blog. Instead of taking time to complain, take the time to find the write person to complain to and ask them for your money back. Jeez.
September 12th, 2006 at 2:42 pm
A few weeks ago the check engine light came on, in my car, and it would stall every time I stopped. I spent a good amount of time checking the oil, replacing O2 sensors, etc, trying to find the problem. Anyway, I took the car to a guy I new, who fixed $400 worth of damage. It turned out to be a domino effect; one problem causing another, all starting with a blown vacuum hose.
The moral of the story:
I could have just sat there and bitched about my car having problems just like some of you guys are about the servers, but you know what? Sometimes things just break, deal with it.
And at least dreamhost has a check engine light (dreamhoststatus.com)
September 12th, 2006 at 2:56 pm
Dreamhost is a joke. My little sister is a better network admin. I mean, who upgrades rails and doesn’t test. Welcome to unix admin 101 DH.
I should have canceled before the 90-day money back period. If my site isn’t down, its slow. If it’s not slow, they’ve decided to upgrade rails
without testing. All I read about are hardware problems. I run over 300 Linux and Solaris boxes and rarely have hardware problems. We
have many large netapp installs too without problems. Think about it. Amazing!
September 12th, 2006 at 3:01 pm
Thank you guys for getting around to fixing it. I know you’re probably being bombarded with obnoxious comments at the moment. I do have to say I am slightly disappointed in how this turned out, our site has been horribly laggy for quite some time, and we’re losing members. Once it gets fixed, I hope to recitify this situation, but notwithstanding, it seems this problem has taken quite a chunk out of the overall productivity of our site. Thank you for listening to my obnoxious speel, and please try to avoid such an issue in the future.
Thank you,
Josh
September 12th, 2006 at 3:02 pm
As I understand, one of there routers are having issues. That is fine but in my experience there should be two core routers for the Dreamhost servers. One as a main and the other as a backup. Cisco contains all this functionality and from what I understnad most major companies go the dual core router route. Why does Dreamhost only use one?