Blingy file server upgrade
Posted (March 10th, 2008 at 9:52 am PST) by KellyOn Wednesday night, March 12, 2008 starting at 10:00pm Pacific time, we will be performing a major OS upgrade on the file server that handles the entire blingy cluster. This upgrade is at the direct recommendation of BlueArc to hopefully resolve periodic slowdown issues and other quirks of the system. The estimated downtime by the BlueArc technician is 1 hour.
Quick update: This is the first upgrade we couldn’t do live behind the scenes. We have already upgraded once to the latest version that could be applied live to the clustered heads.
Update 11:15PM Pacific time: The BlueArc technicians are still working on the upgrade. It seems they are nearly completed, and one of the redundant pair of heads seems to be serving data right now. Stay tuned for a final update.
In other news, while BlueArc and I were standing there next to the file server discussing the problems that have been plaguing us at 7-10pm, it happened! The unit crashed and completely stopped serving data. I left them to gather their crash data and continue the upgrade process.
update 11:58PM Pacific: Everything looks to be better. Please contact technical support if you are having any problems!
46 Responses to “Blingy file server upgrade”
well this is good news…lets hope all goes well
Really? This has been a problem for over a month, and their brilliant solution is an OS upgrade? Could they not have done that a month ago when all this started? Is it not pretty basic computer troubleshooting to check firmware, bios, OS updates, and things like that? I hear there are these guys called Microsoft that do it automatically over this crazy thing called the internet.
Perhaps it is Dreamhost that should dump Bluearc, since this outage has caused so many to dump Dreamhost. It appears that even the most noobish noob could have figured out that you weren’t running the latest version of the software.
Great to see an update. Server issues make me nervous considering my last host, but I’m sure this will go fine.
Thanks for letting us know.
@Jon: Running the latest version is NOT always the best thing to do. Moreover, the LAST thing I’d want my production servers to do is automatically patch themselves just because there is a new version of some software from some vendor. DreamHost is following a much more robust and mature approach to system updates: analyze what is actually wrong, determine _if_ software updates would correct any of the problems, then decide how to proceed with any useful upgrades.
@Nyhm: I think you misunderstood the joke. It’s more about timing than automation. Are we supposed to analyze and determine for over a month? Is that mature or robust?
I’m just hoping everything comes through okay… I’m getting tired of my FTP being buggy when it feels like it. Hope it works…
Fingers crossed…
Ok, so now I get my site to actually get decent page loads versus my site literally hanging. I just hope this doesn’t break more than it fixes….sigh..
you suck
will be re-hosting soon
I’m seriously worried. I’ve only been a customer for about 3 weeks and all I’ve had to deal with were constant issues that are not on my end.
Wow, I can’t believe people are actually complaining about this… This is great! The way I see it, yes, these planned maintenances mean a little downtime, but overall, everything will hopefully be much faster.
I wonder if anyone would be complaining if Dreamhost were more like 1and1 or something and not even let people know about their planned maintenance. You know, before you go and complain again, why don’t you go and get some redundancy and switch your records to them when you hear of this planned maintenance, and when that company has planned maintenance (which is TYPICAL of any computer, even yours), just hope and pray they let you know about it so you can switch back.
My guess is, your site doesn’t get any traffic to begin with, otherwise you would have done something a little smarter than get non-redundant web hosting!! All of you idiots who are complaining aren’t even linking your website (let alone a Dreamhost hosted website), so all I can do is guess that you either don’t host with Dreamhost, or are one of the losers who work elsewhere?
Oh, and just in case you are wondering, I don’t work at Dreamhost, even though, I would like nothing more than to work for such a wonderful company who actually cares about their customers enough to let them know what is going on.
-Daniel
actually, in IT, “newest” DOESN’T always mean “better”. So please stop being so childish, i’m happy that dreamhost keep us informed about these - most of the hosts doesn’t.
Well, my luck would have it that I signed up for Dreamhost and ended up on this file server, so my page has had problems from day one.
That being said, I am very pleased that there have at least been status updates posted about this problem. If you guys think DH isn’t handling this well, try hosting your site with Host Rocket. When your website is having problems, they don’t tell anybody what’s wrong. You can’t get service status on ANYTHING and they act like nothing is wrong.
At least when I noticed my site being slow I got a quick response leading me here, where I could see that they’re aware of the problem and working on it. That’s 100 times better than my Host Rocket, who will leave you in the dark for weeks.
@Jon may I also note that MS is NOT really a great idea for servers, there are far too many vulnerabilities to call that “secure” in any way. So before you say that Microsoft is great because they do that for you, you need to realize that there are many advantages to not running a MS server OS.
@Jon : Point taken. If there was a “joke” in there, I missed it.
@Mike : I totally agree (and then some): MS Windows is a TERRIBLE server platform. I’ve had experience with Windows NT and 2000 servers vs. Solaris. I’m no Solaris fan, but trying to use the MS server in a production environment was the worst experience imaginable. Windows is a tinker-toy OS compared to Solaris, various Unixes, and even Linux. (Just my personal experience… not trying to insight a flame war.)
I am glad to see such positive interaction from DreamHost with their customers. I came to DreamHost from another provider who went supernova conglomerate on me and just plain don’t care about their customers anymore, and to this day, they are still having far worse problems on a good day than DreamHost has on a bad day. I got tired of 3 minute page response times being the norm and them rebooting their servers midday, only for their tech support turn around and blithely tell me “no problem here, must be your Windows settings.” I moved here and never looked back.
@Mike @Nyhm : I agree with both of you. I came from an IIS host, and will never go back to Windows. I was making a joke at the vendor actually, because this problem has been going on so long and now they want to upgrade the OS… Perhaps they should adopt a more proactive support model in the first place, rather than letting issues go on forever hurting their client and their client’s clients. I chose MS for the comparison because I figured it would be the most insulting.
I may have gone too far when I insinuated that a file server vendor did not know what the internet was :). It’s hard not to take a long term issue like this personally.
I am not able to access my domain again and i really want to know wat the problem is
I’ve been w/DH for about 9 mos, have enjoyed all of the up-time and have been very pleased with tech support. Even with that billing snafu a few months ago, they were responsive and humble about it. I was simultaneously using another host whose email server went down about once a week and would mark a support ticket “resolved” even when they answered with a “hmmm, we don’t see a problem.” It put me in a very uncomfortable position with my clients to pass on such lame explanations. Not good - some of them receive orders via email forms. DH does a good job of accepting that it’s an imperfect world and makes it as painless as possible for their customers (and thus, our customers). It’s a good business practice to be up front, *especially* when things suck.
That said, wouldn’t it just figure that this was the week that I planned to move my other client’s site over to DH and wash my hands of Host B? It is 15 hours after the scheduled upgrade. What’s up? I’d like to see a posted okey-dokey before I tell my client to throw the switch on the DNS.
Let us know! Thx!
I don’t know whats going on i cant use my ftp server the way i want to. The way i am paying for this is a issue hasn’t gone away. I have small problems with dreamhost witch doesnt bother me. but when i want to upload files more then 20mb it will go halfway then quit. So i hope this does stop all the issues with this.
Dude what the fucki I joined dreamhost a month ago after hearing good things. This is fucking nuts, my site has been down for the entire month it seems!
My website is due to launch soon but all this down time is taking away from it. I hope this gets resolved soon. Who is gonna makeup for my lost time and my programmer’s lost time? I am losing money like that.
We should have some of our hosting money refunded. This is too much.
Good luck with.. I hope it works i need to check my email ( eek.. )
Cindy: GET. DEDICATED. HOSTING. Do not bitch about being on a virtual shared server.
Sweeet, a free upgrade! Check this out people! I have had nothing but good experiences with DH. Like some of you who are experiencing issues, get some rest, you look like you need it. 
A few problems should not create such a negative atmosphere. Keep up the great work DH (and BlueArc). I’d really appreciate if you please be very timely. I have some work to do on my site. :S
Thanks very much.
Hah.. i finally get a moment to work on this thing.. and my site is gone.
Oh well.
Anyone complaining needs to realize they are working late hours and you are paying peanuts for alot of space and bandwidth
@Tiki Barber: Yeah, No Big Deal…
People who can’t appreciate the hard work and dedication it takes to run a system like DH, should be dragged out into the street and hung by their toes. 
I bet you folks that are up and whining about it; didn’t even think that some poor soul from BlueArc is working late too. I’m also pretty sure, some nice DH employees are up burning the midnight oil. 
How would it make you feel, if someone was dependent on you, and could do nothing but complain about an upgrade?
Try looking at the bright side of something for a change?
WOW! I cannot believe some of you are complaining about this. Sure it sucks to be dead in the water, but get over it guys. At least I know what’s going on, unlike my former host.
My website was running pretty smoothly prior to the system upgrade, but now it’s sluggish, though I’m not sure if it’s because the system is still in its start-up phase.
Wow, I did not realize that so many other people were having the same issues as myself. No idea why they kept telling me it was my fault, my script using too much memory, etc… I just hope this fixes it AND nothing is lost (data wise) as that would totally suck.
My site is working again, but it won’t let connect via http://FTP. It keeps saying the connection was refused. Is this just me or part of the updating?
FTP just started working for me, and my website is running very fast now.
not working for me, I’m on bowser and I’m unable to ping the server.
Seems to be better so far…
AWESOME!
Just got an error:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@anythingindie.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request
Whatever. Site is yet again unreachable. No response. And this happenes every god damn day!
Whatever you do, NEVER EVER sign up with nightmarehost. They are not reliable. They always make excuses.
This is going into Day #3 of my site being down. I’ve never seen anything like this before. This is unacceptable. Can you all try going to http://www.aljera.com and let me know if it’s down your end also? Thanks.
@Chris Craft, site pulls up fine for me from Comcast in MS as of 10:54 pm Cent
Works just fine for me, Chris.
Also, my site’s performance is way up. None of the sporadic slowdowns and non loading; it’s running smoothly. Even the forums, whch tend to be a slowdown hotspot.
Thanks, Dreamhost!
My site still doesnt work. And ssh runs like shit. What the fuck now Dreamhost?
I’ve only been a customer for a couple of weeks, but paid three years out… hopefully the performance gets better than it has been the past couple of days, we’ve been experiencing poor performance, extreme slowdowns, and dropped connections. Simple wordpress blog with a handful of posts… 
Maybe it is just me, or maybe my impression of performance is now tainted, but the performance problem still seems to be persisting …
The performance problem is more or less constant. Site works as intended 5% of the time, the rest it is either down, nonresponsive or extremely slow. I have told the fuckers to move me to a better service, but to no avail. They don’t give a shit if your site is down. I so fucking hate dreamhost!
thanks
توبيكات
Good luck. Filer upgrades always make me nervous.