File server software problem (Update!)
One of our file servers was rebooted to clear an error with its floppy drive, of all things, and is not properly coming back online. We are working on identifying the problem and resolving. No ETA currently.
–
The file server is now booted up. I swapped out the bum floppy drive and it booted up on an older version of the OS. I do not know why, but that floppy drive was causing some weird errors in other parts of the system, so it could have prevented the updates from being written. I made and booted up off the 9 floppy disks the OS takes up, only to have disk 7 be bad, remade it on a new floppy and the machine booted up just fine.
I will now be researching how to ensure the OS on the hard drives is the correct one to prevent this in the future, and fixing any crashed servers this may have caused.
Tue Nov 7 09:46:17 GMT [rc:notice]: The system was down for 4145 seconds
.
November 7th, 2006 at 1:53 am
I wonder if this is the same culprit that has been plaguing our site for the last couple of days?
November 7th, 2006 at 1:54 am
Shucks…and I was makin’ progress, too.
Good luck y’all.
Server performance through the 2nd and 3rd quarters of this year was a bit shaky, but things have improved a lot lately. Hopefully everything will be OK soon…
November 7th, 2006 at 2:05 am
Yeah, shucks.
I’ve been pretty happy with Dreamhost and even made many recommendations to clients and friends.
Of course, just as we are launching something a little more serious/mainstream - it’s definitely been shaky and causing me to wonder if we should be looking for a new host. I’d hate to move now - I was just getting comfortable!
November 7th, 2006 at 2:06 am
are you guys on “ugh” at all?
November 7th, 2006 at 2:14 am
Not on “ugh”
But this is about the 3rd time in 2-3 days where the site(s) have become seriously slow, unreachable or down.
November 7th, 2006 at 2:16 am
Seems to have gone from bad to worse…now all the sites are down, not just the mySQL-fed sites.
Let’s hope it’s due to the all magical “reboot” taking place just this second.
November 7th, 2006 at 2:24 am
Yesh True… We need to find better host now. If this will happen next time i surely will remove my account from your @#^#.
November 7th, 2006 at 2:30 am
This is just crazy. We’ve been having problems all day with Dreamhost and our site. It’s funny, I was literally about to reup for another year with Dreamhost- literally today- and I just dont think thats going to happen. This should never happen to a company like DH. VERY dissapointing. My site is 100% down- this is my business! When it’s down, I’m not making money.
November 7th, 2006 at 2:36 am
*arches eyebrow* Seriously, I’ve bene a customer for… oh … let’s see, years now. IN that ENTIRE time I have had few problems, and 99% of those have been resolved in a VERY short time. And a ‘big company’ cannot predict other, underlying hardware faliures, etc. all the time. Being that I own home computers which have literally just up and died without ANY warning watsoever … I can fully understand a webserver going completely screwball over something so minor.
Because I work on my own system and seen others, and have laid witness to some fo THE most retardedly screwed up errors on earth which cause outages.
THIS is completely minor compared to what I’ve witnessed on other hosters, and heard from others who used other hostings. If you can’t live with a little downtime (which I may add is an absolute going to happen with ANY host - there IS no 100% uptimes!) then you need to really stop and think a lot.
Seriously, I’ve seen downtimes form other hsots be WAY more then “one day” even … I’d rather ride the storm out here, and remain with someoen who’s been reliable and quick to fix problems, then chance myself anywhere else and be bothered to deal with resetting up, etc.
In fact, 90% of the internet has issues ANYWAY … considering I’ve been browsing all day and feelign the lag/issues and loads… and I KNOW it’s not just me as I’ve asked friends.
But hey, you want to leave that’s your perogative. Though if it’s fixed ASAP from this, can I laugh? Really, I have a few downed databases, and it’s NOT killing me. Even if ti was money … the world will realize stuff happens.
November 7th, 2006 at 2:43 am
their up time is very good compared to many and there service is great too, compared to many i’ve experienced over the years. sometimes all the issues come at once and other times long periods of up time. dh’s power supply is probably not as good as here in new zealand. dh has had a few power issues during hot weather.
November 7th, 2006 at 2:44 am
Oh noes, the internet si broken. Umm, are you even reading what you’re saying dude? I’d check my ISP before proclaiming that the internet is malfunctioning…
November 7th, 2006 at 2:48 am
@Grant - yes, im on ugh and can’t get a response from my sites.
I suppose i had it coming… ugh has been completely unaffected in the past from all other issues
November 7th, 2006 at 2:51 am
hi Matthew .. i just wish my AdWords campaigns could pause themselves when things like this happen.
November 7th, 2006 at 2:51 am
It happens. A lot. But what do you expect for 10 bucks/mo??!!
November 7th, 2006 at 2:56 am
hm, system is still up but ugh is still busted
November 7th, 2006 at 3:03 am
I’m on ugh and I’m actually getting very tired of the downtime. I’ve contacted support and all they have done (supposedly since I never received the confirmation email from them) was debug the database that holds my WP installation. It isn’t the database it’s the server, yet no one has EVER addressed ugh. This has been going on for the last 8 months.
November 7th, 2006 at 3:25 am
Hmm, the file server may be up, but is it possible that there is a problem with (shared) web servers? Do they have to reboot too? Cuz mine seems to be down ATM
November 7th, 2006 at 3:33 am
My site is still not working
November 7th, 2006 at 3:42 am
i can take the down time, anything is better than my last hosting co. (hostdepartment.com)
November 7th, 2006 at 3:43 am
Sites are back. Thanks for the speedy response
November 7th, 2006 at 3:46 am
nothing is back
November 7th, 2006 at 4:04 am
MySQL Server ’snarf’ is down!!! (and alss my sites)
See what the panel says when I tryed to create a new DB: “INTERNAL ERROR CREATING DB: connect_admin failed for snarf:mikki. Please let support know!”
November 7th, 2006 at 5:10 am
I’ve tried to ftp to vanowen.dreamhost.com and it does not accept connections. It replies to ping, but my site is down and ftp isn’t working… :/
November 7th, 2006 at 6:00 am
I would have thought dreamhost would have some kind of “mirror” system in place, just in case its servers died, that meant a backup server would be used to host the clients homepages while fixes were made to the problematic server… obviously that is not in place as my site is down down down…
November 7th, 2006 at 6:15 am
Puta que pario… Warning: mysql_connect(): Lost connection to MySQL server during query why???????
November 7th, 2006 at 7:34 am
We’re really livin’ it up with these slowdowns and outages!
November 7th, 2006 at 7:37 am
At 10$ the only thing they mirror is disk space and using nsf mounts they can make a new machine replace an old machine in about 10 minutes. That’s pretty damn good in my book.
hostdepartment.com - 99.99% uptime on windows servers. Yeah, right.
And they claim no limit on anything - that sounds like a shared recipe for disaster.
November 7th, 2006 at 7:39 am
Actually, it doesn’t appear they’re all windows. Maybe they aren’t as dumb as I thought they were. But I’m not goin’ anywhere.
November 7th, 2006 at 8:11 am
Floppy disks! Floppy disks!!!! Get with it granddad
There appears to be a problem with the server called ‘payday’ also.
In my opinion Dreamhost are pretty good at dealing with outages, I’ve been on other hosts who have lost me all my files, left me without email or website access for a week, and other such horrors. Comparatively speaking Dreamhost are good.
November 7th, 2006 at 8:11 am
Guys, chill. Seriously. DreamHost never promised 100% uptime for your account, and even with a few hours here and there of downtime, it’s by far the best deal going that I’ve ever seen.
Do I get frustrated when my sites are down and folks are barraging me with e-mails me asking what’s going on? Yup. But so far, nobody’s died, no locusts have invaded, no frogs have fallen from the sky. If your site is soooo important that you absolutely, positively can’t handle any downtime, then you should build your own datacenter and host it yourself. (Although I have to warn you, if you think you can do a better job yourself, you’re most likely very, very wrong.)
And for those who are saying, “I’m going to go somewhere else,” I’ve been at several other hosts and paid a lot more money for a supposedly better quality of service. If you think that DreamHost support is non-responsive and their uptime is bad, well, I feel sorry for the pain and anguish you’re going to experience at your new home.
November 7th, 2006 at 8:42 am
Dreamhost is definitely not a reliable host for anyone running a business. The last time this happened (there was a **THREE DAY OUTAGE** just a few months ago) I swore I was going to switch and got distracted. Now it’s moving up on my to-do list,
November 7th, 2006 at 8:51 am
My site is down.
November 7th, 2006 at 8:55 am
site down too, opened a ticket, hopefull we can get an ETA so I can answer to my superiors…
November 7th, 2006 at 9:02 am
At least DH is upfront about every outage and details what is going on regarding the problem and the solution. My last two hosts only acknowledged a problem if I went to them and even then they were very vauge about what the issue was and how they fixed it. Give DH a break man.
On the other hand, while I am sure there is some good technical reason (I don’t know anything about DH’s file servers), floppies? Ugh…
November 7th, 2006 at 9:03 am
You had to boot from floppies??? Haven’t you heard of bootable cds? Come on guys!
November 7th, 2006 at 9:05 am
it looks like all my emails are down now, but the sites and forums seem to be working….so far.
November 7th, 2006 at 9:42 am
Yes, it seems like all of my sites are down. Fantastic job guys.
Been here less than a month too.
November 7th, 2006 at 10:40 am
My mail was unreachable via POP for about three hours today (8AM-11AM EST)…could still check via webmail, so not exactly The End Of The Net. Though I was unable to log in to panel.dreamhost.com to report the problem at the time. Other than the time spent trying to determine whether it was DH or my ISP that was at fault, no big.
November 7th, 2006 at 10:46 am
Many of my sites seem to be affected on the ’skittle’ server as well. Cron jobs, phpBB installs, dokuwiki installs and olate download are all returning errors. I do understand that problems do arise and they are, no doubt, working towards a resolution.. but I think that problems that are affecting so many users should be immediately posted to dreamhoststatus.com so people know that effort is being given to ‘fix things’. Don’t keep us in the dark, Dreamhost!
November 7th, 2006 at 10:50 am
My alternative hosting account, that is £129 a month for a dedicated server, goes down for a few minutes every week, just like DreamHost. I know which one I’d rather be paying for.
November 7th, 2006 at 11:25 am
my sites been lagging, not loading images, getting 503 Service Temporarily Unvailable messages for a week now. And my servers not listed in any of the ones having problems. Even with all that my service up to this point has been wonderful and no problems. One incident is not going to keep me from changing either. Unless of course this is a monthly thing lol.
November 7th, 2006 at 12:33 pm
Silver, seems to work fine for me. Heh, nice favicon BTW.
I’m not sure why so many people keep having issues lately, my sites have been working just fine except for the occasional glitch every few weeks (or random latency spike). I have started monitoring my sites from an external VPS, and even have some nice pretty graphs showing everything is A-OK. The graphs are exported from the VPS via SSH to reside on my actual dreamhost server as static images (or I can view them on the VPS in real time if there is a problem). I should probably have it start graphing load average and memory usage, and make it actually do login attempts to SMTP, IMAP, and POP and graph those results.
November 7th, 2006 at 1:50 pm
I tried Silver (to see the favicon) and while I got the splash page, was not able to enter.
Although my site is accessible again.
November 7th, 2006 at 1:52 pm
Floppy discs? Do people still USE those?!
November 7th, 2006 at 1:55 pm
I’m on fanta.
Slow since day one (signed up less than a month ago). Today it’s terrible.
Load averages on Fanta are through the roof.
Takes 12 seconds just to load the
`ps`command:[fanta]$ time ps -ef >/dev/null
real 0m12.239s
user 0m0.020s
sys 0m0.010s
November 7th, 2006 at 1:55 pm
my site has been down all day
November 7th, 2006 at 2:11 pm
No worries, their whole network just took a dive again. You won’t be bale to notice any of the miriad of other problems.
November 7th, 2006 at 2:18 pm
My sites are down too. (on taffy)
November 7th, 2006 at 2:20 pm
Hello there!
I’m looking for a new host, please tell me if DH is a good option and if these kind of things happens often.
Thanks a lot!
Steph
November 7th, 2006 at 2:35 pm
Steph, you pretty much just have to look through this status site to see that it happens constantly.
But they are cheap. And you get gazzilions of bandwidth (when your server and their network are actually up at the same time and functioning properly).
November 7th, 2006 at 3:12 pm
Ah, well the problem seems solved, but look at homer’s uptime:
14:10:33 up 5 days, 9:57, 6 users, load average: 293.06, 103.59, 39.26
I don’t like the look of that load average, ouch!
November 7th, 2006 at 3:25 pm
Floppy discs? Do people still USE those?!
Heh. Older* Netapp Filer models use floppies, nowadays they all boot off of Compact Flash cards (like the kind you would use for a digital camera or PDA).
They at one point had models that booted off of CD, and carried a CD-ROM drive. Sounds sensible, right?
But the only problem is that sysadmin retards kept sticking all sorts of stuff in that CD-ROM drive (like Windows PC video games- Even though Netapp filers run a totally proprietary Operating System code that’s based off of Unix/BSD) and asking “Why doesn’t it load my computer games???” “How do I install Windows 2000 on this Unix Filer thingy???” etc
* - But the hardware is still good even though it’s older, if it’s like a 700 or 800 series model - They can still run the latest OS kernels, and are pretty slick storage mo-sheens; fast as heck, tons of capacity and otherwise reliable. It’s just that, in certain situations where these models fail and can’t read the OS from disks, you have to boot it up using like 8 floppy disks, which is a pain in the butt. Viva la flashcard!
November 7th, 2006 at 6:25 pm
I’ve had no problems at all from my server, not even when everyone else’s was down. Cupcake is doing quite well, and my HTTP uptime tracker had only 2 downtime reports on the same day. All is well.
Check my HTTP stats- http://www.siteuptime.com/statistics.php?Id=28805&&UserId=28591
-Jared
November 7th, 2006 at 7:04 pm
shoot! our site is down for 30 minutes now. Anyone else on chiclet or demeter having trouble?
November 7th, 2006 at 8:18 pm
Ugg,
I read too much of that before realizing that it possessed all the social trappings of wading in an ocean after the plane has gone down.. ~
; ^ )
If you are working your sites hard enough to warrant 100% downtime control, you should direct all of your email through GMail (pop or forwarded) and your domains through GoDaddy (or an affiliate) and have at least 2 hosts ready to be propagated.
- To propigate the other host, just log into your GoDaddy account (Or the like) and point the dns server to your mirror and sometimes it goes through in just a matter of hours.
These are hard working people though - They’ll get it handled.
Over all Dreamhost is still A+
November 7th, 2006 at 8:58 pm
Steph - What you’re seeing here is as bad as it gets. Know that they have hundreds if not thousands of servers so some can appear to have issues at any time.
Some of us have significantly less trouble than this site represents.
November 7th, 2006 at 9:49 pm
Is this the same outage that affected my sites on compton? It was down for nearly 12 hours and I was curious as to why.
November 7th, 2006 at 10:31 pm
Ok, this is the 3rd time today that my site is down.
What is going on?
November 8th, 2006 at 12:04 am
how many servers are there? how OFTEN they have down time???
November 8th, 2006 at 8:00 am
okay i love dreamhost but now my site is down every now and then… please fix it guys. thanks
November 8th, 2006 at 8:23 am
Hey,
Any idea when things will be fixed?? My Site has been down for 2 days now…..
November 8th, 2006 at 8:30 am
Goodness people…
Bob says: Dreamhost is definitely not a reliable host for anyone running a business. The last time this happened (there was a **THREE DAY OUTAGE** just a few months ago) I swore I was going to switch and got distracted. Now it’s moving up on my to-do list…
Yea, I was there too… I was in the middle of my biggest project EVER during that time, and was averaging 25000 hits a day. yes, 25K hits… I moved the site to ipowerweb duuring that outage… I was lucky that the domain name that was published was not on DH or ipower, as it wa maintained by the co I was working for… little JavaScript… redirected… ALL THAT TO SAY: ipower went down for 2 days, 15 minutes after DH came back up. I was lucky, but there were bigger things last summer than just DH’s issues…
Andy says: Floppy discs? Do people still USE those?!
Heh. Older* Netapp Filer models use floppies, nowadays they all boot off of Compact Flash cards (like the kind you would use for a digital camera or PDA).
Brand new DELL servers still can’t be restored from CD… you HAVE to make a floppy boot disk to read off the CD… Moreover, Symantecs Backup software (c)2006 MUST be restored from floppy… it looks Directly for drive a:. and won’t allow any sort of remapping to a:… Sure the world doesn’t use floppys, but anyone who wants to leave DH because they’re “behind the times” needs to learn a bit about enterprise systems… and I wish them luck to find the greener grass…
I’ve been there, I’ve switched, I’ve gone back. DH has been reliable (I’ve been with them since 03)… and they have actually listened AND RESPONDED to their customers… good luck finding that elsewhere for your 10 bucks a month…
November 8th, 2006 at 9:35 am
okay ive lost an hour’s worth of traffic and it’s still not up… am i the only one experiencing this?
my tracker stopped at 23:00 … it’s already 9 Nov 2006 / 00:03 and still no luck. kindly fix it guys.
http://extremetracking.com/open;ref1?login=mghrulz
November 8th, 2006 at 10:37 am
no, you’re not the only one…mine has been down all morning as well.
November 8th, 2006 at 10:43 am
The fanta server is very low since 5/11 and today is the third reboot, please fix it.
November 8th, 2006 at 10:55 am
It’s still down
November 8th, 2006 at 1:20 pm
I can’t get our site stats. Every time I click on daily stats it says Inactive Report
This report is not currently active.
Go to:
* Main Stats Page
* Monthly Report
* Longterm Report
* Daily Report
When can we expect to have this problem fixed?
November 8th, 2006 at 1:23 pm
If your sites are down then you need to send in a support request and verify there is a network outage. It may a residual issue of the filer server software problem, but posting here only helps you vent as support folks don’t troll in here…and this is not the place to request dreamhost support.
November 8th, 2006 at 5:30 pm
->Catprotector
I believe DH said before that by default, they were going to turn off daily stats, so save resources or something. But if you still want them, then go to control panel - under the ‘Status’ tab - Site Statistics - then for your domain, select ‘customize’ and check the box to turn on daily stats.
November 8th, 2006 at 10:28 pm
How disappointing.. almost two days later and my site is not down as much as it is r.e.a.l.l.y. s.l.o.w. It takes a minute just to load a 70kb jpeg.. How often does this happen? Once a month, twice a month?? Any long-term customer please share your experiences. Thanks.
November 8th, 2006 at 11:54 pm
We’re waiting to see if things improve also.
It is cheap and HAS been good up til the last 2-3 weeks and “seems” to be rapidly declining.
Support team is trying to respond, but we’re still waiting for improvements.
Slow, down, slow, slow and tonight DIRT slow.
November 9th, 2006 at 5:49 am
Thanx for the GMT reference on the update…
November 14th, 2006 at 3:26 pm
RE: “I’m going to another host because you have too much downtime / your servers are too slow, etc”
Please, think about your comments for a moment before you post them. It’s not true that 100% uptime (or very, very close to it) is “impossible”, but it’s certainly ridiculous to ask for something like that at this price point. Implying that you would change web hosts because of the “unreliable” servers is like buying a crappy, used laptop on ebay for $300 and expecting its battery to last a decade without recharging… while you’re playing World of Warcraft 24/7. Insane, right?
Even the “big guys” i.e. Google, etc. have had major outages / are “dirt” slow sometimes, etc. Please note that, I think it was back in ‘01, “part of the Internet” was actually “broken”, not literally of course, but due to a ruptured pipe in the Pacific Ocean, some sites in Asia were unreachable from the west coast for a pretty long time (a day or two, not 5 minutes); bottom line is that these things happen. If you’re “fanatical” about uptime / performance, contact rackspace. But expect to pay up.