File server hardware swap

One of our file servers for the spunky and randy clusters is experiencing watchdog timeouts. This means every couple of hours it is simply freezing up and rebooting itself! As you can imagine, thats not good, so we are performing an emergency hardware swap before the problem gets any worse. Your data, if on this server, will be inaccessible for about 10 minutes starting at 2:15pm.

We’re back online! The file server was swapped out, and the new one was reporting correctable ECC errors on one of the memory modules. We already swapped that out and the file server is back online. All services and servers should recover shortly. If you are still having problems in about 15 minutes, please let our technical support staff know!

.

78 Responses to “File server hardware swap”

Pages: [1] 2 » Show All

  1. 1
    Bob Says:

    DAMN your service has sure been spotty lately. I just had a major email go out too, and within a minute our site went down. Way to make you customers look bad.. I would BET that it won’t be back up within 10 minutes.

    =(*

  2. 2
    Alex Says:

    here we go again

  3. 3
    t0mcioo Says:

    Strange - spunky is mail server, but also 2-3 mins ago bottlecap went down - if will not came back in 10 mins will need to send e-mail to dh

  4. 4
    Corey Says:

    Bottlecap went down too…fwiw.

  5. 5
    Dave Says:

    I’m in the same situation as t0mcioo, spunky is our mailserver and bottlecap is definately down.

  6. 6
    Kelly Says:

    @t0mcioo

    Actually, spunky is a cluster of servers that includes both mail and webservers. We’re already back up but it looks like the filer needs different RAM as well. Everything should be right as rain shortly.

  7. 7
    Alex Says:

    wow, actually did only take ten mins

  8. 8
    Bob Says:

    hmm.. well I am down still (as expected), but our server is none of the above.. hahaha

    They’re going to take me away ha ha, they’re going to take me away, hee hee, to the funny farm!

  9. 9
    NenadS Says:

    Bottlecap is still down…

  10. 10
    Alex Says:

    haha, that would be the curse of speaking too soon! im down again! Wahoo!

  11. 11
    Bob Says:

    And to think my billing comes up in a week. I wonder if I will renew. These guys sure don’t seem to care a bunch about us. Oh well… Maybe I will venture outside or something.. =)

  12. 12
    Alex Says:

    down and up, down and up…

  13. 13
    Brian Says:

    down.. when I try to access my admin for wordpress

  14. 14
    Corey Says:

    Confirmed, bottlecap still down. Just received an automated response from DH saying it’s fine. Unfortunately incorrect.

  15. 15
    Dave Says:

    I can confirm Bottlecap still down. Wonder how long it’ll be before it’s back? (and how long it’ll be up for?)

  16. 16
    Forrest Says:

    bottlecap still down…

  17. 17
    Ronan Says:

    Lets’go change from another host.

    I still tired. Aways down.

    Whe are lost, visits. Lost money. Lost publicty.

    Where is the motive? Aways sites down.

    Dreamhost where is the quality of Host of Dreams?

  18. 18
    Lee Says:

    Ahh, so that explains why all my sites have randomly been going down, seem to be working fine at the moment though :)

  19. 19
    NenadS Says:

    bottlecap is still down…

  20. 20
    Dave Says:

    Status “Resolved: Yes” at top of page. Bottlecap still down :-(

  21. 21
    Bob Says:

    Dreamhost: Service quality you can only *dream* of.

  22. 22
    NenadS Says:

    “Please accept our apologies for the trouble. We are working to correct
    an issue on your server and should have things back to normal
    momentarily. Thanks for your patience.

    Thanks!

    Scott”

    30 minutes ago…

  23. 23
    Ronan Says:

    The problem is. EVERY DAY the sites DOWN.

    If gos one casually troble whe accepted the apologies

    but EVERY DAY???

    Whe are pay for what? For see us website off?

    Whe are don’t whant appologies, whe are quant QUALITY….

  24. 24
    Gene Steinberg Says:

    @Ronan: LIsten, you’re supposed to file support tickets for these things, not just post them in a place where they will do you absolutely no good whatever.

    Are your sites down every day? Well, maybe, but mine aren’t. So maybe you should find out whether it’s confined to your setup or your server.

    Peace,
    Gene

  25. 25
    Ronan Says:

    I send tickets from suport.

    And all i recive is, “The problem goes solucionate”.
    Ok

    And tomorrow? I whant my site up aways. And dont quant wirte one ticket every day.

    I gess the dream host only suport blogs. I dont have one blog, i have my portal whit my site and my forum work.

  26. 26
    Dave Says:

    Bottlecap appears to be back online again.

  27. 27
    Tom Says:

    I dont know what people are saying, hosttracker didnt even report an outage and I’m on spunky. That means outage was less than 15 minutes.

  28. 28
    A customer Says:

    When is dreamhost going to address the GENERAL failure of their hosting service? Your customers deserve an apology and we deserve to know what you’re going to do fix the situation. 2 months of non-stop issues?

    Are you going to fix your infrastructure?

  29. 29
    Dave Says:

    @Tom - less than 15 minutes eh?

    Maybe for “spunky”, but “bottlecap” was down from 1:23 to about 2:38.

    My site (bottlecap is the webserver I’m on) was down for the duration, and I had an email from Joshua @ DH tech support confirming there was an issue with the server at 2:16 and that they were “investigating”

  30. 30
    Tom Says:

    @Dave

    Sorry to hear about your downtime. But this was file clusters, what is your email server, if it says Spunky or Randy, then this affected you, if not, it’s a coincidence and unrelated to this.

  31. 31
    Diogenes Says:

    It looks like that my server (Rootbeer) is rebooting itself too, my sites is down in this moment and I can´t connect to SSH…

  32. 32
    Diogenes Says:

    It´s back online again but to prove my point:

    [rootbeer]$ uptime
    19:55:20 up 15 min, 3 users, load average: 4.59, 5.63, 3.61

    High load average and just 15 min after a reboot.

  33. 33
    Diogenes Says:

    What´s happening ??
    [rootbeer]$ uptime
    20:25:04 up 45 min, 3 users, load average: 76.76, 23.07, 10.59

  34. 34
    Logan Says:

    I wish you guys would give Dreamhost a little bit of a break. If you’d notice, Dreamhost is one of the largest web hosting providers around. Thousands of servers, hundreds of thousands of websites hosted. Don’t you think one or two out of the thousand is okay every now and again?

    And when the major catastrophe stikes, everyone lays the blame on Dreamhost and their “non-exsistance fault tolerance.” I want you to–with your crew of 25, some of who also handles the business and customer service part as well–go manage two huge datacenters [with thousands of servers and hundreds of thousands of websites] perfectly. No failures. No outages. No nothing. Perfect 100% uptime. Can’t do it? Didn’t think so. It’s hard enough to manger a few–like some IT departments. If you’re an IT engineer managing a business’ IT department you’re going to have maybe 10 servers. Multiple your workload by three or four HUNDRED times.

    For those of you who are moving to another host, make sure their status page details outages like Dreamhost. You’re going to be lucking to even get a notification that your site was out, much less a blog where the problems are reported.

    So, think twice before dising Dreamhost. I’m a long time customer–seven years in a couple of months–so I’ve seen it all from them.

    - Logan

  35. 35
    Explorer Says:

    My sites are working fine :-)

    Hope we all can have a nice weekend!

  36. 36
    Roy Says:

    The server I am on is rootbeer also, I get similar problems with the site going offline every 1 - 2 days but its usually only for about 15 minutes or so then everything is back up and running again. I don’t really mind, in the end dreamhosts service is still far better than other companys. You should try godaddys shared hosting, you would be lucky to get on there almost anytime without a timeout, their servers are jammed full of so many sites and the transfer speeds are pathetic. Atleast Dreamhosts downtime is only short, transfer speeds are good, support is excellent and the price is cheap.

    If your site is so important it cant be offline for 15 minutes a day why don’t you change to a dedicated server for your website?

  37. 37
    Drew Meyers Says:

    my site is down!!

  38. 38
    Billy Says:

    my site is down too :(

  39. 39
    Brian Says:

    site is down

  40. 40
    Tommy Says:

    All four of my sites are down! Beinng on GMT timeframe I cannot work out how long ago this all started! Any word on when this will be rectified?

  41. 41
    Bob Says:

    down…

  42. 42
    AdBux.org Says:

    The servers that are affected went down at 9:30 California time… or 12:30 EST time. The ETA is between 11:30 to Midnight. That’s 2-3 hours of downtime. Not good.

  43. 43
    Pat Says:

    down!

  44. 44
    Tommy Says:

    Sites are back. I hoep everyone else experiences the same

  45. 45
    alana post Says:

    databases on all of my sites have been down for almost 2 hours ): right in the middle of finishing a project, too. i’m on burbank. can’t complain too much since i still think dreamhost is the best, just poor timing right now for me.

  46. 46
    Nabeel Says:

    I have to upload the new update to my site and now I’m getting an error saying server not found. What’s the deal??

  47. 47
    Angry Dog Says:

    Site taking a couple of minutes to respond, and no email at all. Evian. grrrrrrr

  48. 48
    Eric Gruber Says:

    I’m officially shopping for a new hosting company.

  49. 49
    netbuilder Says:

    For the last several days my site will not rebuild with movable type. I noticed this the first time I tried to update a page it says the server is timing out - I’m assuming modifying the database. Still having this problem and it is Friday 4/13/07 at 9AM Central.

  50. 50
    Van Says:

    Looks like mysql on barkley is a bit screwed and crawling along. Static content loads fine for my site so the webserver appears ok.

    Support ticker raised.

Pages: [1] 2 » Show All

Leave a Reply