Babysitting Bugsy
Posted (October 28th, 2008 at 9:17 pm PST) by Jason CIt would appear that the server “bugsy” had a few issues that required the loving attention of a member of our admin team. As it stands, the server is going thru a deep file system check that requires some close monitoring due to possible intervention. Once it is done with that, everything should hopefully come back up and return to normal. If that doesn’t happen, we’ll be sure to post an update here.
UPDATE: We have been working on Bugsy throughout the evening and we are at a point where we are working on repairing the disks. If that does not work, we’ll restore from backups and then repair the disks. Unfortunately, we have no ETA on this process, but we will update here as soon as we know more.
UPDATE 10/29 3:10p Pacific: Unfortunately we still do not have a solid ETA on a resolution for bugsy. We are however attacking the issue with both solutions listed above. We have a new bugsy up and we are copying data from our backups to this server. The new bugsy is the live server so you may notice that your files are missing. Do not fear, they are being copied from backups. We are also still trying to repair the disks on old bugsy. If that is successful then old bugsy will go back live. So we’re basically in a situation where the first solution to finish will be the one we go with.
Update 10/30 11:00a Pacific: The data being copied from backups to the new Bugsy is roughly 50% done. This unfortunately means that we are likely looking at more than 24 hours for the moves to be complete. As your files are moved your sites will be working, but we unfortunately do not have a way to prioritize the moves by account at this time. The disk repair on old Bugsy was not 100% successful unfortunately. The plan is to let the copy of the backups finish, then run another copy of data on old Bugsy to the new one. The backups we have are unfortunately 2 weeks old, so after we get that data up we will run the second copy in order to get your most up to date files. Sorry about the problems this is causing you. A free month of hosting will be added for customers on the Bugsy server.
Update 10/31 3:30p Pacific: The data moves from the 2 week old backups to the new server have finished. We are now starting the data move from old Bugsy to the new Bugsy. Those of you mentioning you out like to opt-out on the second restore have been set to skip. We are also not going to overwrite any files currently on the server if they have a newer date than the ones copying over. So your sites should be up now. Sorry about the problems this has caused you.
October 28th, 2008 at 9:23 pm
In the tests, it says this server has been under a consistent heavy load. I wonder roughly how many accounts are hosted on it.
October 28th, 2008 at 10:08 pm
I first reported this issue late last night/early this morning to no avail.
October 28th, 2008 at 10:24 pm
It was working for my sites until about 20:30 pacific, so we’re about two hours into the outage now.
October 28th, 2008 at 10:43 pm
I had my site moved off of its previous server last night due to large amounts of outages in the last several months. I’m not real happy that it is down again for the second time in two days after it has been moved.
October 28th, 2008 at 10:54 pm
All of my sites are experiencing this problem at the moment. Hope a fix comes soon!
October 29th, 2008 at 1:03 am
Geez, things haven’t been right with my site for 2 days now. I can’t run a business with all these outages.
October 29th, 2008 at 2:53 am
What’s the SLA on Bugsy?
October 29th, 2008 at 4:04 am
In Spanish Please
October 29th, 2008 at 4:22 am
Ok.. so I just went to a big event to spread the word on my work… and the site if offline?
I really hope this points to some compensation in the future.. and of course this wont make me lose any svn commits for the last days.
….
October 29th, 2008 at 4:33 am
I think I’m going to survive.
All of you with worries about losing money and business need to worry less.
If you’re worried about future downtime and you set up big events you cannot miss…get off of this host. Downtime happens when you pay less.
October 29th, 2008 at 5:45 am
Domains works again but they are empty. What happen with the websites?
October 29th, 2008 at 6:15 am
Where are my files and my database??? What happen with my directories??? I need it!!!!
October 29th, 2008 at 6:34 am
I guess this is a risk with shared hosting.
October 29th, 2008 at 6:46 am
Awesome…why didnt they notify the affected account holders instead of having to find out they are working it ? Hostmonster is looking better and better.
October 29th, 2008 at 7:33 am
Surely they can update us on expected times - that’s all I need to know - that would help me plan a little better. Right now I am refreshing my sites in the hope they will come up so I can continue my work. How long does it take to restore a backup?
October 29th, 2008 at 7:38 am
Sent them a ticket but did not even get an answer.
What is wrong? How long will it take?
Just hope they have a good backup so i have not lost any data.
October 29th, 2008 at 7:57 am
Its been quite a long time guys. No servers set aside as backup?
October 29th, 2008 at 8:08 am
I’m waiting for any reaction from dreamhost for 5h already. My clients are hunting me
2 tickets with no response. I’ve asked whether they are recovering the date. Now I’m sending a third one. All I need is a latest backup and I’m going to migrate to media temple 
I was very satisfied with DH but this is unacceptable. They should move us on other box already.
October 29th, 2008 at 8:48 am
At least make the mysql available =\
i have my own backups for such cases…
October 29th, 2008 at 8:49 am
Curious why this isn’t posted on the status feed: http://feeds.dreamhoststatus.com/?q=bugsy+bennett+homie
Or am I just missing it?
October 29th, 2008 at 9:07 am
Hey…this is the third time in so many months that I’ve not been able to get email for MANY hours. Clearly there is a problem that needs a longer term fix by Dreamhost. I’m guessing that many of us, as much as we like Dreamhost, are considering going elsewhere for service. After all a host’s primary deliverable to its customers is dependability. What is being done?????
October 29th, 2008 at 9:11 am
com on guys… no time to wait for a scandisk… put the backups back on on other machines….
October 29th, 2008 at 9:23 am
What is going on….where is the communication. My site has been down all day and I dont even know if there is backup!! I am still waiting for a response from support arter 3hrs!!!
October 29th, 2008 at 9:27 am
Same here: website down, zero response from supoort, losing money and hoping there’s no lost files…
October 29th, 2008 at 9:32 am
CC dispute/chargeback time ?
October 29th, 2008 at 9:52 am
“Mike Says:
All of you with worries about losing money and business need to worry less.
If you’re worried about future downtime and you set up big events you cannot miss…get off of this host. Downtime happens when you pay less.”
Personally I’d be willing to pay DreamHost more, if they could get whatever they needed to make less downtime. Trying to find another Green/Carbon Neutral/Employee Owned host with good backend isn’t easy.
Com’on DH let’s get you what you need! We will all benefit, and it sounds like you have some customers who would be willing to pay extra cash per month for reliability.
October 29th, 2008 at 10:05 am
MY FILES WEB LOST.
FTP LOST FILES
WHERE IS THE FILES???
October 29th, 2008 at 10:06 am
What about montecarlo, complete down … Thank you DH for ruining my business, it is 30+ hours i can not locate any of my servers, you are ruining my business, 18000 unique visitors was sent to fucking 404 page, MOVING …
October 29th, 2008 at 10:15 am
NOT the right place, BUT………
eldorado is DOWN!
October 29th, 2008 at 10:17 am
Site down.
LeSabre.
Not happy again. Surpass here we come.
October 29th, 2008 at 10:29 am
I’ve been having troubles on Frisky (no site). I’ve posted a few comments at the bottom of the last Frisky-related status report: http://www.dreamhoststatus.com/2008/10/27/not-feeling-so-frisky/#comment-219927
Anyone else with frisky problems?
October 29th, 2008 at 10:30 am
@WebDesing: My files were gone for a few minutes, too. They’re back, but site still down: http://www.potentialgames.com/blog/
October 29th, 2008 at 11:21 am
Should have stuck with my other provider… DH came highly recommended and now all of my sites have been down for what appears to be more then 13 hours. If it seems too good to be true it’s probably not true.
Does this type of thing happen at DH often?
October 29th, 2008 at 11:31 am
@james This is the first time I’ve had more than a 3 hour outage. Still a MUCH better track record than any of my previous hosts.
October 29th, 2008 at 11:49 am
The “Mail Dir” directory is back, but not the content of the other directories.
October 29th, 2008 at 11:54 am
I am a new webmaster and i previously employed a developer to set everything up so this is the first time I have actually logged into dreamhost as a result of my site being down. I am very concerned since i cannot find any of my website files in my account, and im worried all my data is lost. The lack of support has been very poor. After this episode i will be changing host and also looking for alternative backup options, can anyone suggest the best way to independantly back up your data??
October 29th, 2008 at 12:09 pm
@chris bunton: I keep local and NAS copies of all my hosted stuff synched using GoodSync. It periodically (you choose the period) automatically does comparisons with what is out on an FTP server (or network drive or machine folder…)
It makes restoring things a snap.
October 29th, 2008 at 12:09 pm
Man, this sucks, i hope my files are back up soon, it’s 6 domains suffering from this. Every host i’ve been with, and there’s many, have had problems with up-time, it’s part of the game on a consumer level. But no host, until now, have lost all my files. Put em back up please!
October 29th, 2008 at 12:18 pm
whats the likelyhood of these files being lost altogether?
October 29th, 2008 at 12:20 pm
ps thanks Ben. Also what is your page showing…Index?
October 29th, 2008 at 12:23 pm
Joans, I have the same problem, but i have a graffiti-gallery with ~1725 * 3 pictures! I don’t want to lose them! I have copied only 1344 * 3 of them to PC. Hope the rest is all right! And many of my visitors wrote me e-mails about that problem. But i can just sit, wait and click F5…
October 29th, 2008 at 12:26 pm
@chris bunton: Yeah, I’m showing /index. They have all the files backed up. Since it’s wide spread, it’ll probably take quite a while to get everything restored.
October 29th, 2008 at 1:01 pm
Note the “Comments posted here may not be viewed by DreamHost staff at all. Please note that this is not a way to contact DreamHost.” under the “Leave a Reply”.
Contact DH using the “Contact Suport” link (https://panel.dreamhost.com/index.cgi?tree=support.msg&) in your Adm Panel. That´s the right way.
October 29th, 2008 at 1:15 pm
@melo
That seems to have the same effect or get as much attention as this page.
October 29th, 2008 at 1:37 pm
Yeah, but at least it´s “official”…
October 29th, 2008 at 1:48 pm
Hey Melo, how did you manage to put that “Bad server” msg in your website?
October 29th, 2008 at 1:51 pm
To Alexandre:
Aproveitei o diretório em branco e coloquei um index.html pra não aparecer o “Parent Directory”
To all the rest:
Just put a “index.html” in the empty directory. The “Parent Directory” message is too little info for my visitors…
October 29th, 2008 at 1:57 pm
@chris button: Over the year-and-a-half I’ve been on DH, I’ve had several “brief” (under an hour) outages, but I have never ever lost any data. My site seems to be back (but still slow).
October 29th, 2008 at 1:59 pm
Lucky you, Nyhm.
Slow is better than none.
October 29th, 2008 at 2:07 pm
Valeu, Melo! Abraco!