Security Breach
UPDATE: 2007/06/07 6:49PM PDT - We are in the middle a more thorough investigation and some new information has turned up. While we did detect some unauthorized access to our user web control panel, in at least some cases it looks like that may not be to blame for the compromised ftp accounts. In some isolated cases it appears that there may be security problems on end-user computers as well. If you have been affected by this, please do whatever checks on your own computer you can as a precaution. Our investigation is covering all possible attack points and this is one of the possibilities.
Also note that we now have confirmed information that these ftp account hijackings are happening on other web hosts as well and it looks very likely like there’s more to this situation than just the security problem we detected within our own system.
We are now forcing all of the affected users who have not yet changed their passwords to do so before they will be able to upload anything again. This is necessary so we can continue to monitor the situation and see clearly what’s going on.
Original post:
A very small subset of our user accounts have been compromised due to a security flaw in our web control panel software. We have already notified those of you affected directly via email, aside from dedicated server customers who are being notified right now. If you are not on a dedicated server and you have not gotten an email from us your account has not been compromised and is likely safe. It’s still a good idea to change your ftp and web control panel password as a precautionary measure.
The security flaw allowed the attackers to log into our customer web control panel with the access privileges of another user. From our web panel they were able to access individual user password information. The attackers also attempted to gain access to our central database and billing information but were ultimately thwarted in that attempt. No credit card information or customer personal information was obtained.
.
June 6th, 2007 at 4:13 pm
Why the hell are you storing FTP passwords in plain text in the first place?
June 6th, 2007 at 4:16 pm
Nobody said they were.
they said “access individual user password information”
June 6th, 2007 at 4:19 pm
@gavin
Because most users don’t, and they tend to forget their passwords…..
June 6th, 2007 at 4:20 pm
I’ve always ignored mass downtimes, slowness of download speeds, general low perfomance and high page generation speeds, but now with this, I think it’s enough. Already moved onto another host which I paid a lot more for and I’m satisfied.
Thanks for the cheap service I enjoyed, but there have been too many problems and I really can’t complain considering I got it for a really low price (coupon), but at the end it came to what’s been proven million times - you get what you pay.
Bye.
June 6th, 2007 at 4:20 pm
So if they had access to the panel, mail passwords are compromised as well.
Nice - that’ll be fun to change all of those.
June 6th, 2007 at 4:24 pm
That is a really bad explanation ‘free iCal hosting’. Only because some people can’t remember their passwords or type them on a piece of paper and keep it in their wallet or something doesen’t mean that DH should store passwords in plain files. And yes, they do that. If you ever played with their panel, you’d notice that for whatever type of service you’ve set your password, you could simply read it afterwards - where most other hosting companies handle it on a way smarter and secure way: hash passwords, if user forgets it, let him change it and update blog/forum/whatever configuratio files. Dreamhost-style definitely isn’t the way to go.
June 6th, 2007 at 4:24 pm
One more thing. Is DH still emailing out passwords for accounts suchs sFTP upon creation or password change. Maybe that should stop.
June 6th, 2007 at 4:35 pm
Even things like cpanel and plesk have security flaws like this… a quick search of google will confirm this.
http://www.google.com/search?q=security+advisory+cpanel
I’m not happy that my account was potentially hacked into, but as with any web software these things always pop up no matter how careful you are.
This is the first security issue i’ve heard of since being with DH (signed up Aug 2002…) so i’m not overly worried about this matter. Annoyed, but I’m more worried about people hacking via my websites forum or installed apps.
June 6th, 2007 at 4:35 pm
My accounts were hacked for a second time about 5 hours ago, having first been hacked on the 24th. Obviously the problem still hasn’t been addressed. The hack attempts are all using FTP (which I no longer use), and are coming from the same IP addresses that everyone else is reporting.
June 6th, 2007 at 4:36 pm
ste, I agree. There’s a suggestion dated 2004-07-01 that says “Don’t ever send user (chosen) passwords via plaintext email.” It’s almost inconceivable to me that this isn’t considered a higher priority, but please go vote for that suggestion if you haven’t already.
June 6th, 2007 at 4:39 pm
Too little, too late,
June 6th, 2007 at 4:52 pm
Simon. If you not longer use FTP have you disabled all your FTP accounts or convert them to sFTP/SSH? What account are they signing is with?
June 6th, 2007 at 5:01 pm
I highly recommend that all DreamHost users change their passwords and really check out their web sites to make sure no one inserted these spam links. There are reports that people who did not get notified from DreamHost also got their accounts hacked. It appears that the 3,500 accounts are a low estimate.
I just did a complete backup of my web site, deleted everything, and reloaded WordPress from a freshly downloaded copy. I am then going to make sure all of my plugins are up-to-date and my theme was not modified. Also changed all of my passwords (dreamhost panel, ftp, e-mail, mysql, etc.).
June 6th, 2007 at 5:02 pm
ste: if you have an ssh account, it automatically gets ftp access.
There is no current way, that I know of, to disable this.
Even in the admin interface, it says (Shell account - allows FTP plus ssh/telnet access.).
*sigh*
June 6th, 2007 at 5:05 pm
What is most interesting that a hack attack was reported to me by my client last Friday (5 days ago). I reported this to DH. They told me that our FTP password was compromised. The password was “very random”, but I accepted the possibility that maybe the client had stored password on thier computer and a trojan or a virus had nicked it. So I recovered the modified pages from backups.
Now one of my other clients got notification and his sites are down. I checked the source and judging by the footprint left by the hacker it was the same exploit. What they did is simply append, or sometimes overwrite links at the bottom of the page. All inserts begin with comment code. First client had links pointing to some zoo porn and another one to some gallery site which was already down when I looked at it.
June 6th, 2007 at 5:05 pm
ste, you do use FTP, if you ever upload a file to your site, or change it, you use FTP. FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server. Even web-based file uploaders use FTP, so…you’re wrong.
And DH, I didn’t get an e-mail, so I assume I haven’t been hacked, but my site suddenly went down, it worked one minute, then I restarted my computer and got back on about 10 minutes after it restarted (so about an 11 minute break) and the site wouldn’t load. Web based FTP works, but I can’t do windows FTP and I can’t get onto any of the domains or sub domains.
June 6th, 2007 at 5:19 pm
I don’t think anything has been exploited more than CPanel.
While this is not a good thing, I doubt any panel out there has had fewer holes poked in it than DH’s.
June 6th, 2007 at 5:20 pm
@ HaLo2FrEeEk
Chill. Then read: SSH file transfer protocol. While it is technically FTP it’s secure, as in encrypted, and i assume that is what ste meant.
>
June 6th, 2007 at 5:23 pm
@HaLo2FrEeEk
you do use FTP, if you ever upload a file to your site, or change it, you use FTP. FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server.
No, that is almost entirely incorrect. Yes, that is what “FTP” stands for, but there are other ways to upload files to Dreamhost - rsync, sftp and scp (all different protocols running over ssh). Disabling FTP entirely would be an excellent idea - any legacy (usually Windows) software that doesn’t support those protocols really really really needs to be fixed.
Note that while disabling FTP is a good idea in general (storing and sending plaintext passwords over the Internet is just asking for trouble), it doesn’t appear it would have helped in this case.
June 6th, 2007 at 5:23 pm
@HaLo2FrEeEk
>FTP stands for File Transfer Protocol and it is the only protocol you can use to transfer an entire file to a server.
Rubbish. SFTP is encrypted. There are other flavors of encrypted file transfer as well (FTP with SSL, etc.)
June 6th, 2007 at 5:24 pm
By “our web panel,” are you referring to the panel in general, or an admin level panel that you guys use? (Kinda like what WHM is to CPanel?)
I read that part it like it’s saying the person had the same access as an employee… or am I misunderstanding?
June 6th, 2007 at 5:26 pm
If anyone’s using an FTP client that doesn’t support SFTP, Filezilla is a nice choice–and it’s free.
http://filezilla.sourceforge.net
June 6th, 2007 at 5:43 pm
This is scary.
June 6th, 2007 at 5:50 pm
OK, I’m suspicious that this incident is as minor as DH is claiming based on the description. They contacted me and said 6 of my user accounts had been affected, but none of those accounts could log into the webpanel, they were ftp/ssh only accounts. The only webpanel login that could access them that I’ve given permission to would be… me. So that leaves only my login or one of DH’s own as the culprit.
The thing is, if they exploited this flaw to gain access to the panel with my login privileges then they could see the information for all 200+ user accounts I have. While they apparently haven’t logged in to any of the others yet, that doesn’t mean they don’t have the info and can’t.
Basically, I think if you’ve had one account affected by this then all your accounts are at risk. Changing your Web Panel login would be a good idea, but you probably need to change the password info for every user account you have as well. I could be wrong, but as currently described it sounds like this is one very, very major security breach and DH isn’t doing us any favors by playing down the severity of it. Not to mention that they’re probably better equipped to help us get all these hundreds of passwords changed in short order than the web panel will allow us to.
Hopefully the blog post will clear things up one way or the other, I know it took me most of the day changing passwords and verifying that my users files were unmodified (and fixing the single modified file I did find) and that was just for 6 user accounts. I have no clue how long it’d take me to reset the passwords for every single user I have.
June 6th, 2007 at 6:22 pm
So perhaps DH needs to allow the ability to disable FTP access for accounts that are set up for sFTP or SSH/Shell. This way someone doesn’t get in using the unsecured channel. Of course that just plugs up one hole. If someone has the password, there’s really no stopping them. So I would reiterate that I think DH should stop automatically sending out passwords in clear text via email.
Plus, I’d hope that the DB of passwords is encrypted as well as the Credit Cards in the case that those DH DBs are compromised. In fact, I believe it’s part of the Terms of Service for doing business with Visa/MC/Amex/etc. that all CC numbers are encrypted in all forms of electronic storage. In addition the CVV2 code can never be stored encrypted or not. The only exception is during point of transmission to processor/gateway.
June 6th, 2007 at 6:33 pm
@ste
>So I would reiterate that I think DH should stop automatically sending out passwords in clear text via email.
Sure, that would be a good idea but has nothing to to with this exploit. If the intruder has access to your panel, all bets are off. Picking off the odd email with a plain text password in it out of the sea of emails is a low percentage activity. Nowhere near the payoff of an exploit that gets you access to the panel either.
I second the suggestion of not exposing the current passwords in the panel. If someone forgets a password let them change it only. If that policy was in place this exploit would have been exposed much earlier as the attackers would have had to have changed passwords rather than just retrieve them. That would have alerted users very quickly that something was amiss.
June 6th, 2007 at 6:38 pm
Since it is the password database that has been compromised, I think it is possible that people can SSH into their account, and use “passwd” to manually change their password. Then it would be different from the password stored in DreamHost’s database.
Disabling FTP and using only SFTP/SSH is certainly not going to help here, as the attacker can still grab your password from the compromised database.
June 6th, 2007 at 6:44 pm
would all of you chill the hell down. They said on dedicated servers, or been notified by e-mail. Have any of you been notified by e-mail? If not i suggest you stop wining
June 6th, 2007 at 6:59 pm
I received the email, so stfu.
June 6th, 2007 at 7:13 pm
I suggested this in another post. DH needs to allow us to DISABLE FTP. I ONLY use SSH to access our files, but I know of others who have used FTP from time to time. It’s time to stop sending anything over insecure protocols. It’s time for DH to stop letting people send anything over insecure protocols.
June 6th, 2007 at 7:15 pm
And why the hell was FTP made into a link? It should point to open ssh dot org. I’d link to it myself, but I got spam flagged.
June 6th, 2007 at 7:39 pm
how ever my credit card was banned from the bank and they said coz “due to insecure transaction to a web hosting company in america.” I only doing business with DH. so I think this incident have a connection to that. how ever no money was obtained though. But I had to apply for a new card.
June 6th, 2007 at 8:10 pm
At least announce which servers have been compromised. I didn’t receive any email but I’m still freakin worried.
This security breach is bad.. and together with all the downtimes and network problems… not looking so good.
June 6th, 2007 at 8:22 pm
I see three issues here, two of which I’m appalled at:
(1) There is/was a security hole in the panel, which allowed unauthorized access — this can happen, and hopefully they patched it ASAP. As long as they were keeping things updated, that’s a tragedy, but it’s a tough Internet out there. Which is why…
(2) You should never store passwords as plaintext — it’s appalling that an attacker was able to actually recover passwords after gaining access. Passwords stored in a database or elsewhere on disk should be hashed so as not to be recoverable. Totally ridiculous.
(3) Why are only hearing about this now, and why in such poor fashion? The rumors have been floating around for at least a day, and your emails clearly didn’t get to everyone who was compromised. Likewise, this message is extremely vague — what types of passwords were affected? Were email, database, panel accounts all harvested, or just FTP?
This is a DH pattern — get caught with your pants down, then downplay the severity of it and blame the users if they complain. We’ve seen it with the outages, and now this.
June 6th, 2007 at 8:33 pm
I did email support to find out when they thought the issue first started, so I could verify current files against backups from before that time-frame, and they said it probably started around 10 days ago (which would be 11 days now I guess).
Not sure how long it was until they discovered it though. I didn’t ask.
June 6th, 2007 at 8:36 pm
As bad as it is, this is nothing compared to what CPanel hosts constantly have to deal with.
A day? I guess they should have done it like this:
Step 1: Let the whole world know you have a security hole.
Step 2: Let the affected customers know.
Step 3: Fix it.
Doesn’t seem like the right order to me.
I’d also add that regardless of how the passwords are stored, once they access your panel, you are at their mercy. Can’t see a user’s password? Don’t need it to delete the user. Delete some domains, email accounts, etc…
Considering the access that was gained, the damage was minimal.
Again, not good… but I can’t help but think the people that are blowing this up have never heard of CPanel.
In reference to the vagueness, that’s a good way to keep things until you know it’s all over. They’ve stated there would be a bigger post on the subject later.
What passwords were affected? Why not change all of them, just to be safe?
And where did DH blame the users? They said it was a security breach due to a flaw in their panel.
June 6th, 2007 at 10:25 pm
Mike, you’re misleading on a few points so I figured I should clarify:
Well, the whole world already knew. We found out through non-DH blogs. Customers tend to get a little angry when they have to find out that their web host has been hacked through other blogs on the internet. DreamHost’s status blog was supposedly set up to keep customers up to date on any kind of problem and the status of that problem. But when it takes several days for them to finally give an official acknowledgment, customers get a little nervous. What if something happens to our account? Will we also have to wait a few days to get any official word and instead get our information through outside blogs? Thanks, but no thanks.
Also, the steps don’t have to happen independent of one another. You know, it’s actually possible for a medium-sized company to do many things at once, like notify affected customers, start fixing it, and posting a status note to notify customers that they should be on the lookout for nefarious HTML and to change their passwords.
So because it’s CPanel’s fault, it completely absolves DH of any responsibility for securing their servers?
Supposedly it’s mentioned here or in the comments here: http://mezzoblue.com/archives/2007/06/05/unsettling/ but I’m too lazy to read the entire thing right now. So you may be right that DH didn’t *explicitly* blame the users.
June 6th, 2007 at 10:53 pm
Hi,
Mine is one of the accounts affected (I got the email this morning, and found tonight a bunch of files with invisible IFRAMEs inserted this afternoon, and mailbox full of complaints from my messageboard users). I don’t know the full extent of the possible attack symptoms, since this is my only account.
In this case, it appears that the attacker (or most likely script) has iterated through all the FTP directories…any file with a name containing *index* (regardless of extension or the rest of the name, e.g. index.htm index.php index.old index.zzz___ messageIndex.file etc.) was either 0 bytes or replaced with a file of 130 to 132 bytes, which contains in part the following (HTML tags removed of course) :
iframe src=’(http) m-gallery (dot) org/images/111/index.php’ width=1 height=1 style=’visibility: hidden;’
The compromised files were in some cases buried deep within directories I’d long ago forgotten existed, several dozens of hacked files in all. If you got The Email, your files may be compromised similarly (or even entirely differently), and your passwords NEED CHANGING ASAP because this @#$%er is still at it.
After you change ALL your users’ passwords and Web Panel password(?), here is a way to help catch any unauthorized changes to your files. Make sure your user has shell access, then login to your Dreamhost shell account (see the instructions at http://wiki.dreamhost.com/Shell ). Type the following command to produce a full listing of all your files and directories with datestamps:
ls -a -c -lt -h -R > /yoursite.com/dirlist.txt
where /yoursite.com/ is any directory you can access by FTP.
(The explanation of this command is: ls (directory listing), [a]ll files, show last-modified time and sort by it, human-readable file sizes, and [R]ecurse subdirectories, and direct the output to file /yoursite.com/dirlist.txt.)
Now you can download the file and search for the most recently modified files (all mine were modified this afternoon, so searching for the string “Jun 6″ turned them all up). You can also search for e.g. ‘index’ to identify other likely compromised files.
HTH
Tim / Drmn4ea
PS. If you have multiple ftp users, doublecheck that you have changed ALL their passwords and did not miss any! I changed the obvious ‘FTP’ user showing in the web panel (turned out not a real user, actually a remnant from some older DH system), but missed the other one, which actually hosted all my files! Late this night I noticed my old FTP password still working when investigating the complaints about my site not loading properly, and slapped my forehead.
PPS. This means the attacking IP(s) were still not blocked, half a day after DH concedes they were aware of the problem…
June 6th, 2007 at 11:18 pm
I have 12 shell/ssh accounts with DH, and 11 of them were compromised (yep, got 11 identical emails from DH telling me so).
What’s even more distressing to me than this particular incident is the lackadaisical attitude DH has about security. This week’s incident is just an example of the consequences.
Case in point: For years now when DH has to move a server or swap out hardware they simply generate a new SSH key for that server. This means the next time a user tries to log in to that server they get a stern warning from their SSH client that the key has changed, and that this may be evidence of an attack. This is because this is exactly what a ‘man in the middle’ attack looks like (the second most common kind of SSH password recovery attack). When I’ve contacted support to complain they just say ‘oh yeah, we swapped out a server’ or worse, simply ‘don’t worry about it. It’s normal. Go and delete the key from your hosts file and it’ll work fine.’
This fosters exactly the kind of complacence and uncertainty in users that makes SSH’s formidable security worthless. It’s no different than installing a big heavy door for your home, but opening it whenever someone knocks.
The plaintext password is just another instance of this careless attitude. Even worse, it’s a security risk for legitimate uses. If I run a company with 100 employees and give them all shell accounts, I give them an initial password with instructions to change it immediately. It’s extremely bad form for I-the-employer to be able to see those passwords, because it’s simply a fact that many of those passwords ill also happen to be the passwords they use for their personal email, online banking, or ATM PIN. Just because they work for me or are on my server doesn’t give me the right to — or them the expectation that I am able to — see their passwords.
If a password is lost then a new one can be generated and distributed. If a password is maliciously reset, then awareness of that intrusion comes much sooner when the rightful user’s credentials fail. believe it or not this is a Good Thing because it prevents spying and helps accelerate damage control.
To keep me and my clients as customers Dreamhost needs to demonstrate that they take data privacy and security seriously, and not just gloss this over with an ‘oops there was this bug, but it’s fixed and we’re all right now’. I get enough of that from Microsoft.
Sincerely,
Kevin Fox
DH customer since 2001
June 6th, 2007 at 11:21 pm
I don’t know about you guys, but when I create a user with SFTP-only access then they can’t login with FTP. Unfortunately, if you need shell access then DH gives FTP access automatically, which makes no sense to me …
Also, make sure you don’t confuse SFTP with FTP+SSL, because SFTP is a completely different protocol and is in no way related to FTP. And for that matter, SFTP is not tunneling FTP through SSH … it’s a completely different protocol from FTP. One of the account options DH gives is the SFTP user which, to my knowledge, does not allow FTP access (FTP access doesn’t work for me at least).
June 6th, 2007 at 11:32 pm
For the paranoid who think they should have gotten an email but didn’t, to see if you’re effected:
1. SSH (telnet *only* if you don’t have a choice, and only if you’re an idiot) into your account(s). If your accounts are all on the same server you need only SSH into one of them.
2.Type last [username], replacing username with your actual shell username, of course, and without the brackets.
3. It’ll show you a list of logins, both via FTP and via telnet/SSH. If you see a login on a date you don’t remember logging in on, from an IP address you don’t recognise, someone probably gained access to your account.
Of note, on some servers (Scipio for one), the file it pulls that information from is apparently reset at the first of the month, if not weekly. But at least you can rule out any *recent* access violations. From the same login, you can perform a last command on any username you have set up through the control panel, if and *only* if they’re on the same server. If the whole idea of trying to figure out what your IP address is is a little beyond you, then apply this thought process. If you didn’t log into your account via FTP in the last week, then FTP connections in the last week to that account are probably from people who shouldn’t have access. At which point, start looking for a backup of your files, and change your password. Which you should be doing anyway but now you have a reason to.
June 6th, 2007 at 11:43 pm
Can’t even check to see if I was or not, all my sites are down(again). This just goes from bad to worse. I already moved my production sites off of DH, all that was left was all of my parked domains. Not those are all offline, unreachable (again). I just want to grab my stuff and move it to a real host (what DH wants to be when it grows up and I can’t even log in now.
Friggin joke company run by kids on drugs.
June 6th, 2007 at 11:59 pm
Yep, I have the same. All my sites are down. Email is not workin. So I do not even know if I have been comprimised. There is no email in my support history, so hopefully all is OK, besides that everythin is down.
June 7th, 2007 at 12:03 am
My problem is my webpanel doesn’t show any of the domains or users or any relevant information. Its as if the account has been reset so I can’t change any FTP settings, etc.
Any ideas what I can do?
June 7th, 2007 at 12:17 am
My site has been breached as well. I would like to change the passwords but when I log into the webpanel none of the domain show up and there are no users listed. It’s as if all webpanel info got erased.
Any suggestions to this problem?
June 7th, 2007 at 12:23 am
Here’s a useful shell command (i.e. log in to your domain via SSH to run it) to show you files modified within the past N number of days:
find ~/mydomain.com/ -mtime -3
Replace the 3 with a 1, or a 2, etc, to search within a different number of days.
June 7th, 2007 at 12:28 am
My sites have been down for hours. No response to trouble tickets, no status page update, nothing! DH has my money, and I have nothing. I am beginning to despise this company.
June 7th, 2007 at 12:44 am
My sites are down too. The server reponds to ping though.
Spunky & York. What are yours?
June 7th, 2007 at 12:47 am
Guys don’t panick and agree so much here. DH staf are I am sure workin very hard now and overtime to solve all the issues. Yes my sites are also down and email also. Be patient and all will be fine.
Go out for a cup of coffee or a beer and tomorrow is a new day
June 7th, 2007 at 1:01 am
Supong. Easier said than done. Having your account compromised is like having your wallet hanging open for all to go through and take what they like. Sure, many will be able to restore they’re account data from backups if they did them but with their accounts compromised any sensitive data they may have is likely in the hands of the folks who pulled this off.
This is a major faux pas that will likely have Dreamhost customers leaving in droves. Unfreaking believable in this day and age.