Security Breach
UPDATE: 2007/06/07 6:49PM PDT - We are in the middle a more thorough investigation and some new information has turned up. While we did detect some unauthorized access to our user web control panel, in at least some cases it looks like that may not be to blame for the compromised ftp accounts. In some isolated cases it appears that there may be security problems on end-user computers as well. If you have been affected by this, please do whatever checks on your own computer you can as a precaution. Our investigation is covering all possible attack points and this is one of the possibilities.
Also note that we now have confirmed information that these ftp account hijackings are happening on other web hosts as well and it looks very likely like there’s more to this situation than just the security problem we detected within our own system.
We are now forcing all of the affected users who have not yet changed their passwords to do so before they will be able to upload anything again. This is necessary so we can continue to monitor the situation and see clearly what’s going on.
Original post:
A very small subset of our user accounts have been compromised due to a security flaw in our web control panel software. We have already notified those of you affected directly via email, aside from dedicated server customers who are being notified right now. If you are not on a dedicated server and you have not gotten an email from us your account has not been compromised and is likely safe. It’s still a good idea to change your ftp and web control panel password as a precautionary measure.
The security flaw allowed the attackers to log into our customer web control panel with the access privileges of another user. From our web panel they were able to access individual user password information. The attackers also attempted to gain access to our central database and billing information but were ultimately thwarted in that attempt. No credit card information or customer personal information was obtained.
.
June 8th, 2007 at 10:00 am
eric, no the keylogger is downloaded when you try to access your own domain’s webpage. that’s the whole problem of how dreamhost got hacked and in turn are hacking us. our of sensitive data such as credit card and whatever else we type on our computer, such as if you change the password on your dreamhost account or any other, it will be logged and sent to the hacker, and they have your NEW password.
June 8th, 2007 at 10:04 am
Still nothing in the official blog about this :S.
I haven’t been affected by this *YET* I’ve been diffing daily backups to make sure and have changed all my passwords. I’ve scanned my PC with 3 different antivirus packages and a well known trojan hunter. I have a linux based firewall between me and the net and also run a local xp firewall *comodo* and have seen no popups for outgoing/incoming connections.
I’m doubtful that it is a problem with end users machines.
June 8th, 2007 at 10:05 am
@gramps….dreamhosts servers are all linux…how are they going to have a virus/keylogger?
June 8th, 2007 at 10:07 am
I just scanned with PC-Cillin and found nothing on my Vista desktop machine.
If you’re going to report that you’ve scanned and found something (or didn’t find anything), please mention which program you used to scan!
June 8th, 2007 at 10:22 am
@Jumpy
Virus Scanners = Avast, AVG professional, F-Secure Anti-Virus
Trojan Scanners = Trojan Hunter + Search and Destroy.
@eric
The keylogger doesn’t run on the linux web server, it is the payload which is attached to a web page then using a browser vulnerability it is downloaded and executed on your windows machine.
June 8th, 2007 at 10:26 am
Maybe people shouldn’t be using I.E6!!! I.E7 has been out for ages, and then theres FF, there’s no reason for sloppyness!
And has it occurred to the few, that you may have had the keylogger on there before this whole thing started, and the only reason you’ve found it is because you’ve just checked!
June 8th, 2007 at 10:46 am
So who is Dreamhost’s biggest competitor? That’s who I’ll be sending all of my clients to after this lovely debacle. Anyone?
June 8th, 2007 at 10:48 am
can somebody walk me through the exact steps required to roll back a file from a snapshot? i have 139 hacked index files to roll back to their one-week-ago state. i have no competence at the command line, though i have managed to log in (it had been years). i get about this far:
todd@vega:~$ ls .snapshot
hourly.0 hourly.1 nightly.0 nightly.1 weekly.0 weekly.1
todd@vega:~$ cd weekly.0
-bash: cd: weekly.0: No such file or directory
todd@vega:~$ ls weekly.0
ls: weekly.0: No such file or directory
supposing i want to replace the index.php file at the same level as the .snapshot. what do i type?
June 8th, 2007 at 10:48 am
I’m confused about something and I hope you can help. I can’t see anything wrong with my sites - they are all working fine with no links added that I can see. BUT, traffic to the sites is down by around 60% compared to yesterday. Is it possible that my sites “look” ok but are, in fact, stuffed? If so, how can I tell?
June 8th, 2007 at 10:54 am
@Adam - The last 48 hours have been a bit hit and miss with DH hosted sites. My own site has been unreachable several times in the last 36 hours. It could have something to do with it.
Do a “view source” on your index page (using your browser) and check that there are no unrecognised links or iframes in the source.
June 8th, 2007 at 11:02 am
Thanks Van. I’ve checked the source code and its all looking normal. Funny thing is, I can reach the sites, but the traffic is still well down. In fact, yesterdays traffic wasn’t too bad - its today that I appear to have taken a hammering. Lets just hope this passes quickly.
June 8th, 2007 at 11:20 am
I guess you asked them and that’s what they told you (they wouldn’t lie)–because you didn’t come up with that based on any research. It’s common sense, or in the case of this site, rare sense.
I know you’re stupid, but try this very simple task. Search Google for “CPanel Exploits.” See all those results? Know what CPanel is? It’s the most used panel out there and NOT what Dreamhost uses.
Now, take all of those exploits and match them up with another host’s super honest status site. What? There aren’t any?!?!?! You mean all of those exploits and rooted servers didn’t lead to a single problem for customers?? Amazing!
Idiot. Some of you are way too stupid to comment on anything.
If that’s your idea of research, you’re too stupid to be on the internet.
For the retards that still haven’t figured it out, and certainly never will on their own: Any site recommendation you get here is spam. Yeah, good plan… dump an honest host for one that needs people to spam DH’s sites.
If that’s the type of advice you’re looking for, why don’t you just go to one of the spam sites that was injected into your index pages. You’ll get just as good of a recommendation from the spamming losers there as you will from the spamming losers here.
June 8th, 2007 at 11:58 am
If this type of comment didn’t come from such a qualfied wanker as yourself it might be taken seriously.
Just to clue you in, Einstein, you are arguing against people who are enjoying laughing at you. Keep it up sunshine…
June 8th, 2007 at 12:28 pm
@Gramps&everyone else having problems with keyloggers/spyware; Try running http://www.superantispyware.com and Spybot Search&Destroy: http://www.safer-networking.org - make sure to run an update before you scan though.
June 8th, 2007 at 1:00 pm
And Mike is an asshole apologist. Glad we cleared up who’s what.
But, for the record, my ‘misison-critical’ host doesn’t screw up like this and *gasp* doesn’t use cpanel (or should I say “that piece of shit cpanel”). And I know not because I asked, but because I’ve relied on them for the past 8 years. But that’s not why we or our projects are here at DH is it. We’re not here because we’re expecting “world class” hosting, we’re here because we’re either poor, cheap or desperate. When I can afford to move each project to a more reliable host (with the accompanying higher bandwidth and disk space costs), I will.
But I would still like that long-promised more-informative blog entry that this entry promises… because reading entries about awards when a lot of us are sitting here going “you say I’m not compromised but you also admit the problem is deeper than expected”, well it leaves us more than a bit concrned.
June 8th, 2007 at 1:21 pm
I’ve been hosted with DH for about 4 or 5 months now (paid up for a year). I chose DH after reading lots of golden reviews on the web saying how great DH was in comparison to the rest of the providers.
Well it was bliss for 2 weeks after joining (excellent response times and no downtime) after my honeymoon period things went downhill rapidly. Since Jan/Feb 2007 things have been going wrong sometimes more than once a week for all tiers of DH customers. I, like many others keep thinking that it’s short term and eventually DH will reclaim their great reputaion as a great provider once again. Well! it’s six months in now and things have not improved. Is this over selling? or maybe lazy and complacent staff? I really don’t know but things have definately gone to shit in the last 6 months.
I won’t jump ship as I can’t be bothered to check out other hosts and also my site is a (none profit) hobby site.
But I do feel very sympathy with the users who are trying to run busines sites hosted on DH. If DH are not up to running mission critical sites they should say so in their advertising as the latest problems are even affecting their dedicated hosting clients.
My 2 pence.
June 8th, 2007 at 1:23 pm
Dreamhost doesn’t use Cpanel
there is a rampant russian keylogger loose and it is effecting more than just Dreamhost, although perhaps dreamhost’s transparency, they are the most vocal, and allow their customers to be vocal too.
I for one would rather support resolve the problem, get that iframe off of their customer’s websites to at least disembowel this from all of dreamhosts websites, then take the time to write the blog.
Even though I am on a macintosh, I scanned my computer this morning…JIC
Even though none of my clients webid’s, or ftp’s were effected and none of them got an email, I still went and changed all webid passwords, then ftp passwords. again…JIC
I don’t feel that I am impervious to attacks, nor do I trust implicitly that all of my clients using ftp or the panel (whether mac or pc) are 100% secure, so, I requested that they too run a full scan on their computers.
we live in a world now where stealing/identity theft is the only way some can survive and where the people got the keylogger in the first place may never be determined…
June 8th, 2007 at 1:59 pm
The only thing you’re doing is crying. I’m the one that laughing at how stupid you idiots are.
At least this time, it’s a girl that’s crying like a little girl. That’s a first here.
Pointing out that idiots are idiots has nothing to do with defending Dreamhost. They’d still be idiots if they were hosted somewhere else.
And if you think your other host has never been exploited, you’re not too bright. There’s a BIG difference between something not happening and something being kept quiet, or simply not affecting the server you’re on.
June 8th, 2007 at 2:47 pm
Are you guys serious? IE actually has bugs that allow random web sites to download and install keyloggers onto your machines??? Aside from DreamHost sucking etc etc, you really really reallly need to start using a less horribly broken browser.
June 8th, 2007 at 3:05 pm
The ‘keylogger’ theory is complete bunk. Several users, including myself, use Firefox/Mac exclusively and there are no known invisible firefox/mac trojan horse exploits. Even if there were, the chances that a mass keylogging attack would reveal itself in thousands of dreamhost users using different browsers in different OSes would all be hit simultaneously is completely unrealistic.
A master account at DH was penetrated in some way, and the attacker walked in and collected passwords. Can anyone at DH say this isn’t true?
And on the subject of MITM attacks: I haven’t logged in to my DH panel in weeks until after I received DH’s warning, so there was no panel password to sniff. Let’s clear out the FUD and get the facts. DH, got any updates?
June 8th, 2007 at 3:26 pm
Kevin I got an update for you: your website looks like a piece of shit, no wonder you’re on dreamhost hahaha
June 8th, 2007 at 3:32 pm
@Kevinisafag You are very good evidence for the need of a moderation system here. Off-topic and Childish (-5)
June 8th, 2007 at 3:40 pm
At Maggie: Have a clue… they’re free. DH doesn’t use CPanel either. You’d know that if you 1: did your research and 2: actually listened to everyone else in these comments who said the same bloody thing. Oh, and the first time your mission critical websites go tits up and you don’t get any information without screaming about it, don’t come back to DH. You don’t *like* being notified when fuck-ups happen.
June 8th, 2007 at 9:10 pm
Kevin Fox
don’t believe yourself!!! http://www.symantec.com/enterprise/security_response/writeup.jsp?docid=2007-052712-1531-99
keyloggers are not only for ie6 and below!!!
June 8th, 2007 at 9:15 pm
and, in case you can’t follow all of the links
http://www.symantec.com/enterprise/security_response/weblog/2007/05/mpack_packed_full_of_badness.html
June 8th, 2007 at 9:20 pm
Do you blame a deer for being shot by a hunter? Yes there are some apparent security holes, but the blame for this belongs firmly to the fucking hacker cunt that perpetrated this, yet there is hardly anyone on this entire thread other than myself that has said anything about those who are actually responsible for this shit. Fair weather friends.
June 8th, 2007 at 11:25 pm
so this explains the hacked index page of my site
there is no way for my computer to contain keylogger kindda program, I’m using genuine weekly updated windows xp sp2 and have antivirus installed and do a regular checks for spams etc.. and I have s/w and hardware firewalls installed
things in my website were just an index.html file in root and couple other non-executable file (no php, perl ertc..) in other folders
and When I saw the hacked index page of my website believe me I couldn’t stop laughing
DH it’s good to inform users to keep their computers secure but you know maybe you should do the same
June 9th, 2007 at 1:08 am
You’re joking, right? I thought so.
June 9th, 2007 at 1:38 am
Eric, if you are using windoze you DO have a virus and you WILL have a system that has backdoors open. Thats just the windoze way. Its crap, its insecure, and it will let you down.
June 9th, 2007 at 2:28 am
Ubuntu FTW!
June 9th, 2007 at 5:42 am
If you have a firewall that blocks all outgoing connections, I think you’re pretty safe from keyloggers, even using Windows.
June 9th, 2007 at 7:22 am
If I have to read one more word from Josh Jones, I’m gonna puke!
It’s good that he has a nice little blog to express his creative urges. But why the Hell do I have to read it to hear the results of Dreamhost’s last fuckup?
And where is that report, anyway?
June 9th, 2007 at 7:49 am
@ Ryon / Josh : what blog? Where - i want to read it
June 9th, 2007 at 7:52 am
btw. I’m trying to edit some mail accounts and right now Panel access is slooower than chilled honey
June 9th, 2007 at 12:44 pm
@Amar: http://www.dreamhoststatus.com/2007/06/09/major-ddos/
June 9th, 2007 at 5:39 pm
Something is still stinky guys.
I received a notice about my account on the 6th and immediately changed the password.
(I should note that I have NEVER accessed using ftp either before or after - I ALWAYS use ssh with a dsa key to access.)
There has been further ftp access since the time I changed my password.
Obviously whatever is insecure with your panel - conitnues to be insecure.
PHT
June 9th, 2007 at 5:49 pm
FYI: - I tried opening a ticket - but the panel is borking when I try and open one - it spins - saying ‘creating ticket.
I was able to email support - by replying to the original breach notice.
PHT
June 9th, 2007 at 6:26 pm
and still no updates about it here or on the blog..
June 9th, 2007 at 6:33 pm
I’ve found another user (who also would DEFINITELY not use FTP) - that has had further ftp access subsequent to his password change on the 6th.
Very bad -
June 9th, 2007 at 7:06 pm
Amar: It’s referenced at the top of this thread. Right between “Security Breach” and “Check there for further information.”
Heinrick: That apears to be a different, um, anomaly.
Joshua Juran: You can’t be Josh Jones. There were no family pictures, and not even one chapter about yourself.
(Where is that report, anyway?)
June 9th, 2007 at 7:23 pm
@Peter Hope-Tindall:
> Something is still stinky guys.
> I received a notice about my account on the 6th and immediately changed the password.
> (I should note that I have NEVER accessed using ftp either before or after - I ALWAYS use ssh with a dsa key to access.)
> There has been further ftp access since the time I changed my password.
> Obviously whatever is insecure with your panel - conitnues to be insecure.
Yup ditto here. The logging on continued on Friday even after passwords were changed. I shut off ftp as soon as I saw the post with the new option. We’ll see if that stops it.
June 9th, 2007 at 7:52 pm
The point is though - if someone has a way of getting the password from an insecure database/panel.
The password can be used to access using ssh.
While this will log in the lastlog - it won’t for remote commands.
This is bad.
June 9th, 2007 at 9:41 pm
My website was hacked as well. For what it is worth I have been on vacation for two weeks. I did not log into the panel, ftp, ssh, for that entire time. If my password were sniffed it would have to have been before May 27th.
June 9th, 2007 at 11:45 pm
@Adam & James
It’s interesting to see how linux newbies tell windows is crap, more than 90 percent of people around the world use MS products including businesses, if you have a good firewall no matter what you use, you will know what comes in and goes out. not to mention only small amount of these MS users get hacked because they dont use antivirus or firewall.
in addition how comes everyone got hacked by the so called keylogger on the same day? they probably planned to to send the trojan to our computers then waited a long time to see hey now 100 computers are infected lets start, HAVE SOME LOGIC
June 10th, 2007 at 12:07 am
dreamhost moved me to a new server, saying that its a better place
but the new place is down a lot
website is slow like hell
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
June 10th, 2007 at 3:57 am
This happened to my websites back in April. I noticed in on April 18.
On the 17nth, I got a delisting notice from google saying that my pages violated
their guidelines and would be delisted.
I looked, and sure enough there was a bunch of content on my pages that I
did not create.
I logged in, and voila there were a few directories I did not create.
I talked to my dreamhost and they said that I had been ftp’d into
when I had not done so.
The hackers installed a number of directories onto my website, with names
like reviews, temp, misc
These had subdirectories of about 80 folders, each with keyword php files
numbering usually 1012 with words like anal-lesbian-fisting.php.
I first changed the name of the folder so that any php scripts would not
work, and then changed the permission, so that only I could access it, —
then I have been deleting these files.
Also, all .html files had appended at the bottom an invisible iframe, just
before the closing body comment. Index files of the domains had added
html that added a set of advertisements to the bottom (and sometimes the
top) of the file. All of these were replaced from my backups, and/or
their html visually scanned line by line looking for stuff I didn’t put
in.
There were in a couple of instances php files installed in the
directories– I removed those pronto– I don’t use php for anything.
I’ve mostly recovered from that, changed all my passwords, etc. I’m still not where I was in the listings on one keyword that’s bugging me, but most of my websites are back where they were.
June 10th, 2007 at 4:26 am
@Peter Hope-Tindall: Did you change your control panel password?
June 10th, 2007 at 8:06 am
Henrik - yes - I changed both ftp and control panel passwords
June 10th, 2007 at 9:09 am
Did you contact abuse regarding this?
June 10th, 2007 at 11:58 am
I’m in exactly the same boat as Peter above (”Something is still stinky guys.”). I only use SSH+key. After changing my passwords (control panel and account), more connections come through via http://FTP. I use strong, unique passwords. I run OS X. ClamAV says I’m clean. I run the latest version of Firefox and every other application.
DH has a big problem.