We don’t believe, as others do, that web host’s have a responsibility to clean up their customer’s website if it’s are hacked, unless it was caused by something on the host’s end. They do have responsibility to properly handle the security of their own systems, which as of just a few weeks ago Dreamhost didn’t do in a basic way. We also think they have a responsibility to not be giving bad security advice to their customers or the public at large, as Dreamhost does. It does a serious disservice to their customers, including the potential that customer’s hacked websites could remain vulnerable to being re-hacked because of their bad advice. The bad security advice, which certainly isn’t limited to web hosts, also makes it harder for people that deal with security issues, and take security seriously, to do their job. We have to spend to much time explaining to clients that the information they are seeing and being told is in most cases wrong. What makes Dreamhost worth focusing on is that in addition to the bad advice they give, they don’t bother following the good advice they give others.
In this post will focus mostly on comments they made in a recent post on their blog and some communication between them and one of their customers, whose website we recently cleaned.
From the blog post:
Based on the sites we have cleaned up already, these attacks have almost universally been due to insecure website software running on the site in question. You could have the best passwords in the world, but if the apps you’ve installed on your server have any security vulnerabilities or aren’t kept up to date, attackers can still find their way in.
Based on the websites we clean up and other data we collect the claim that attacks are “almost universally been due to insecure website software running on the site in question” is completely false. Vulnerabilities in web software are one of the top two things, but the other, compromised FTP credentials, has nothing to do with the software running on the website. From what we see, which one of issues is the biggest changes from time to time. For example, we might see a spike in web software attacks after a new vulnerability is discovered. There are other things that are lesser but not insignificant causes of attacks, like security vulnerabilities at web hosts.
What could explain the discrepancy between what we and Dreamhost see? It could be that they have systems in place that are effective at preventing other types of hacks. It could be that they only look for certain types of hacks, so they only find what they are looking for. Or it could be that don’t actual bother looking at the cause of attacks at all.
Determining if Outdated Web Software is the Cause of a Hack
Recently we were cleaning up a hacked website hosted at Dreamhost. From reviewing the information we had available it didn’t look the hack came through a security vulnerability in web software. Unfortunately, we couldn’t review the necessary logs to be able to determine how the hack occurred because of poor log retention at Dreamhost.
To determine if the website was hacked due to a vulnerability in web software, we would have reviewed the log of HTTP activity. That show all the requests made to the website, so it would show any exploit of software running on the website. Unfortunately, those are only stored for a very short time at Dreamhost. If Dreamhost actually wanted to determine that websites were in fact being hacked by vulnerabilities in web software they would also need to look at that as well. Except for cases where the hack is found immediately, how could they make a strong determination that the hack was due to a vulnerability in web software? The answer seems to be they don’t even try, as will get to in a little bit.
The other thing we wanted to look at was the log of FTP activity. If this is available for the time period of the hack it is easy to see whether the hack came through FTP by reviewing the log. Our client was told that the logs are only stored for two weeks and because the hack occurred before then the log wasn’t helpful. The short retention of the FTP log also means that Dreamhost couldn’t check if the hack came through FTP if it occurred more than two weeks before, which would limit their ability to know that hacks are not coming through FTP as they seem to think is the case. This also makes another claim in the post seem questionable. In regards to a recent hack of Dreamhost they stated that “An extensive investigation has revealed that no customer FTP or SSH user accounts have been maliciously accessed due to this password breach.” If you don’t keep logs longer than two weeks how could you claim to do an extensive investigation, as you can’t look at data older than two weeks once you discovered the issue?
While we couldn’t make a strong determination on the source of the hack, Dreamhost didn’t have a problem doing that. They said:
Also, “hacks” usually occur due to outdated web software, and since you’re running joomla, it’s safe to say, the hack occurred there.
There hasn’t been the kind of vulnerability in Joomla that has led to the software being hacked in a long time. That isn’t to say that something couldn’t be found, but if somebody is telling you that a hack likely occurred from that or that Joomla is not secure then they don’t know what they are talking about. From the hacked websites that we see that are running Joomla, the issues that lead to hacks are vulnerable extensions, compromised FTP credentials, a hack of the hosting provider, and recently we have had a number of clients hit by a dictionary attack campaign that is targeting Joomla websites. It clearly isn’t safe to say the hack occurred because they were running Joomla, as Dreamhost, unless they have done a thorough investigation and found evidence to back that up.
Let’s say you did believe that the cause of a hack was due to a Joomla vulnerability, there would be more work to do after that. You would need to determine if the vulnerability has been fixed in Joomla before you tell somebody “simply reinstall your software, and make sure you’re always keeping it up to date”, as Dreamhost did, because doing that would have no impact if the vulnerability hasn’t been fixed in Joomla.
If the hack is on a website was running the latest version of Joomla it would mean that the vulnerability hasn’t been fixed or more likely that it wasn’t a vulnerability in Joomla.
For a website running an older version you would need to compare what information you had about the hack to the vulnerabilities that have been fixed in subsequent versions to see if the probable vulnerability has been fixed. For example, there was recently fixed vulnerability that could make it easier for some to guess a user’s password after causing it to be reset. To check for that you would ask the client if the password stopped working recently, which would have happened if it was reset. If the logs were available, you could also check those for additional information. If you don’t find a corresponding vulnerability it would mean that the vulnerability hasn’t been fixed or more likely that it wasn’t a vulnerability in Joomla that caused the hack.
Dreamhost didn’t inquire into what version of Joomla was running when the website was hacked or doing anything else, that is another good indication that don’t know what they are doing or they just don’t care. If this customer interaction is any indication of how they handle this type of situation for other customers, it doesn’t appear they do any checking at all and instead just assume a hack must have been caused by software on the website.
Failure to Warn About Vulnerable Software
When it comes to dealing with outdated web software that is actually vulnerable to being exploited, and has been exploited, Dreamhost fails their customers. Dreamhost has tool that scans a customer’s files to find the web software in the account and warn if it is out of date. As of a couple days ago it was listing OpenX 2.8.7 as being “Up-to-date.”, despite the fact that OpenX 2.8.8 was released back in November. OpenX 2.8.7 isn’t just out of date, it contains a serious security vulnerability that was being exploited even before version 2.8.8 was released. You have wonder how they wouldn’t be aware of this if they were actually determining how websites were being hacked or just keeping track of security issues in web software.
Dreamhost Runs Outdated Web Software
In another post by the same author they recommended “upgrading website software as soon as there is an update available”, which is actually good advice. It just isn’t advice that Dreamhost follows.
If you visit the Dreamhost blog with our Meta Generator Version Check extension (available for Firefox and Chrome) you would get an alert the blog is running an outdated version of WordPress, 3.2.1. That version is not that far out of date, it’s only one major version out of date and two months out of date. (Also, contrary to recent claims by Websense there are not known vulnerabilities in 3.2.1.) But considering that Dreamhost is telling people how important it is to keep software up to date and they don’t do it themselves then it seems like Dreamhost either does not care about the security of their website or they don’t believe what they are saying.
What make this more ridiculous is that the person that wrote both of those posts, who is described as Dreamhost’s Web Application and Security administrator, will be giving a talk titled “Security 101″ at a WordPress event. Should you be telling other people about WordPress security if you don’t even bother to take the basic step of keeping WordPress up to date?
It’s not just that blog that is running outdated software, the Dreamhost Wiki is running an outdated version of MediaWiki, 1.16.5. Support ended for that major release, 1.16.x, two and half months ago. Like the WordPress installation that is not far out of date, but for whatever reason Dreamhost doesn’t feel it is important to keep the software running on their website up to date.