Testing security software is a challenging task and it’s tempting to take clever shortcuts. However, while doing so might save the tester time and other resources, it doesn’t always produce useful results. And if the results aren’t accurate then the test becomes less valuable to you when you’re choosing which product to use.
Can anti-malware stop hacking?
We are big supporters of the idea of full product testing. This means installing the security product the way it was intended to be used, on systems commonly used in the real world and ensuring that every component of that product has a chance to defend the system.
In practice this means that we installed the anti-malware products tested in this report on regular PCs that are connected to a simple network that has unfiltered internet access. We visit malicious websites directly, where possible, and use a special replay system when the bad guys start to interfere with our activities.
Since the beginning of this year we started including targeted attacks in our testing. These types of attacks try to compromise the target using infected documents and browser exploits. Once an exploit has succeeded we then continue ‘hacking’ the target. This step is crucial because in many cases it is these post-exploitation hacking activities that can trigger an alert.
Full product testing doesn’t just mean turning on (or leaving enabled) all of a product’s features. It also means running a full attack as realistically as possible. Testers should not make assumptions about how a product works. You need to act like a real bad guy to understand how these products protect the system. Can anti-malware stop hacking? Test like a hacker and find out. And read our results!
Why does software seem so insecure? Massive software companies seem incapable of fixing their products for any length of time. Is it their fault, or are they fighting a battle they can’t win?
At its core, Windows is the result of several decades of constant development. Despite this, Microsoft is still obliged to observe Patch Tuesday each month, when users receive the latest fixes to installed products. A large number of these updates fix security vulnerabilities.
Patching for security
This month, for example, Patch Tuesday includes 16 security update bundles covering in excess of 40 new security holes found in products as diverse as IIS, Microsoft’s web server, and the supposedly secure Edge browser. How can it be that this monthly ritual is still required? After all, it’s not like Microsoft is a small company caught out by sudden success, while trying to manage a huge ball of undocumented code. On the contrary, it is literally one of the biggest, best funded tech companies in the history of the planet.
Let’s take another case. Adobe’s Flash and Reader products are also mature, stable software. Yet black hat hackers love them as the gifts that keep on giving. This week brought news of yet another critical Flash vulnerability, which is already being exploited in the wild.
Complexity breeds bugs
The complexity of some software, despite its maturity, makes it vulnerable. It needs to be all things to all (wo)men at all times. In the case of Adobe Reader, every PDF document it loads must display perfectly, regardless of its complexity or the limitations of the software used to create it. Anything not explicitly forbidden is, therefore, permissible. Reader will always try to render the file you give it.
Such complexity leads neatly to a fundamental question: If companies such as Adobe and Microsoft can’t find all the exploitable bugs in their code, how come private researchers and black hats can?
Fuzzing for answers
The answer lies in a technique called fuzz testing, or fuzzing.
In his presentation to CodenomiCON 2010 , Charlie Miller showed that with a little thought, a few lines of Python code and some time, it’s possible to use fuzzing, in the form of completely random mutations to a file, to find a number of hitherto unreported and potentially exploitable crashes in Adobe Reader.
He took 80,000 PDFs from the internet and reduced that total to just over 1,500 based on their uniqueness from each other. From these files, he generated 3 million variations containing random mutations.
When loaded into Reader, these corrupted files caused crashes in over 2,500 cases. Miller showed that several crashes revealed exploitable situations, some of which were subsequently found, reported and patched by Adobe, but others were new.
Given that there are a total of 2 ^ NUMBER_OF_BITS theoretical mutations that can be made to a PDF, and the ease with which each mutation can be automatically evaluated, PDF readers alone should remain a goldmine for new exploits for some time. Meanwhile, there are many other programs and file types that can be also attacked with various fuzzing methods.
Tracking flaws
Bugtraq has been highlighting software vulnerabilities for years
Take a look at the Bugtraq mailing list archive and you’ll see what I mean. Every day brings a new crop of reports and proofs of concept for all kinds of software. In fact, another six were added while I wrote this blog post. Buried amongst the plethora of obscure libraries and applications are often complete howlers in major products. How are these bugs being found? In the case of closed source software, fuzzing techniques can be the primary tools.
Fuzzing comes in many forms, with some methods and frameworks being more intelligent and guided than others, but the aim is always to automate the discovery of exploitable bugs by finding situations for which complex software either hasn’t been tested or cannot be tested.
Death by fuzzing
You may be wondering why, with their wealth and resources, major software manufacturers don’t fuzz their products to death, as well as performing more traditional testing. The short answer is that they do, but due to the sheer number of possibilities and the time required, all they can do is fuzz as much as possible before the release deadline. The overwhelming majority of possible tests may still remain to be run by other, potentially malicious individuals and groups.
Security holes in software are not going away any time soon, so ensure that the security software you run is capable of protecting you. How? Checking out good anti-malware reviews that include exploit attacks such as ours would be a good start.
I am proud to announce the first public reports from SE Labs, a new security testing company that tests a whole range of security products, from the sort of anti-malware program you run on your home PC to complex combinations of enterprise endpoint agents and appliances. We’re calling it Next-Generation Security Testing.
The new website will be live in the next day or so, after we’ve ironed out what I hope will be the last few wrinkles. (Update: 12/05/2016 – the website is live now).
Since January 2016 we’ve been testing endpoint security products by exposing them to live web threats and targeted attacks. The results are very interesting and will probably cause some controversy.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkPrivacy policy
All posts
