All postsAnd how can you tell?
If you are in charge of protecting an organisation, you need good data to help make buying decisions. The consequences of simply trusting internet reviews, vendor sales pitches and instinct are extremely serious. So which security product tests are the best?
Testing computer security products and services comes with its own unique challenges and it is hard to assess the assessments. The industry is not known for its transparency in product effectiveness, and that extends to some testing.
Keeping security product tests (very) real
To test a security product properly, you have to behave like a real attacker. There are countless clever ways to simulate attacks, automate testing and so on, but at the end of the day nothing beats sitting down and manually hacking away at a target for realism.
But there’s no point in testing if the threats aren’t relevant – if they don’t relate to real-world situations. A case in point is applying software updates. Everyone knows they should keep their systems patched and up to date, and if they did then attackers would not still be using old exploits, because they wouldn’t work most of the time. But the reality is there’s always some server somewhere that, for a dozen different reasons, gets missed.
What you really want to know is, will your security pick it up?
Intentionally vulnerable
As it happens, last month, we got hacked. Or rather the network we’d set up to test was hacked and, you guessed it, we deliberately included an unpatched but well-known vulnerability. To us, this was a good result, as it proves our tests are as real and as relevant as they come.
I’m also pleased to say that the EDR product that we were testing at the time picked it up, albeit after the initial breach occurred.
Stamp of quality?
Until recently, there was no official way in which to demonstrate that tests of anti-malware and other security products were, in fact, even basically trustworthy. And that the tests were carried out in a way that could be validated and even repeated if necessary.
The Anti-Malware Standards Organization is a community of over 60 security and testing companies from around the world. In mid-2018 it approved and adopted the AMTSO Testing Protocol Standard. A test that complies to this Standard must demonstrate that the testing has been conducted fairly and transparently. In a nutshell, it means that you say what you’re going to do. Do it! Then be prepared to prove it.
Full disclosure: I am on the board of AMTSO, so it’s no surprise that SE Labs was the first testing lab to engage with the Standard. We ran both private and public pilots. We then complied with the Standard immediately, once it was official. You can read more about it in our annual report.
But AMTSO isn’t the only standard you should look out for. Alongside ISO standards (we comply with ISO 27001 and 9001), there are also others such as NetSecOPEN‘s to take into consideration. For example, this Standard helps to specify network traffic when testing performance levels to check if appliances such as firewalls or routers still operate correctly when under stress.
Transparency, transparency, transparency!
With so many published computer security product tests available on the internet sometimes it’s hard to know which ones to trust. However, a simple check of the transparency in its methodology, and the standards adhered to, should go a long way to proving its validity. In turn, this should increase your confidence in the results and help with that next massive IT security purchase.
