How hard should a security test be?

Thank you for opening this report. I hope you’ll be able to use it to get a better idea about which anti-malware products you might want to buy (or get rid of!)

Our reports help you choose the best enterprise anti-malware product that can protect you from ransomware and other types of attack.

Enterprise Anti-Virus Testing

The report starts off with a list of products, each of which win impressive-looking awards. But have you considered what those awards mean? How come there aren’t any massive losers in the list? How hard is this security test anyway?

Baseline Testing

There are lots of ways you can test products. You could prod a teddy bear and say, “well, that looks good enough,” or you could take it to pieces and analyse every component forensically for build and functional quality. “This toy looks safe, its parts are large, soft and non-toxic, and we can’t burn it easily. Plus, it’s got big, cute eyes.” This could be a baseline for cuddly toys: SAFE, with cuteness as an extra bonus.

For anti-malware products we have to consider a few different things, including the following:

• Is it really an anti-malware product? Is it at least basically functional?
• Can it determine a good quantity of common malware, without blocking lots of useful software?
• Can it stop the malware, as well as simply detecting it?

Everyone tells you that you need it, but which one?

Which enterprise anti-virus is the best? Our reports help you choose the most appropriate enterprise anti-malware product that can protect your organisation from ransomware and other types of attack.

Choose the best enterprise anti-malware solution

Classic cybersecurity advice always includes a plea to, “install anti-virus” or “use endpoint protection software”. Journalists, bloggers and even governments hand this information out, as if it helps. Most platforms, including Microsoft Windows and Apple macOS, include anti-virus so the question then becomes, “which enterprise anti-virus?”

How do you choose?

The UK’s National Cyber Security Centre (NCSC) provides some mature and detailed advice but stops short of helping readers work out which products might be most suitable. The only time it tries to help in this respect ends in a bizarre suggestion that you might prefer a product that implements the Anti-Malware Scan Interface (AMSI). This feature is only relevant if you are developing security software yourself.

The Cybersecurity and Infrastructure Security Agency (CISA) in the US gives advice on stopping ransomware. At the bottom of the list, including useful items such as, “update and patch” and “keep your personal information safe” is the instruction to, “install antivirus software, firewalls and email filters.” Nowhere does any such organisation help you choose which is the best or most appropriate for your organisation.

When you search for, “best business anti-virus” or, worse, “best home anti-virus” you’ll see millions of links to better or lesser-known magazine websites and slightly shady reseller’s blogs. Some of the most respectable technical websites run sensible and unbiased reviews and there’s where you start to get somewhere: opinions on interfaces and prices. But how effective are these products?

Find the best reviews

The best reviewers delegate the really technical business of testing endpoint security to the professional testers. If you read their reviews you’ll see our name in there somewhere.

This is because organisations such as SE Labs spend all of their time testing security products – it’s what we specialise in. In our case we learn how the criminals behave and then copy them closely. This produces the most realistic results you can hope to see in a public security test. We also ensure that our reports are reviewed by the Anti-Malware Testing Standards Organization (AMTSO) to validate that we’ve tested fairly.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Everyone tells you that you need it, but which one?

Which home anti-virus is best? Our reports help you choose the home anti-malware product that can protect you from ransomware and other types of attack.

Choose the best home anti-malware solution

Classic cybersecurity advice always includes a plea to, “install anti-virus” or “use endpoint protection software”. Journalists, bloggers and even governments hand this information out, as if it helps. Most platforms, including Microsoft Windows and Apple macOS, include anti-virus so the question then becomes, “which anti-virus?”

How do you choose?

The UK’s National Cyber Security Centre (NCSC) provides some mature and detailed advice but stops short of helping readers work out which products might be most suitable. The only time it tries to help in this respect ends in a bizarre suggestion that you might prefer a product that implements the Anti-Malware Scan Interface (AMSI). This feature is only relevant if you are developing security software yourself.

The Cybersecurity and Infrastructure Security Agency (CISA) in the US gives advice on stopping ransomware. At the bottom of the list, including useful items such as, “update and patch” and “keep your personal information safe” is the instruction to, “install antivirus software, firewalls and email filters.” Nowhere does any such organisation help you choose which is the best or most appropriate for your organisation.

When you search for, “best home anti-virus” you’ll see millions of links to better or lesser-known magazine websites and slightly shady reseller’s blogs. Some of the most respectable technical websites run sensible and unbiased reviews and there’s where you start to get somewhere: opinions on interfaces and prices. But how effective are these products?

Find the best reviews

The best reviewers delegate the really technical business of testing endpoint security to the professional testers. If you read their reviews you’ll see our name in there somewhere.

This is because organisations such as SE Labs spend all of their time testing security products – it’s what we specialise in. In our case we learn how the criminals behave and then copy them closely. This produces the most realistic results you can hope to see in a public security test. We also ensure that our reports are reviewed by the Anti-Malware Testing Standards Organization (AMTSO) to validate that we’ve tested fairly.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Read this SE Labs assessment of world-leading endpoint security products and discover how they handle well-known threats and targeted attacks.

Everyone tells you that you need it, but which one?

Which small business anti-virus is the best? Our reports help you choose the most appropriate small business anti-malware product that can protect you from ransomware and other types of attack.

Choose the best small business anti-malware solution

Classic cybersecurity advice always includes a plea to, “install anti-virus” or “use endpoint protection software”. Journalists, bloggers and even governments hand this information out, as if it helps. Most platforms, including Microsoft Windows and Apple macOS, include anti-virus so the question then becomes, “which anti-virus?”

How do you choose?

The UK’s National Cyber Security Centre (NCSC) provides some mature and detailed advice but stops short of helping readers work out which products might be most suitable. The only time it tries to help in this respect ends in a bizarre suggestion that you might prefer a product that implements the Anti-Malware Scan Interface (AMSI). This feature is only relevant if you are developing security software yourself.

The Cybersecurity and Infrastructure Security Agency (CISA) in the US gives advice on stopping ransomware. At the bottom of the list, including useful items such as, “update and patch” and “keep your personal information safe” is the instruction to, “install antivirus software, firewalls and email filters.” Nowhere does any such organisation help you choose which is the best or most appropriate for your organisation.

When you search for, “best small business anti-virus” or, worse, “best home anti-virus” you’ll see millions of links to better or lesser-known magazine websites and slightly shady reseller’s blogs. Some of the most respectable technical websites run sensible and unbiased reviews and there’s where you start to get somewhere: opinions on interfaces and prices. But how effective are these products?

Find the best reviews

The best reviewers delegate the really technical business of testing endpoint security to the professional testers. If you read their reviews you’ll see our name in there somewhere.

This is because organisations such as SE Labs spend all of their time testing security products – it’s what we specialise in. In our case we learn how the criminals behave and then copy them closely. This produces the most realistic results you can hope to see in a public security test. We also ensure that our reports are reviewed by the Anti-Malware Testing Standards Organization (AMTSO) to validate that we’ve tested fairly.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Choose the best enterprise endpoint security solution

Welcome to the first edition of the Enterprise Advanced Security test that compares different endpoint security products directly. We look at how they handle the major threats that face all businesses, from the Global 100, down to medium enterprises. And most likely small businesses, too.

We give an overall score but also dig down into the details that your security team will care about. This report explains the different levels of coverage that these products provide.

An Endpoint Detection and Response (EDR) product is more than anti-virus, which is why it requires advanced testing. This means testers must behave like real attackers, following every step of
an attack.

While it’s tempting to save time by taking shortcuts, a tester must go through an entire attack to truly understand the capabilities of EDR security products.

Full attack chain testing

Each step of the attack must be realistic too. You can’t just make up what you think bad guys are doing and hope you’re right. This is why SE Labs tracks cybercriminal behaviour and builds tests based on how bad guys try to compromise victims.

The cybersecurity industry is familiar with the concept of the ‘attack chain’, which is the combination of those attack steps.

Fortunately the MITRE organisation has documented each step with its ATT&CK framework. While this doesn’t give an exact blueprint for realistic attacks, it does present a general structure that testers, security vendors and customers (you!) can use to run tests and understand test results.

The Enterprise Advanced Security tests that SE Labs runs are based on real attackers’ behaviour. This means we can present how we run those attacks using a MITRE ATT&CK-style format.

Endpoint Detection Compared

You can see how ATT&CK lists out the details of each attack, and how we represent the way we tested, in Appendix A: Threat Intelligence, starting on page 15. This brings two main advantages: you can have confidence that the way we test is realistic and relevant, and you’re probably already familiar with this way of illustrating cyber attacks.

Choose the best enterprise security product

By understanding the rules of security testing

Our reports help you choose the best enterprise security product that can protect you from ransomware and other types of attack.

Choose the best enterprise anti-malware solution

This report contains security testing results. You can compare the performance of a variety of products that claim to protect you against online threats. This, in theory, will help people and businesses choose the best security product.

But this is a free report. How can you trust that the high-scoring vendors didn’t just pay for their ranking? Do you suspect that some low-scoring vendors dropped out of the report? Or asked to be retested until they scored better?

What are the rules behind the scenes in security testing?

With security testing the stakes are high. From a customers’ perspective, the wrong decision could be disastrous to a business. Or a personal life.

So we, as testers, have a massive responsibility to do the right thing, meaning the honest thing. That means trying to involve as many reputable security vendors as possible in our tests and treating them all fairly.

Security vendors want to sell products and will do what they can to achieved strong marketing. That can involve appearing in weak tests or engaging with more ‘flexible’ testers. One strategy could be to test enough privately against competitors and then release the one report that shows your product at the top of the list.

We focus on strong technical testing and avoid purely marketing-led initiatives. We have awards for vendors who do well, but we stand out by assessing technology deeply and helping improve things for everyone.

Five simple rules

In our blog post Public and Private Testing we explain our five simple rules to help maintain the integrity of our reports. If you want to peak behind the curtain, to see how we work with security vendors, the information is all available online.

Testing Standards

For this report we also followed the only available Standard for anti-malware testing, the one run by the Anti-Malware Testing Standards Organization. This ensures that we do what we say we’ll do, and can prove it.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Choose the best small business security product

By understanding the rules of security testing

Our reports help you choose the best small business security product that can protect you from ransomware and other types of attack.

This report contains security testing results. You can choose the best small business security product by comparing the performance of a variety of products that claim to protect you against online threats. This, in theory, will help people and businesses choose the best security product.

But this is a free report. How can you trust that the high-scoring vendors didn’t just pay for their ranking? Do you suspect that some low-scoring vendors dropped out of the report? Or asked to be retested until they scored better?

What are the rules behind the scenes in security testing?

With security testing the stakes are high. From a customers’ perspective, the wrong decision could be disastrous to a business. Or a personal life.

So we, as testers, have a massive responsibility to do the right thing, meaning the honest thing. That means trying to involve as many reputable security vendors as possible in our tests and treating them all fairly.

Security vendors want to sell products and will do what they can to achieved strong marketing. That can involve appearing in weak tests or engaging with more ‘flexible’ testers. One strategy could be to test enough privately against competitors and then release the one report that shows your product at the top of the list.

We focus on strong technical testing and avoid purely marketing-led initiatives. We have awards for vendors who do well, but we stand out by assessing technology deeply and helping improve things for everyone.

Five simple rules

In our blog post Public and Private Testing we explain our five simple rules to help maintain the integrity of our reports. If you want to peak behind the curtain, to see how we work with security vendors, the information is all available online.

Testing Standards

For this report we also followed the only available Standard for anti-malware testing, the one run by the Anti-Malware Testing Standards Organization. This ensures that we do what we say we’ll do, and can prove it.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Choose the best home security product

By understanding the rules of security testing

Our reports help you choose the best home security product that can protect you from ransomware and other types of attack.

Choose the best home anti-malware solution

This report contains security testing results. You can compare the performance of a variety of products that claim to protect you against online threats. This, in theory, will help people choose the best security product.

But this is a free report. How can you trust that the high-scoring vendors didn’t just pay for their ranking? Do you suspect that some low-scoring vendors dropped out of the report? Or asked to be retested until they scored better?

What are the rules behind the scenes in security testing?

With security testing the stakes are high. From a customers’ perspective, the wrong decision could be disastrous to a personal life.

So we, as testers, have a massive responsibility to do the right thing. That means the honest thing. We need to involve as many reputable security vendors as possible in our tests. And treat them all fairly.

Security vendors want to sell products and will do what they can to achieved strong marketing. That can involve appearing in weak tests or engaging with more ‘flexible’ testers. One strategy could be to test enough privately against competitors and then release the one report that shows your product at the top of the list.

We focus on strong technical testing and avoid purely marketing-led initiatives. We have awards for vendors who do well, but we stand out by assessing technology deeply and helping improve things for everyone.

Five simple rules

In our blog post Public and Private Testing we explain our five simple rules to help maintain the integrity of our reports. If you want to peak behind the curtain, to see how we work with security vendors, the information is all available online.

Testing Standards

For this report we also followed the only available Standard for anti-malware testing, the one run by the Anti-Malware Testing Standards Organization. This ensures that we do what we say we’ll do, and can prove it.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Zero to Neo

Our reports help you choose the best home anti-malware product that can protect you from ransomware and other types of attack

Choose the best home anti-malware solution

There seems to be no limit to the powers of cyber criminals. In 2021 the public became aware of the advanced capabilities of the NSO group, now infamous for helping governments spy on dissidents and others.

The SolarWinds attack compromised some of the largest organisations in the world (and my implication, their customers – and so on, down the supply chain). And the US’ largest oil pipeline company was breached, and its systems held to ransom.

Most analyses of these incidents recognise that endpoint security was attacked. As we alluded to in our annual report this year, breaches are a process. The initial stages of these famous attacks may not have involved a Windows PC but if your organisation grinds to a halt because everyone’s laptop is showing a red warning and a Bitcoin demand then the endpoint was compromised at some point. It needs protection, regardless of other security layers in play.

We include targeted attacks in our endpoint protection tests because hackers can use a variety of techniques to attack endpoints. Not all targeted attacks are as sophisticated and focussed as the automatic iPhone exploits used by the NSO Group. Sometimes a targeted attack can be as simple as someone using a basic tool downloaded from the internet. Your adversary might be your neighbour, rather than a government-backed organisation. In fact, that’s possibly more likely.

Protection is expected

It doesn’t really matter who represents a threat to you: a resourceful cyber ninja or an idiot colleague. When you or your business buys an endpoint protection product you expect it to stop attacks, sophisticated or otherwise. When you read the results in this report, remember that all it takes is one successful attack to ruin your day or your company.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Zero to Neo

Our reports help you choose the best enterprise anti-malware product that can protect you from ransomware and other types of attack. Attackers can have almost no ability, or nearly unlimited resources: Zero to Neo.

Choose the best enterprise endpoint solution

There seems to be no limit to the powers of cyber criminals. In 2021 the public became aware of the advanced capabilities of the NSO group, now infamous for helping governments spy on dissidents and others.

The SolarWinds attack compromised some of the largest organisations in the world (and by implication, their customers – and so on, down the supply chain). And the US’ largest oil pipeline company was breached, and its systems held to ransom.

Most analyses of these incidents recognise that endpoint security was attacked. As we alluded to in our annual report this year, breaches are a process. The initial stages of these famous attacks may not have involved a Windows PC but if your organisation grinds to a halt because everyone’s laptop is showing a red warning and a Bitcoin demand then the endpoint was compromised at some point. It needs protection, regardless of other security layers in play.

We include targeted attacks in our endpoint protection tests because hackers can use a variety of techniques to attack endpoints. Not all targeted attacks are as sophisticated and focussed as the automatic iPhone exploits used by the NSO Group. Sometimes a targeted attack can be as simple as someone using a basic tool downloaded from the internet. Your adversary might be your neighbour, rather than a government-backed organisation. In fact, that’s possibly more likely.

Protection is expected

It doesn’t really matter who represents a threat to you: a resourceful cyber ninja or an idiot colleague. When you or your business buys an endpoint protection product you expect it to stop attacks, sophisticated or otherwise. When you read the results in this report, remember that all it takes is one successful attack to ruin your day or your company.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.