Email security services tested

Cyber criminals often use email as a way to start an attack

A primary vector for cyber attacks, email services need to be secure. Email platforms provide one level of security while third-party companies offer additional services, claiming to increase protection.

Choose the best email security solution

Email security services don’t handle all threats in the same way. Some will be stopped dead, while others can infiltrate fully. Somewhere in the middle we see email quarantine systems, Junk folders and edited messages – emails that have their links, attachments and even the words in the message tampered with.

This tampering may effectively remove a threat, or it may not. There is a lot to assessing an email security solution!

How seriously do you take the email threat?

The approach that we take is to measure everything and then judge how important each result is. Our view is that keeping threats as far away from the user as possible is best. But sometimes security personnel need to see what’s coming in, so quarantines can be useful investigation tools. We have devised a scoring method that credits or penalises services according to our view on best outcomes.

Choose the best enterprise security product

By understanding the rules of security testing

Our reports help you choose the best enterprise security product that can protect you from ransomware and other types of attack.

Choose the best enterprise anti-malware solution

This report contains security testing results. You can compare the performance of a variety of products that claim to protect you against online threats. This, in theory, will help people and businesses choose the best security product.

But this is a free report. How can you trust that the high-scoring vendors didn’t just pay for their ranking? Do you suspect that some low-scoring vendors dropped out of the report? Or asked to be retested until they scored better?

What are the rules behind the scenes in security testing?

With security testing the stakes are high. From a customers’ perspective, the wrong decision could be disastrous to a business. Or a personal life.

So we, as testers, have a massive responsibility to do the right thing, meaning the honest thing. That means trying to involve as many reputable security vendors as possible in our tests and treating them all fairly.

Security vendors want to sell products and will do what they can to achieved strong marketing. That can involve appearing in weak tests or engaging with more ‘flexible’ testers. One strategy could be to test enough privately against competitors and then release the one report that shows your product at the top of the list.

We focus on strong technical testing and avoid purely marketing-led initiatives. We have awards for vendors who do well, but we stand out by assessing technology deeply and helping improve things for everyone.

Five simple rules

In our blog post Public and Private Testing we explain our five simple rules to help maintain the integrity of our reports. If you want to peak behind the curtain, to see how we work with security vendors, the information is all available online.

Testing Standards

For this report we also followed the only available Standard for anti-malware testing, the one run by the Anti-Malware Testing Standards Organization. This ensures that we do what we say we’ll do, and can prove it.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

SE Labs tested IronNet IronDefense against a range of hacking attacks designed to compromise systems and penetrate target networks in the same way as criminals and other attackers breach systems and networks.

How we test

Full chains of attack were used, meaning that testers behaved as real attackers, probing targets using a variety of tools, techniques and vectors before attempting to gain lower-level and more powerful access. Finally, the testers/attackers attempted to complete their missions, which might include stealing information, damaging systems and connecting to other systems on the network.

Product factsheet:

APT groups include:

• FIN7 & Carbanak
• OilRig
• APT3
• APT29

Advanced Security (EDR): BlackBerry Protect and Optics

SE Labs tested BlackBerry Protect and Optics against a range of hacking attacks. These were designed to compromise systems and penetrate target networks in the same way as criminals and other attackers breach systems and networks.

We used full chains of attack , meaning that our testers behaved as real attackers, probing targets using a variety of tools, techniques and vectors before attempting to gain lower-level and more powerful access. Finally, the testers/ attackers attempted to complete their missions, which might include stealing information, damaging systems and connecting to other systems on the network.

Product factsheet:

BlackBerry Protect and Optics

Zero to Neo

Our reports help you choose the best enterprise anti-malware product that can protect you from ransomware and other types of attack. Attackers can have almost no ability, or nearly unlimited resources: Zero to Neo.

Choose the best enterprise endpoint solution

There seems to be no limit to the powers of cyber criminals. In 2021 the public became aware of the advanced capabilities of the NSO group, now infamous for helping governments spy on dissidents and others.

The SolarWinds attack compromised some of the largest organisations in the world (and by implication, their customers – and so on, down the supply chain). And the US’ largest oil pipeline company was breached, and its systems held to ransom.

Most analyses of these incidents recognise that endpoint security was attacked. As we alluded to in our annual report this year, breaches are a process. The initial stages of these famous attacks may not have involved a Windows PC but if your organisation grinds to a halt because everyone’s laptop is showing a red warning and a Bitcoin demand then the endpoint was compromised at some point. It needs protection, regardless of other security layers in play.

We include targeted attacks in our endpoint protection tests because hackers can use a variety of techniques to attack endpoints. Not all targeted attacks are as sophisticated and focussed as the automatic iPhone exploits used by the NSO Group. Sometimes a targeted attack can be as simple as someone using a basic tool downloaded from the internet. Your adversary might be your neighbour, rather than a government-backed organisation. In fact, that’s possibly more likely.

Protection is expected

It doesn’t really matter who represents a threat to you: a resourceful cyber ninja or an idiot colleague. When you or your business buys an endpoint protection product you expect it to stop attacks, sophisticated or otherwise. When you read the results in this report, remember that all it takes is one successful attack to ruin your day or your company.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

Advanced Security (EDR): Kaspersky

SE Labs tested Kaspersky Endpoint Detection and Response against a range of hacking attacks. These were designed to compromise systems and penetrate target networks in the same way as criminals and other attackers breach systems and networks.

We used full chains of attack, meaning that our testers behaved as real attackers, probing targets using a variety of tools, techniques and vectors before attempting to gain lower-level and more powerful access. Finally, the testers/ attackers attempted to complete their missions, which might include stealing information, damaging systems and connecting to other systems on the network.

In this report we explain the threats used and explore how the tested product interacts with them. You might notice a similarity between the way we present this information and the way that the MITRE ATT&CK framework illustrates threat chains. This is not a coincidence. Our goal is to share information in ways that are familiar and easily understandable by the security community and its customers.

Read more of our reports here.

Advanced Security (EDR): CrowdStrike

SE Labs tested CrowdStrike Falcon against a range of hacking attacks. These were designed to compromise systems and penetrate target networks in the same way as criminals and other attackers breach systems and networks.

We used full chains of attack, meaning that our testers behaved as real attackers, probing targets using a variety of tools, techniques and vectors before attempting to gain lower-level and more powerful access. Finally, the testers/ attackers attempted to complete their missions, which might include stealing information, damaging systems and connecting to other systems on the network.

In this report we explain the threats used and explore how the tested product interacts with them. You might notice a similarity between the way we present this information and the way that the MITRE ATT&CK framework illustrates threat chains. This is not a coincidence. Our goal is to share information in ways that are familiar and easily understandable by the security community and its customers.

Read more of our reports here.

SE Labs Security Awards

Find out which products won in our annual awards

In this report, we reveal the SE Labs Security Awards for 2020-2021 and provide the latest security testing updates.

The third annual report from SE Labs charts the successes and failures of security companies, their customers and the criminals who keep relentless pressure on us all. Working from home is at its highest level in human history, which emphasises the need to secure all devices, everywhere.

Our annual awards recognise great performance in tests and in the real world.

Focus on endpoint protection results – 6 years of testing

After six solid years of testing endpoint protection we’ve produced a review that examines some of the trends and data points we’ve identified. How did your favourite anti-virus behave over the last few years?

What can we test, and how do we do it?

Meet the team behind SE Labs and find out which security solutions we test, and how we do it more realistically than anyone else.

Enterprise Ransomware Protection Tested

Red screens means red faces. Prevent, don’t pay!

Our reports help you choose the best enterprise anti-malware product that can protect you from ransomware and other types of attack.

Choose the best home anti-malware solution

Ransomware is causing all the rage right now. It’s the type of threat that gets attention because a successful attack is extremely visible (the attacker needs you to know it’s worked, or you won’t pay!) Also, there is a direct and substantial cost attached to it. In addition to paying security specialists to help, there’s a fat ransom demand sitting on your screen.

While much hacking is subtle, stealing information silently, ransomware is in your face. It stops businesses in their tracks. It gets the attention of the finance directors. It provides powerful ammunition to security teams arguing for more resources. And, of course, it makes headlines.

Ransomware attacks used to be splashed around automatically. Anyone could be unlucky enough to click on the wrong link and face a red screen asking for a Bitcoin. And these ‘commodity’, untargeted ransomware attacks still exist. Some of the threats used in the making of this report are just such attacks.

We pride ourselves on a level of transparency that elevates our work above the less open reports available. But don’t just take our word for it. This report has gone through the AMTSO certification process to ensure that we say what we’re going to do; do it; and can prove it. Our results help vendors improve their products and buyers choose the best for their own needs.

SE Labs tested VMware NSX Network Detection and Response against a range of hacking attacks designed to compromise systems and penetrate target networks in the same way as criminals and other attackers breach systems and networks.

VMware NSX Breach Response Test

Full chains of attack were used, meaning that testers behaved as real attackers, probing targets using a variety of tools, techniques and vectors before attempting to gain lower-level and more powerful access. Finally, the testers/attackers attempted to complete their missions, which might include stealing information, damaging systems and connecting to other systems on the network.

Tested products from:

VMware

APT groups include:

• FIN7 & Carbanak
• OilRig
• APT3
• APT29