SE Labs Team, Author at SE LABS ®

All posts

Can you trust security tests?

Written By

SE Labs Team

Published on

February 20, 2019

Clear, open testing is needed and now available to help people trust security tests

Latest reports now online.

A year ago we decided to put our support behind a new testing Standard proposed by the Anti-Malware Testing Standards Organization (AMTSO). The goal behind the Standard is good for everyone: if testing is conducted openly then testers such as us can receive due credit for doing a thorough job; you the reader can gain confidence in the results; and the vendors under test can understand their failings and make improvements, which then creates stronger products that we can all enjoy.

The Standard does not dictate how testers should test. There are pages of detail, but I can best summarise it like this:

Say what you are going to do, then do it. And be prepared to prove it.

(Indeed, a poor test could still comply with the AMTSO Standard, but at least you would be able to understand how the test was conducted and could then judge its worth with clear information and not marketing hype!)

Trust security tests

We don’t think that it’s unreasonable to ask testers to make some effort to prove their results. Whether you are spending £30 on a copy of a home anti-antivirus product or several million on a new endpoint upgrade project, if you are using a report to help with your buying decision you deserve to know how the test was run, whether or not some vendors were at a disadvantage and if anyone was willing and able to double-check the results.

Since the start of the year we put our endpoint reports through the public pilot and then, once the Standard was officially adopted, through the full public process. Our last reports were judged to comply with the AMTSO Standard and we’ve submitted these latest reports for similar assessment.

At the time of writing we didn’t know if the reports from this round of testing complied. We’re pleased to report today that they did. You can confirm this by checking the AMTSO reference link at the bottom of page three of this report or here. This helps people trust security tests.

Ask us

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

SE Labs uses current threat intelligence to make our tests as realistic as possible. To learn more about how we test, how we define ‘threat intelligence’ and how we use it to improve our tests please visit our website and follow us on Twitter.

This test report was funded by post-test consultation services provided by SE Labs to security vendors. Vendors of all products included in this report were provided with early access to results and the ability to dispute details for free. SE Labs has submitted the testing process behind this report for compliance with the AMTSO Standard v1.0.

Our latest reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

All posts

How well do email security gateways protect against targeted attacks?

Written By

SE Labs Team

Published on

December 12, 2018

Email security gateways protection: Email security test explores how and when services detect and stop threats

Latest report now online.

This new email protection test shows a wide variation in the abilities of the services that we have assessed.

You might see the figures as being disappointing. Surely Microsoft Office 365 can’t be that bad? An eight per cent accuracy rating seems incredible.

Literally not credible. If it misses most threats then organisations relying on it for email security would be hacked to death (not literally).

Email security gateways protection

But our results are subtler than just reflecting detection rates and it’s worth understanding exactly what we’re testing here to get the most value from the data. We’re not testing these services with live streams of real emails, in which massive percentages of messages are legitimate or basic spam. Depending on who you talk to, around 50 per cent of all email is spam. We don’t test anti-spam at all, in fact, but just the small percentage of email that comprises targeted attacks.

In other words, these results show what can happen when attackers apply themselves to specific targets. They do not reflect a “day in the life” of an average user’s email inbox.

We have also included some ‘commodity’ email threats, though – the kind of generic phishing and social engineering attacks that affect everyone. All services ought to stop every one of these. Similarly, we included some clean emails to ensure that the services were not too aggressively configured. All services ought to allow all these through to the inbox.

So when you see results that appear to be surprising, remember that we’re testing some very specific types of attacks that happen in real life, but not in vast numbers comparable to spam or more general threats.

Threats at arm’s length

The way that services handle threats are varied and effective to greater or lesser degrees. To best reflect how useful their responses are, we have a rating system that accounts for their different approaches. Essentially, services that keep threats as far as possible from users will win more points than those who let the message appear in or near the inbox. Conversely, those that allow the most legitimate messages through to the inbox rate higher than those which block them without the possibility of recovery from a junk folder or quarantine.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

All posts

Business Tips

Setting up 2FA: Join the 1% of most secure internet

Written By

SE Labs Team

Published on

November 19, 2018

Hackers have spent well over 20 years stealing users’ passwords from internet companies.

They’ve almost certainly got yours.

The good news is it’s very easy to make your passwords useless to hackers. All you do is switch on Two-Factor Authentication (2FA).

2FA is a second login layer

It works much like the second lock on your front door. If someone’s stolen or copied your Yale key, that double-lock will keep them out.

A digital double-lock is now vital for protecting your online accounts – email, banking, cloud storage, business collaboration and the rest. It’s up there with anti-malware in the league of essential security measures. And it’s much easier to pick a 2FA method than choose the right anti-malware (our Anti-Malware Protection Reports can help you there).

So 2FA is essential, easy, and doesn’t have to cost a thing. It’s a security no-brainer. So how come hardly anyone uses it?

Join the one per cent elite!

Earlier this year, Google revealed that only 10 per cent of their users have ever bothered setting up 2FA. Just a fraction of those – we estimate around one per cent of all internet users – use the most secure type of 2FA, a USB security key.

In this article we’ll show you how to join that elite one per cent for less than £20. If you’d rather watch a step-by-step demo, here’s our YouTube video.

(This blog reflects the views and research of SE Labs, an independent security testing company. We never use affiliate links.)

Why everyone in your business should use 2FA

You’re not the only person who knows your usernames and passwords. Head over to Have I Been Pwned? and type in your email address to find out how many of your accounts have been hit by hacking attacks.

While you’re digesting those results, here’s a sobering statistic. More than 90 per cent of all login attempts on retail websites aren’t by actual customers, but by hackers using stolen credentials (Shape Security, July 2018).

Nearly everyone has had their passwords stolen. But hardly anyone protects their accounts using 2FA. We’re all leaving our front doors unlocked.

And as hackers plunder more and more big-name services (as well as all those services you’d forgotten you had accounts with), the more chance they have to steal the passwords you use everywhere.

This is why you must never using the same password twice. Don’t be tempted to use a pattern to help you remember them, either (‘123amazon’, ‘123google’ and so on). Hackers decode that stuff for breakfast. We’re also not keen on password managers. They’re Target Number One for hackers.

Instead, store your passwords where no-one can find them (not online!) and deadlock your accounts using 2FA. It’s the only way to make them hack-proof.

Why a USB key is the best way to lock your accounts

The ‘memorable information’ you have to enter when logging into your online bank account is a watered-down version of 2FA. Hackers can easily create spoof login pages that fool you into handing over all your info, as demonstrated in our NatWest phishing attack video.

Proper 2FA methods are much tougher to crack. They involve more than one device, so a hacker can’t simply ransack your computer and steal all pertinent data. Without the separate device, your passwords are useless to them.

Use more than one 2FA method if offered. This double-locks your double-locks – and also gives you another way into your account if one method fails. See our 2FA YouTube video for a step-by-step guide to doing this for your Google account.

Here’s a quick run-through of your options, starting with the most basic.

Google prompt
How it works: Tap your Android screen to confirm your identity.
Pros and cons: Very quick and easy, but only works with Google accounts and Android devices. Useful as a backup option.

SMS code
How it works: You’re texted (and/or voice-messaged) a PIN code to enter after your usual login.
Pros and cons: Authentication is split between two devices. It works on any mobile phone at no additional cost. But it can be slow, and the code may appear on your lock screen.

Authentication app
How it works: A free app, such as Google Authenticator, generates a unique numerical security code that you then enter on your PC.
Pros and cons: Faster and more reliable than SMS, and arguably more secure, but you’ll need a smartphone (Android or iOS).

Backup codes
How it works: A set of numerical codes that you download and then print or write down – then keep in a safe place. Each code only works once.
Pros and cons: The perfect backup method. No need for a mobile phone. A piece of paper or locally-stored computer file (with disguised filename) is easier to hide from thieves than anything online.

And the most secure 2FA method of all…

USB security key
How it works: You ‘unlock’ your accounts by plugging a unique USB stick (such as this YubiKey) into your computer.
Pros and cons: A whole list of pros. USB keys are great for business security, because your accounts remain locked even if a hacker breaches your phone. They’re convenient: no need to wait for codes then type them in. And they cost very little considering how useful they are. One key costs from £18, and is all you need to deadlock all your accounts. Buy one for all your employees – and clients!

Deadlock your Google account: a 2FA walk-through
Google lets you lock down your entire account, including Gmail and Google Drive, using multiple layers of 2FA (which it calls 2-Step Verification). It’s one of the most secure 2FA configurations you’ll find, and it’s easy to set up.

Here are the basic steps. For a more detailed step-by-step guide, see our YouTube video.

Order a USB security key. Look for devices described as FIDO (‘Fast IDentity Online’) – here’s a FIDO selection on Amazon – or head straight for the Yubico YubiKey page. Expect to pay from £18 to around £40.
Go to Google’s 2-Step Verification page, click Get Started then sign into your account. Choose a backup 2FA method, click Security Key, then plug in your unique USB stick. Google automatically registers it to you.
Choose a second 2FA method such as SMS code, plus a backup method such as a printable code, Google prompt or authenticator app.
That’s it – welcome to the top one per cent!

Deadlock all your online accounts in minutes

All reputable online services now offer 2FA options. But, as you’ll discover from the searchable database Two Factor Auth, not all services offer the best 2FA options.

For example LinkedIn only offers 2FA via SMS, and doesn’t support authenticator apps or USB security keys – the most secure types of 2FA. Even Microsoft Office 365 doesn’t yet support security keys. We expect better from services aimed at business users.

What’s more, 2FA settings tend to be well buried in account settings. No wonder hardly anyone uses them. Here’s where to click:

Amazon: Go to Your Account, ‘Login & security’, enter your password again, and then click Edit next to Advanced Security settings.
Apple: Go to the My Apple ID page then click Security, Two-Factor Authentication.
Dropbox: Click the Security tab to set up SMS or app authentication. To configure a USB security key, follow Dropbox’s instructions.
Facebook: Go to ‘Security and login’ in Settings and scroll down to ‘Use two-factor authentication’. Click Edit to get set up.
LinkedIn: Go to Account Settings then click Turn On to activate SMS authentication.
Microsoft: Log in, click Security, click the ridiculously small ‘more security options’ link, verify your identity, and then click ‘Set up two-step verification’. Doesn’t yet support USB security keys. Some Microsoft services, such as Xbox 360, still don’t support 2FA at all.
PayPal: Go to My Profile then click My Settings, Security Key and then Get Security Key. Don’t accept the offer to get a new code texted to you every time you log in, because then a hacker can do it too!
TeamViewer: Go to the login page, open the menu under your name, click Edit Profile then click Start Activation under the 2FA option. Supports authenticator apps only, not SMS.
Twitter: Go to ‘Settings and privacy’, Security, then tick ‘Login verification’.
WhatsApp: In the mobile app tap Settings, Account, ‘Two-step verification’.

All posts

Latest security tests introduce attack chain scoring

Written By

SE Labs Team

Published on

October 25, 2018

When is a security breach serious, less serious or not a breach at all? Attack chain scoring is important in helping you find out.

Latest reports now online.

UPDATE (29/10/2018): This set of reports are confirmed to be compliant with AMTSO Standard v1.0 by the Anti-Malware Testing Standards Organization.

Our endpoint protection tests have always included targeted attacks.

These allow us to gauge how effectively anti-malware products, in use by millions of customers, can stop hackers from breaching your systems.

We penalise products heavily for allowing partial or full breaches and, until now, that penalisation has been the same regardless of how deeply we’ve been able to penetrate into the system. Starting with this report we have updated our scoring to take varying levels of ‘success’ by us, the attackers, into account.

Attack chain scoring

The new scores only apply to targeted attacks and the scoring system is listed in detail on page eight of each of the reports.

If the attackers are able to gain basic access to a target, which means they are able to run basic commands that, for example, allow them to explore the file system, then the score is -1.

The next stage is to attempt to steal a file. If successful there is a further -1 penalty.

At this stage the attackers want to take much greater control of the system. This involves increasing their account privileges – so-called privilege escalation. Success here turns a bad situation worse for the target and, if achieved, there is an additional -2 penalty.

Finally, if escalation is achieved, certain post-escalation steps are attempted, such as running a key logger or stealing passwords. A final -1 penalty is imposed if these stages are completed, making possible scores for a breach range between -1 and -5 depending on how many attack stages are possible to complete.

We have decided not to publish exact details of where in the attack chain each product stands or falls, but have provided that detailed information to the companies who produce the software tested in this report and who have asked for it.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

All posts

Network security appliances vs. Word and PowerShell

Written By

SE Labs Team

Published on

September 12, 2018

Network security appliances tested. Over the last few months we have seen a surge in attacks using apparently innocent documents that install malware covertly on victims’ systems.

Unless you are running specialist monitoring tools, or very effective security software, you probably won’t see any symptoms of the attack.

The goals of these attacks are varied. In some cases they provide remote access to hackers. In others so-called cryptocurrency mining software is installed. These programs (ab)use your systems’ processing power in an attempt to generate cryptocurrencies such as Monero. The attackers get rich off your power bill.

While there are variations in how the attacks work, the typical path to compromise involves opening the document, which could be in Microsoft Word format, after which an exploit runs a PowerShell script. This, in turn, downloads and installs the malware.

Network security appliances

In this report we investigate how effectively some very popular network security products are at handling these and other threats.

As usual, we have also thrown in some particularly devious targeted attacks that appear to be completely legitimate applications but that provide us with remote access to unprotected targets. When we gain this access we try to hack the target in the same way a real attacker would. This gives the security products the best chance of detecting and potentially blocking the bad behaviour.

The good news is that all of these products were able to detect many (if not all) of the threats. Some were able to block most, although complete protection is not guaranteed. As always, a layered approach to protection is best. For advice on which endpoint software to choose see our Endpoint Protection test results on our website.

Latest report (PDF) now online.

Featured podcast:

All posts

Detected, blocked, quarantined, cleaned?

Written By

SE Labs Team

Published on

July 24, 2018

What happens when your choice of security software handles an attack? Does is detect, block, quarantine or clean? Or fail?

Latest reports now online.

It should be simple. You’ve clicked on the wrong link, opened a malicious email or installed something inadvisable. A threat is now attacking your PC and it’s up to your choice of anti-malware product to handle things.

But what does it actually do under the hood?

Detected, blocked, quarantined, cleaned?

Detection is important. The product should recognise that a threat exists, even if it can’t fully handle it. At least you can receive an alert and seek help (or an alternative anti-malware program!)
Blocking threats is also very important. Ideally the protection system will prevent the malware from running. Sometimes that doesn’t happen and the malware runs. In that case one hopes that the security software would recognise that bad things are happening and stop them. This is what we call ‘neutralisation’.

Following a neutralisation your computer might not be completely clean. There could be some rogue code still on your hard disk, possibly even on your Desktop. There might also be entries in the Registry and elsewhere that will try to run this code (or code that has been deleted or quarantined).

You probably want your system to be protected by having threats blocked and, in cases where they are not, that they be removed as fast as possible and all significant traces removed. We call this happy state ‘complete remediation’.

In SE Labs tests we measure all of these outcomes, including the worst one: compromise.

Protection Details

If you want to know how the different products tested in this report handled threats in detail, check out the Protection Details table and graph on page 10 of our reports. We don’t show details of which products completely remediated threats and which did not when neutralising but the Protection Ratings on page eight take these into account.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

See all blog posts relating to test results.

All posts

What’s the difference between SE Labs and a cyber-criminal?

Written By

SE Labs Team

Published on

June 26, 2018

As we prepared this network security appliance report for publication we were also getting ready to present at BT’s internal security conference Snoopcon.

We had been asked to talk about security products and how they might not do what you assume they will.

Network Security Appliance report

Reports like this (PDF) provide an interesting insight into how security products actually work. Marketing messages will inevitably claim world-beating levels of effectiveness, while basic tests might well support these selling points. But when you actually hack target systems through security appliances you sometimes get a very different picture.

Some vendors will support the view that testing using a full attack chain (from a malicious URL pushing an exploit, which in turn delivers a payload that finally provides us with remote access to the system) is the right way to test. Others may point out that the threats we are using don’t exactly exist in the real world of criminality because we created them in the lab and are not using them to break into systems worldwide.

Available tools

We think that is a weak argument. If we can obtain access to certain popular, inexpensive tools online and create threats then these (or variants extremely close to them) are just as likely to exist in the ‘real world’ of the bad guys as in a legitimate, independent test lab. Not only that, but we don’t keep creating new threats until we break in, which is what the criminals (and penetration testers) do. We create a set and, without bias, expose all of the tested products to these threats.

But in some ways we have evolved from being anti-malware testers to being penetration testers, because we don’t just scan malware, execute scripts or visit URLs. Once we gain access to a target we perform the same tasks as a criminal would do: escalating privileges, stealing password hashes and installing keyloggers. The only difference between us and the bad guys is that we’re hacking our own systems and helping the security vendors plug the gaps.

Latest report (PDF) now online.

All posts

Analysis

Big Time Crooks: A Fake Sense of Security

Written By

SE Labs Team

Published on

May 24, 2018

When an online scam becomes too successful, the results can be farcical. And bring a fake sense of security…

In the movie Small Time Crooks, Woody Allen leads an inept gang of would-be robbers who rent a store next to a bank. They plan to tunnel into the vault. As a cover, Allen’s girlfriend (played by Tracey Ullman) sets up a cookie business in the store. Ullman’s business takes off and, to maintain the cover, the gang must work hard. They set up production facilities, hire staff, find distributors and so on.

Why is this relevant? Well, rewind to 2002. The internet had already taken off in a big way. People were pouring online as new opportunities exploded into the public consciousness. Cybercrime was also exploding. The internet presented a new breed of tech savvy crooks with their own set of opportunities. For one gang, an Allenesque adventure was about to begin, bringing people a fake sense of security.

Humble Beginnings

How many times have you browsed a web page that suddenly throws up an alarming warning that your computer is infected? And the only thing that can save you is to immediately buy a special program or call a special number? If you’re up to date with system patches and use a reputable anti-virus solution, you’re rarely in danger from such sites these days.

It was not always so.

For millions of internet users who were running without protection the apparent authority of such “scareware” sites made them act. They downloaded free “anti-virus” software that gave a fake sense of security and infected them with real malware. They parted with real cash and many also paid again to have their computers cleaned by professionals.

Look through the history of scareware, and one company repeatedly appears: Innovative Marketing Inc. This was the name used in US Federal Trade Commission paperwork but the organisation also known by a wide range of other names. Innovative was registered in Belize in 2002.

Despite the appearance of being a legitimate business, its initial products were dodgy. They included pirated music, porn and illicit Viagra, along with sales of “grey” versions of real anti-virus products.

Innovative security

After Symantec and McAfee both put pressure on the company to stop those software sales in 2003, Innovative tried to write its own. The resulting Computershield wasn’t effective as anti-virus protection, but the company sold it anyway as a defence against the MyDoom worm. Innovative aggressively marketed its new product, and according to press reports, it was soon raking in $1 million per month. As the threat from MyDoom receded, so too did profits.

The company initially turned to adware as a new revenue source. This enabled so-called “affiliates” to use malicious web sites to silently install the adware on vulnerable Windows computers. Getting victims to visit those sites was achieved by placing what looked like legitimate adverts on real sites. Click them, and you became infected. The affiliates then pocketed a fee of 10 cents per infection, but it’s through that Innovative made between $2 and $5 from sales of the advertised products.

Fake sense of security

Meanwhile, development of completely fake anti-virus software snowballed at the company’s Kiev office. A classic example is “XP Antivirus 2008”. This also went by a large number of pseudonyms and evolved through many versions. A video of it trashing an XP machine can be found here. Its other major names include Winfixer, WinAntivirus, Drivecleaner, and SystemDoctor.

In many ways, Innovative’s scareware was, well, innovative. It disabled any legitimate protection and told you the machine was heavily infected, even going to the trouble of creating fake blue screens of death. At the time, some antivirus companies had trouble keeping up with the rate of development.

Attempts to access Windows internet or security settings were blocked. The only way of “cleaning” the machine was to register the software and pay the fee. Millions of people did just that. The FTC estimates that between 2004 and 2008, the company and its subsidiaries raked in $163 million.

In 2008, a hacker with the handle NeoN found a database belonging to one of the developers. This revealed that in a single week one affiliate made over $158,000 from infections.

The Problem of Success

Initially, Innovative used banks in Canada to process the credit card transactions of its victims, but problems quickly mounted as disgruntled cardholders began raising chargebacks. These are claims made to credit card companies about shoddy goods or services.

With Canadian banks beginning to refuse Innovative’s business, it created subsidiary companies to hide its true identity, and approached the Bank of Kuwait and Bahrain. Trouble followed, and in 2005 this bank also stopped handling Innovative’s business due to the high number of chargebacks. Eventually, the company found a Singaporean bank called DBS Bank to handle the mounting backlog of credit card transactions.

The only solution to the chargeback problem was to keep customers happy. So, in true Allenesque style, Innovative began to invest in call centres to help customers through their difficulties. It quickly opened facilities in Ukraine, India and the USA. Operatives would talk the customers through the steps needed for the software to miraculously declare their systems free of malware. It seems that enough customers were satisfied with a fake sense of security to allow the company to keep on raking in the cash.

Official complaints

But people did complain, not to the company but to the authorities. The FTC received over 3,000 complaints in all and launched an investigation. Marc D’Souza has been convicted of his role in the company and ordered to pay £8.2 million, along with his father who received some of the money. The case of Kristy Ross for her part in the scam is still going through the US courts. Lawyers are arguing that she was merely an employee.

Several others, including Shaileshkumar “Sam” Jain and Bjorn Daniel Sundin, are still at large, and have had a $163 million judgement entered against them in their absence. Jain and Sundin remain on the FBI’s Most Wanted Cyber Criminal list with rewards for their arrests totalling $40,000.

An Evergreen Scam

Scareware is a business model that rewards creativity while skirting the bounds of legality. Unlike ransomware, where criminal gangs must cover their tracks with a web of bank accounts and Bitcoin wallets, scareware can operate quite openly from countries with under-developed law enforcement and rife corruption. However, the gap between scareware and ransomware is rapidly closing.

Take the case of Latvian hacker Peteris Sahurovs, AKA “Piotrek” AKA “Sagade”. He was arrested on an international arrest warrant in Latvia in 2011 for his part in a scareware scam. Sahurovs then fled to Poland, where he was subsequently detained in 2016.

The hacker was extradited to the US and pled guilty in February this year to making $150,000 – $200,000. US authorities claim the total made by Sahurovs’ gang was closer to $2 million. He’s due to be sentenced in June.

Fake advertising

According to the Department of Justice, the Sahurovs gang set up a fake advertising agency that claimed to represent a US hotel chain. Once adverts were purchased on the Minneapolis Star Tribune’s website, they were quickly swapped out for ones that infected vulnerable visitors with their malware. This made computers freeze and produce pop-ups explaining that victims needed to purchase special antivirus software to restore proper functionality. This case is interesting as it shows a clear cross over from scareware to ransomware. All data on the machines was scrambled until the software was purchased.

The level of sophistication and ingenuity displayed by scareware gangs is increasing, as is their boldness. You have probably been called by someone from India claiming to be from Microsoft, expressing concern that your computer is badly infected and offering to fix it. Or they may have posed as someone from your phone company telling you that they need to take certain steps to restore your internet connection to full health. There are many variations on the theme. Generally, they want you to download software that confirms their diagnosis. Once done, you must pay them to fix the problem. This has led to a plethora of amusing examples of playing the attackers at their own game.

False sense of safety

It’s easy to see the people who call you as victims of poverty with no choice but to scam, but string them along for a while and the insults soon fly. They know exactly what they’re doing, and from the background chatter on such calls, so do hundreds of others. Scareware in all its forms is a crime that continues to bring in a lot of money for its perpetrators and will remain a threat for years to come.

See all blog posts relating to analysis.

All posts

Are you buying solid protection or snake oil?

Written By

SE Labs Team

Published on

May 9, 2018

Sometimes testers need to be tested too. We’re always up for a challenge!

How do you know which security products to buy? Are you buying solid protection or snake oil? Many rely on independent tests to help in the decision-making process. But how do you know if a test is any good or not?

Latest reports now online.

The Anti-Malware Testing Standards Organization (AMTSO) has been working to create a Standard that will give you, the customer, some assurance that the test was conducted fairly.

Benefits of following a Standard

Earlier this year AMTSO has been trying out its Standard, which it has been working on for many months. SE Labs is proud to be involved in this initiative and the testing for this report has been assessed for compliance with the Standard.

If that sounds a bit dry, what it means is that there are experimental rules about how a tester should behave and we have put ourselves up for judgment by AMTSO.

Did participating in this process change the way we worked? Yes, but not in the technical ways that we test. Instead we turned the testing world’s business model on its head.

Are you buying solid protection or snake oil?

Many testers charge vendors money to be tested. Some will test regardless, but charge money if the vendors want to see their results before publication (and have the opportunity to make requests for corrections).

We think that the dispute process should be free for all. SE Labs has not charged any vendor for its participation in this test and we provided a free dispute process to any vendor that requested it. In this way every vendor is treated as equally as possible, for the fairest possible test.

UPDATE (10th May 2018): We are extremely proud to announce that our 2018 Q1 reports have been judged compliant (PDF) with the AMTSO Draft Standard v6.1 – 2018-05-10.

If you spot a detail in this report that you don’t understand, or would like to discuss, please contact us via our Twitter or Facebook accounts.

These reports, for enterprise, small business and home users are now available for free from our website. Please download them and follow us on Twitter and/or Facebook to receive updates and future reports.

All posts

Tough test for email security services

Written By

SE Labs Team

Published on

May 3, 2018

Our latest email cloud security test really challenged the services under evaluation.

Latest report now online.

Last summer we launched our first email cloud security test and, while it was very well received by our readers and the security industry as a whole, we felt that there was still work to do on the methodology.

This report shows the results of six months of further development, and a much clearer variation in the capabilities of the services under test.

The most significant change to the way we conducted this test lies in the selection of threats we used to challenge the security services: we increased the number and broadened the sophistication.

Whereas we might have used one fake FBI blackmail email previously, in this test we sent 10, each created using a different level of sophistication. Maybe a service will detect the easier versions but allow more convincing examples through to the inbox?

We wanted to test the breaking point.

We also used a much larger number of targeted attacks. There was one group of public ‘commodity’ attacks, such as anyone on the internet might receive at random, but also three categories of crafted, targeted attacks including phishing, social engineering (e.g. fraud) and targeted malware (e.g. malicious PDFs).

Each individual attack was recreated 10 times in subtly different but important ways.

Attackers have a range of capabilities, from poor to extremely advanced. We used our “zero to Neo” approach to include basic, medium, advanced and very advanced threats to see what would be detected, stopped or allowed through.

The result was an incredibly tough test.

We believe that a security product that misses a threat should face significant penalties, while blocking legitimate activity is even more serious.

If you’re paying for protection threats should be stopped and your computing experience shouldn’t be hindered. As such, services that allowed threats through, and blocked legitimate messages, faced severe reductions to their accuracy ratings and, subsequently, their chances of winning an award.

Intelligence-Led Testing

We pay close attention to how criminals attempt to attack victims over email. The video below shows a typically convincing attack that starts with a text message and ends stealing enough information to clean out a bank account.

Author: SE Labs Team

Can you trust security tests?

Clear, open testing is needed and now available to help people trust security tests

Trust security tests

Ask us

Setting up 2FA: Join the 1% of most secure internet

Latest security tests introduce attack chain scoring

Attack chain scoring

Network security appliances vs. Word and PowerShell

Network security appliances

Featured podcast:

Detected, blocked, quarantined, cleaned?

Detected, blocked, quarantined, cleaned?

Protection Details

What’s the difference between SE Labs and a cyber-criminal?

Network Security Appliance report

Available tools

Big Time Crooks: A Fake Sense of Security

Humble Beginnings

Innovative security

Fake sense of security

The Problem of Success

Official complaints

An Evergreen Scam

Fake advertising

False sense of safety

Are you buying solid protection or snake oil?

Sometimes testers need to be tested too. We’re always up for a challenge!

Benefits of following a Standard

Are you buying solid protection or snake oil?

Contact us

Get in touch

Feel free to reach out to us with any questions or inquiries