Cybersecurity Product testing FAQs
Learn more about Cybersecurity Product testing with SE Labs.
Full methodologies for all tests are available from the Reviewers Guides page.
General
I am a security vendor. How can I include my product in your tests?
Please contact us at info@SELabs.uk. We will be happy to arrange a phone call to discuss our methodologies and the suitability of your product for inclusion.
I am a security vendor. Does it cost money to have my product tested?
We do not charge directly for testing products in public tests. We do charge for private tests.
What is a partner organisation?
Can I become one to gain access to the threat data used in your tests?
Partner organisations support our tests by paying for access to test data after each test has completed but before publication. Partners can dispute results and use our award logos for marketing purposes. We do not share data on one partner with other partners. We do not currently partner with organisations that do not engage in our testing.
So you don’t share threat data with test participants before the test starts?
No, this would bias the test and make the results unfair and unrealistic.
I am a security vendor and you tested my product without permission. May I access the threat data to verify that your results are accurate?
We are willing to share small subsets of data with non-partner participants at our discretion. A small administration fee is applicable.
Breach Response test
What is the Breach Response test?
The Breach Response test assesses the behaviour of security products by attacking them in realistic ways. It uses the full attack (kill) chain. Each attack uses every step you would see in a real-world threat. The test is compatible with the MITRE ATT&CK framework, but more challenging than MITRE’s testing.
Why do you test this way?
Testing with the full attack chain is the most realistic way to test. We need to check how security products would work under attack from advanced persistent threats (APTs). APTs commonly use a range of exploits and other tactics that affect different security layers.
What threats do you use in the Breach Response test?
We research the latest known threats and group them into different series. Each series in the Breach Response Threat Series (BRTS) is labelled individually. For example, BRTS1 includes the threat groups known as APT3, APT29, APT33 and APT34.
BRTS1: Threat groups used in previous MITRE ATT&CK evaluations
APT3, APT29, APT33, APT34
BRTS2: Threat groups focused on financial gain
FIN7/ Carbanak, FIN4, FIN10, Silence
BRTS3: Threat groups focussed on the energy sector
Dragonfly/ Dragonfly 2.0, APT19, APT34 enhanced
BTRS4: Threat groups covering a wide range of attack techniques and the 2021 MITRE evaluation
FIN7/ Carbanak, APT29, Dragonfly/ Dragonfly 2.0, APT34 enhanced
More details on the threat groups and series can be found on our blog.
How do you choose the threats?
We decide which APT groups and other attacks to represent using a number of factors, including but not limited to:
- The prevalence and impact of specific attacks (e.g. the SolarWinds breach).
- A need to cover major industries facing threats.
- Requests from enterprise and security partners.
Who do you test for?
We test for enterprises that need to understand the capabilities of the security products they have and are considering using. We use the Breach Response test to assess their needs regularly, as products and attacks evolve. The Breach Response test is also used by all major security vendors for in-house quality assurance.
What is the test schedule?
Breach Response testing runs on a (calendar) quarterly basis. Please contact us for more details.
What happens at the end?
We provide all clients with deep technical detail about the threats and how the products performed against them. Some projects provide regular, frequent contact with the testing team during the engagement.
Enterprise customers receive a Confidential report detailing the results and our analysis.
Security vendors receive a report and marketing rights to any award logos achieved. Vendors have options on report publication.