All postsSE Labs announces comparative breach response testing
The next phase in our Breach Response test is to start comparing products. There has been high demand for comparative testing since we first proposed our Breach Response test back in 2016. Clients really want to see breach response products compared.
Comparing products in the Breach Response test
For the last few years our Breach Response reports have been of standalone products.
The first milestone is this summer, when we will be running a private comparative test of products. Comparatives in this space are few and far between. Good ones are even rarer. Therefore we have taken very prudent steps towards this goal over the last couple of years. We plan to publish a public comparative report in the first half of 2022.
What kind of products will be tested? And configured how?
The majority of the products included in this private test run are ‘natural born’ Endpoint Detection and Response (EDR) products. However, we have opened participation to any product from any vendor that would like to demonstrate capabilities against a series of APT groups.
Historically our policy for public testing is that the products need to be configured as they would be in a real-world deployment. Usually the conversation starts with “Treat us as a normal customer.”
The products deployed in this test must reflect a real-world deployment. That means they should be set up with suitable policies and configurations available to the vendors’ customers. We understand that enterprises differ, therefore the deployment should reflect a security-conscious organisation.
The methodology includes testing for false positives, so overly aggressive deployments will be penalised in the scoring.
Threat groups
The choice of attacks, or Advanced Persistent Threats (APTs) was very important. We wanted to cover a range of tactics and techniques. At the same time we didn’t want to treat products that we have never tested unfairly. Some vendors have a long testing relationship with us and have a good understanding of our existing series of threats.
In our advanced threat testing we use a collection of attacks we call a Breach Response Threat Series (BRTS). You can find details of BRTS1, 2 and 3 here.
For this summer’s comparative test we are using a bespoke BRTS that combines all our previous series. In total four APT groups will be tested.
Many vendors are familiar with the groups known as ‘FIN7 & Carbanak’ and ‘APT29’ because these are used in MITRE evaluations. The ‘Dragonfly & Dragonfly 2.0’ and ‘APT34 (Oilrig)’ groups are already established in our own BRTS3.
Click below for details of the tactics and techniques used by each group.
Threat group details
After the test has been completed, exact test scenarios and all test artifacts will be shared with the participants.
False positives
SE Labs works with both security vendors and their enterprise customers, so we are in a unique position to consider points from both sides when approaching a test. Increasing visibility in a potential threat is great but an enterprise SOC team doesn’t want to tear their brain out with every alert coming from a product that is too verbose.
The Breach Response test already has enterprise-focused false positive (FP) testing baked in. However, as an add-on to the testing we are piloting a new scenario focused FP approach. These results won’t impact the final scoring, but we will encourage our participants to provide feedback in the analysis calls after the test.
Dates and analysis calls
The testing will take place on the following schedule:
| Set-up | Testing | Feedback | Report Building | PT Calls |
| 5th – 16th July 2021 | 19th July – 20th August 2021 | 23rd August – 3rd September 2021 | 6th -10th September 2021 | 20th -30th September 2021 |
The scheduling is in line with our usual testing framework. While we are not publishing a report, everyone involved will received a confidential, anonymized report showing their placement in the ranking tables.
We are also adding formalized post-test analysis calls to discuss suggestions for any changes to the methodology and test framework that may arise from any part of the testing engagement. These will be in the format of roundtable discussions at which we present our key findings and areas we are looking at in the future, followed by ideas from the vendors involved.
Outlook
We plan to run a public version of the test in which all involved vendors will agree to participate before the testing starts. We plan to publish two comparative reports in 2022.
The market’s adoption of Extended Detection and Response (XDR) as an approach is great to see and it’s something to which we are paying close attention. From a testing perspective we look forward to building upon this Breach Response test for an XDR test in the future. The team constantly evaluates the evolution of the market and we always aim to adapt to its needs.
SE Labs also continues to focus on prevention-based testing and we aim to run a comparative test in the second half of 2022. We are formulating changes to our prevention-focused methodology this year to better help spot differences in the products we test. The proposed methodology for a comparative in that space will be released in Q4 this year.
We are very excited for this test and its impact on this space. We have been working towards it for a while and, with the help of security vendors and industry analysts alike, we are looking forwards to constantly evolving to suit the market’s needs.
Want to see your favourite breach response products compared?
If you are a security vendor who wants to know more, please contact us. Analysts and enterprises can also request that we include their choice of vendors and products.
