Simulated or real attacks in cyber security testing?
There are many different ways to test cyber security products. Most of the common approaches are useful when evaluating a service or system, but they each have pros and cons. In this article we outline the basic differences and limitations. Can you achieve realistic cyber security testing?
Join SE Labs for in-depth discussions on cyber security testing
The SE Labs Workshop in Wimbledon on 2 July 2025 promises to be an important event for cyber security professionals focused on the future of security testing. Available to both in-person and virtual attendees, this one-day event brings together industry experts and security vendors to share insights on critical topics such as testing security for MSSPs, attack chain simulation, and a significant shift in endpoint protection with the transition to Windows 11.
Business vs. consumer cyber security needs
Attendees will gain a deeper understanding of the increasing complexities in endpoint testing, including varied attack types, a separation between business and consumer security needs, and MacOS-specific testing.
Ignore Business Email Compromise test cases at your peril
Good security testing is realistic, using the kinds of threats customers see in real life. This is why we put a lot of focus on Business Email Compromise (BEC) scenarios, rather than just more conventional threat types (like generic phishing and malware).
But do all endpoint security products now include next-generation anti-virus?
Cyber security protection has evolved and so, top-tier anti-virus solutions are undeniably ‘next-generation’. This term was introduced nearly a decade ago by newcomers to the industry: a marketing device designed to compete with almost unassailable anti-malware brands.
The demand for skilled cyber security professionals has never been higher. We recognised the need to prepare the next generation for this critical field and so we created SE Labs’ cyber security training programme, Level:Up. This comprehensive initiative was designed to equip key stage 4 and 5 students with the knowledge they need to excel in the dynamic realm of cyber security. Following the programme’s resounding success in 2022 and 2023, we expanded it this year to include adult learners. The third year of Level:Up took place last month, empowering even more individuals with the tools to excel in cyber security.
Tailored to adults of all levels interested in cyber security, the programme provided carefully crafted learning modules and practical exercises that mirror industry standards. Through these modules, participants not only gained insights into the fascinating world of cyber security but also learned to grasp the fundamental concepts needed to pursue a rewarding career in the field.
A Practical Approach to Learning
Traditionally, those interested in pursuing careers in cyber security have faced challenges in finding targeted educational resources. We identified this gap and stepped in with the Level:Up programme. Unlike conventional education pathways, Level:Up goes beyond theoretical knowledge, offering students a glimpse into the diverse roles and responsibilities within the industry. By bridging this information gap, the programme ensures that students are well-prepared to take their first steps toward a fulfilling cyber security career.
SE Labs’ cyber security training not only imparts knowledge but also opens doors. Many people interested in cyber security are unsure about the career paths available to them. Level:Up addresses this uncertainty by providing comprehensive information about various routes into the cyber security industry.
A Resounding Success
The third year of the Level:Up programme marked a milestone of achievement. Running for five days, the programme welcomed seven enthusiastic people, all eager to enhance their cyber security skills. The programme showcases SE Labs’ capabilities as a provider of cyber security training.
Join Us
Enterprise-level businesses and global cyber security vendors can use our cyber security training programme to strengthen their in-house knowledge. We help enterprises build security teams, and security vendors to improve product development.
If you are interested in levelling up your cyber security skills get in touch. SE Labs’ Level:Up cyber security education programme is more than just a curriculum; it’s a transformative experience that provides participants with the knowledge, skills, and confidence to thrive in the ever-evolving landscape of cyber security.
And are attackers using it to breach your network?
Artificial Intelligence is ruling the stock market and may be on the verge of ruling the world if you believe the business influencers. If it’s as powerful as some say, surely AI can protect your Windows systems from hackers?
The products our new EPS test almost certainly rely on AI-related technologies to detect and protect against attacks. These technologies have been running in the background for about 20 years. We can argue that not only does anti-virus/ endpoint protection use AI, but it’s been doing so for many years, and certainly before Cylance claimed to be the first.
But I did something sneaky there. I slid in the word ‘-related’. Because when people talk about ChatGPT and other popular ‘AI’ tools, they are usually talking about something else. They are amazed by the utility of Machine Learning (ML) systems, which appear to be able to mimic human thought in a rather magical way.
ML is a subset of AI, so it’s related to AI but it isn’t capable of thought. It cannot reason, in the way that we hope future AI systems will. It is great at recognising patterns, but it can make mistakes and it’s not very good at understanding why it makes those mistakes.
As I wrote this introduction, I asked ChatGPT for a fun fact about SE Labs. It claimed we had run a cyber security ‘bake-off’ that involved employees baking “virus-shaped cupcakes [and] firewall-layered cakes. That sounds fun, and maybe we should do it, but we haven’t, so it’s not a fact. Fun or otherwise.
(I corrected ChatGPT, which responded, “You’re right, I made that up in an attempt to be fun and creative.” Maybe tomorrow’s robot overlords will be “fun and creative” and it won’t be so bad if they take over.)
Being able to match patterns is incredibly useful for cyber security tools, because attackers behave in largely similar ways, with small variations. ML can often detect new variations. Attackers can use ML, as indeed does SE Labs when creating some new threats, to try to evade detection. It’s a cat-and-mouse game, with both sides using computer brainpower to detect or escape detection.
As we launch our first XDR security testing program, we’d like to explain how SE Labs tests XDR solutions. But first, what is XDR? The industry has various opinions!
Extended Detection and Response (XDR) is a combination of security products working together. Its goal is to provide defenders with a coherent response to attacks. This joined-up approach can help defenders identify different stages of each attack without scrambling around using many different tools.
XDR is supposed to make things simpler for defenders, providing a dashboard (a ‘single pane of glass’) that provides complete insight into a network’s security situation.
SE Labs has produced the first comprehensive method of testing XDR solutions. The components of an XDR solution under test can be sold by the same company or different security vendors.
For example, we can test a solution that combines a Cisco email security gateway with endpoint security from CrowdStrike. And we can test a Cisco email security gateway alongside Cisco’s own endpoint security.
An SE Labs XDR test can assess combinations of cloud services such as email and identity alongside on-site firewalls, endpoint protection and Internet of Things (IoT) security products. If there is an XDR integration available, we can test it.
XDR in detail
There are plenty of definitions of XDR in the market. At SE Labs we define an XDR solution as a combination of at least two products, each of different types.
The products deployed do not need to be from the same vendor.
They must either talk to each other or a third management system, which provides the overall dashboard for detection and response.
Here is a list of products that can make up an XDR solution. They can be variously installed on-site or in-cloud:
The SE Labs testing team behaves like a real customer, allowing security vendors to provide and configure their products exactly as they would in a production environment. The testing team then change roles, behaving as attackers. It runs attacks from the beginning to the end of the attack chain, while also monitoring the security system for detections and other behaviour.
As the testers know every stage of the attack in detail, they assess how completely the products (and, more importantly, the combination of products) detect the different parts of the attacks as well as the entire attack episode.
In this way, the SE Labs testing team tests like hackers and analyses like defenders. The results are useful and realistic.
Who should care?
The results are useful, but for whom?
There are two main groups that benefit from SE Labs XDR testing.
Security sellers
The first group comprises the security vendors themselves. They can identify areas where detection is weaker and needs improvement. They may also discover areas where integration between different products could be better. The SE Labs test provides a good opportunity to make changes and strengthen the products, which means stronger protection for their users.
When things work well, security vendors can use SE Labs’ test results to highlight their successes in the market.
Security buyers
Secondly, but no less importantly, security buyers can use either public or bespoke test results to help choose the most appropriate products for their own organisations. Having real test data, showing how products handle threats in the real world, reduces risk and improves value for money.
Not every product works equally well, and that applies not only to general security effectiveness but also integration with other products. Security testing results are always an important resource before investing in a product. They are doubly so when looking to buy or build an XDR solution.
XDR Examples
Here are some examples of XDR implementations. We’ve chosen vendors and products randomly, but sensibly. For example, it makes sense to combine endpoint and email security solutions with a central data repository like a SIEM. It also makes sense to combine products from different market-leading providers, or to use all of the products from a single one.
Infer no judgement about the suitability of specific vendors from this example list:
Microsoft Defender (endpoint); Microsoft Defender (email); Splunk Cloud Platform (SIEM)
In the first example we have combined the detections and other data from two Microsoft products (endpoint and email) and sent them to a cloud-based platform that claims to provide insight into all activity.
In the second example, a simple setup combines the detection capabilities of endpoint and email threat detection products from different vendors.
Thirdly, some security companies are able to provide products for many different areas, such as firewalls, endpoint, email and web security. In this example, one vendor provides and manages all the components of the detection system.
“Turn it off and on again.” This global IT support advice is known to everyone, from Peppa Pig (Mummy Pig at Work) to The IT Crowd (every episode). But why? Why does rebooting a complex computer system solve so many problems? And why am I referring to British TV comedy in a serious report about computer security? We will answer one of those questions here.
Announcing the SE Labs Annual Security Awards 2024. We celebrate security vendors for their exceptional performance in both rigorous testing and real-world feedback from customers. These awards stand as a unique recognition within the industry, acknowledging the mix of strong lab work and practical success.
Best New Endpoint Award
Joining the SE Labs testing program is a serious commitment. Our assessments are renowned as the most stringent in the industry. Only the highest calibre products earn coveted A, AA, or AAA awards.
We always welcome new entrants to our tests and it’s incredibly satisfying to work with partners who work to solve problems as well as to celebrate victories.
The winner of this year’s Best New Endpoint Award is SentinelOne.
Best Network Detection and Response Award
Achieving robust security hinges on comprehensively grasping the capabilities and intentions of potential threats to your IT infrastructure. Endpoint Detection and Response (EDR) serves as the frontline defence, actively identifying, thwarting, and dissecting cyber threats lurking within your network. An optimal solution empowers security teams, streamlining their efforts and enhancing their effectiveness in safeguarding your digital assets.
The winner of this year’s Best Network Detection and Response Award is VMware.
Best Product Development Award
Our testing engagements whether public or private, provide vast amounts of information that can be used to improve and strengthen security products. Our award winner has taken our work and run with it, improving the security of their customers and making life significantly harder for attackers. We commend their adeptness and efficacy with this award.
The winner of this year’s Best Product Development Award is Check Point.
Best Managed Service Provider Solution Award
Following our conversations within the community and rigorous testing, we created shortlist of exceptional companies that support their partners in keeping their MSP and partner community informed of the ever-evolving threat landscape.
The winner of this year’s Best Managed Service Provider Solution Award is Sophos.
Best Email Security Service Award
Email is the primary vector for cyber threats. As such, there is much opportunity for email security services to stop cyber attacks at their earliest stages. With the rising menace of targeted assaults, often leveraging sophisticated social engineering tactics, the imperative for email security services to evolve and counter such threats becomes ever more pressing. This year’s champion has consistently showcased its prowess in discerning malicious intent from legitimate communication, reaffirming its role as a stalwart defender against digital adversaries.
The winner of this year’s Best Email Security Service Award is Trellix.
Best Next Generation Firewall Award
In our evaluations of firewalls, we conduct thorough tests that put their data sheets to the real-world challenge. We assess how swiftly data moves through these devices while adhering to security protocols, gauge their resilience against attacks amid heavy legitimate traffic, and evaluate the ease of managing these complexities. Our award winner has excelled across the board, showcasing outstanding performance and resilience.
The winner of this year’s Best Next Generation Firewall Award is Cisco.
Best Innovator Award
As attackers continually evolve their tactics, security vendors must remain innovative in their approach to detecting and thwarting these threats. Mere marketing buzzwords like “machine learning” are insufficient to earn our coveted Innovator award. Products must demonstrate exceptional effectiveness and deliver substantial added value to your security arsenal to merit recognition. It’s not just about fancy labels; it’s about tangible, impactful solutions that elevate your defence capabilities in the ever-changing landscape of cyber threats.
The best security involves having a good understanding of your enemy and the extent of the impact they could make (or have already made) on your IT infrastructure. Endpoint Detection and Response serve as the frontline guardians, actively identifying, halting, and delving into cyber threats within the network. A superior solution not only empowers security teams but also streamlines their efforts, enhancing overall effectiveness in combating evolving threats.
The winner of this year’s Best Endpoint Detection and Response Award is CrowdStrike.
Best Enterprise Endpoint Solution Award
A comprehensive endpoint security solution is fundamental to safeguarding enterprise networks from increasingly sophisticated threats. The Best Enterprise Endpoint Solution award chooses a product that goes beyond protection, delivering robust, scalable security across all devices within an organisation’s infrastructure. This solution excels at not only defending against known and emerging threats but also at seamlessly integrating with enterprise workflows, enabling swift detection, intelligent response, and proactive prevention.
The winner of this year’s Best Enterprise Endpoint Solution Award is Broadcom.
Best Small Business Endpoint Award
Small businesses face unique cyber security challenges, often balancing limited resources with the need for effective protection against ever-evolving threats. The Best Small Business Endpoint Solution award recognises a solution that provides enterprise-grade security tailored to the needs of smaller organisations. This award-winning solution offers easy deployment and reliable protection across all endpoints without requiring a dedicated security team.
The winner of this year’s Best Small Business Endpoint Award is ESET.
Best Home Anti-Malware Award
Personal devices and home networks face a wide array of digital threats, making effective malware protection essential for safeguarding privacy and data. The Best Home Anti-Malware Solution award recognises a product that delivers robust, easy-to-use protection against viruses and malicious software. This solution combines powerful threat detection with intuitive features that keep users informed, ensuring seamless, proactive security.
The winner of this year’s Best Home Anti-Malware Award goes to Kaspersky.
Best Free Anti-Malware Award
Effective security should be accessible to everyone, and the Best Free Anti-Malware Solution award celebrates a product that delivers high-quality protection at no cost. This award-winning solution stands out for its robust defence against a wide range of threats, all without compromising on essential features. With a user-friendly design and reliable threat detection, it empowers users to secure their devices and data effortlessly.
The winner of this year’s Best Free Anti-Malware award is Microsoft.
Following cyber security best practices is not enough. Excellence in the face of ever-evolving adversaries requires a proactive and strategic approach combined with a solid understanding of cyber threat intelligence. The-C2 conference is the key event in this area.
Why The-C2 conference should be on your professional calendar
The-C2 threat intelligence conference is a gathering of industry leaders, security professionals and seasoned experts.
Find out more about the spirit of The-C2; explore what to expect from this cyber security community; and make a compelling case for why your senior security executives should consider it an indispensable part of their professional calendar.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkPrivacy policy