All posts“You’re thinking: How much truth is in that report?”
Show notes for series 1, episode 5
Companies spend trillions on cyber security each year. But how do they decide which products and services are the best?
Selling Security – The Insider’s Guide
We dig down into the sometimes shady world of cyber security sales, market analysis and product testing. How do the relationships work between clients, salespeople, analysts, testers and the media?
This episode contains opinions from testers, ex-testers, a security vendor and a major security buyer. You won’t have heard such a well-rounded and honest discussion about an industry so fraught with smoke and mirrors.
Simon Edwards (SE Labs) talks to special guests Allison Elizondo (ex-NSS Labs, now SentinelOne) and Nabil Khokhar (ex-Glencore, now DarkGuard)
Please subscribe and join the discussions.
Sign up to our newsletter!
Topics
- Introduction to Selling Security – The Insider’s Guide
- The importance of security testing
- How vendors do/ should work with testers
- What makes a good tester?
- How do large companies choose security products?
- Analysts and the media
- Security conferences are dead
- Bad testing of bad products
- Corrupt testers
Resources
Transcription
(Generated automatically)
Simon Edwards 0:01
Welcome to DE:CODED, Series One, Episode Five – your weekly podcast providing in depth insight into cybersecurity. Show notes, including any links mentioned in the show are available at DecodedCyber.com. I’m Simon Edwards, CEO and founder of SE Labs, the world’s leading security testing lab. I’ve been advising on security for over 20 years.
Marc Briggs 0:29
And I Marc Briggs with over 10 years experience as a British Army officer, and five years in cyber security. I’m now the COO of SE Labs.
Simon Edwards 0:39
In this episode, we take a deep dive into the world of security testing. And what we talk about might surprise you. We’ll kick off by introducing our first guest, someone with a unique perspective on the testing world. Alison Elizondo, thank you ever so much for joining us.
Allison Elizondo 0:56
Yeah, thanks for having me.
Simon Edwards 0:57
And Allison, you have worked in security testing in the past?
Allison Elizondo 1:01
Yeah, that’s right. I spent about four years at a pretty well known test lab called NSS labs, and they closed last year, around October.
Simon Edwards 1:11
And so with with us with any test lab, and NSS, SE Labs and the others, we look at security products to assess their effectiveness. Would you say that most test labs work entirely independently?
Allison Elizondo 1:26
It’s fairly independently, I guess…
Marc Briggs 1:29
It’s a loaded question!
Allison Elizondo 1:29
There’s a lot in that question. You know, I think that we’re human, right. And these test labs are run by human beings. And I guess it’s impossible not to be influenced by certain experiences maybe, that you might have with certain vendors. And I think, more than anything, versus how much that one vendor, that vendor might invest in engaging that test lab, I think the experience that the test lab has with that vendor might feed into their opinion. And of course, if the tech works, that’s always a plus.
Simon Edwards 2:03
There are only a few security test labs that we’re aware of in existence. How important is it that people are keeping an eye on the different security products available, and seeing how effective they are?
Allison Elizondo 2:17
Yeah, I think it’s critical. And I think that, you know, labs, like SE Labs and others are very important, just in that sort of, I guess, ecosystem. It keeps us all honest. The key it holds us accountable. And in the age of, I guess, the online review, you know, the customers are really the ones selling your product. So transparency is important. You know, so I think, yeah, I think there’s a very important place for for test labs.
Simon Edwards 2:46
Yeah, I guess TrustPilot might be good enough. If you want to buy some shoes, or a hat, but spending a million dollars on a firewall, it’s harder to get a sense of what’s realistically good or not.
Allison Elizondo 2:57
Yeah, absolutely. It’s just too important.
Marc Briggs 3:00
And I guess, with the test labs, they give that independent perspective, because when you are an enterprise business that’s looking to spend upwards of a million dollars on a firewall, or you’ve got to take your decision making from the marketing and sales departments of the security vendors, which are naturally biased and will push their own products. Without that independent bias.
Simon Edwards 3:27
Yeah. I think independence is crucial. But how can you judge that, you know, if we the testers sit there in judgment of everybody else? And how effective they are? How can people tell that we’re actually honest, ethical and competent?
Allison Elizondo 3:41
That’s a good question. I guess maybe it starts in my head, maybe with even the way that the test lab monetizes. I mean, that can say a lot about how you’re influenced, I suppose. You know, it’s tricky, though. You know, if you’re if your business model is set up in a way that completely relies on the vendors that you just that could put you in a precarious position. If you could imagine you’re beholden somewhat. To the vendor community for your for your business.
Simon Edwards 4:16
Yes, who watches the watchers, isn’t it? I think one way to get past that because a lot of testing has to involve the vendors, whether they’re paying money or not, is transparency.
Allison Elizondo 4:28
Right!
Simon Edwards 4:28
So saying what you’re going to do and then doing it, and being prepared to prove it. I think that gets rid of a lot of the fear. I think there are some people who will always assume that testers are in the vendors’ pockets. There are some people who will always assume that testing isn’t very effective, or can’t even be done. I’ve heard that before that it’s not even possible to test products.
Simon Edwards 4:53
The old, previous CEO of fireeye has famously said, Well, you can’t test us it’s not possible. Just spend, however many millions on… I might be paraphrasing… buy our product basicly and see the benefit feel the feel the quality.
Allison Elizondo 5:07
Yeah, just trust us.
Simon Edwards 5:09
Yeah, yeah. But I think just being honest and open and putting your hand up when you make mistakes, or acknowledging the limitation of your results as well. I mean, I remember test from the old days where the tester would do one thing and then draw conclusions that didn’t really match what they’d done. And I think that’s a problem.
Allison Elizondo 5:29
Yeah, yeah, I hear what you’re saying. And I, you know, one thing I would I would try. And, you know, during my time at NSS I lead the vendor relations team. And I always felt that it was important to prepare the vendors well, for the experience they were about to have, because it’s, it’s time consuming, it’s very, very, you know, intense. And one way that I would try and get them ready for that experience was to make sure they had the right mindset.
Allison Elizondo 5:57
You know, I think if you go into this with a healthy mindset, as a vendor, thinking to yourself, okay, we’re going to, we’re going to be focused, we’re going to be organized, we’re going to get everything, squeeze every piece of value out of this test experience that we can, it’s going to help make our product better, you know, and then if you kind of go into it with that attitude, you just, you just get as a vendor a lot more out of the experience. And then over time that rapport starts to build between the vendor and the test organization.
Simon Edwards 6:27
Actually, I think you’ve hit the nail on the head there, because when we’ve run projects that have been entirely marketing-led, it’s always been a disaster, because the idea that a vendor will always get exactly the results that they want for marketing purposes, is quite low, because no product is perfect.
Allison Elizondo 6:45
That’s right.
Simon Edwards 6:45
But if you’re focusing on improving the product, then it’s a win win. Either the product does really well, in which case, great, have a badge do a podcast, or it doesn’t. Or it doesn’t, in which case, well, there’s some work to be done. But we’re improving things, which is, which is fantastic!
Allison Elizondo 7:02
Yeah, and I think I think the customer least I’m a customer of some things, and I always appreciate that transparency to say, “Hey, we have a great product. It’s not perfect. here’s the here’s the ways in which it isn’t perfect. And we’re addressing that for you the customer.” I think that’s a healthy. Yeah.
Simon Edwards 7:18
I forgot because you because you used to work for a tester, I forgot you actually worked for a vendor now.
Allison Elizondo 7:23
I do. I’m on the other side. Yeah.
Simon Edwards 7:26
So when, when you’ve obviously worked with different testers, and I just just to give people outside of this weird little world that we occupy a sense of, of how things are, how does it feel to be a vendor, and then some people either come to you and cooperate and some people maybe operate more on their own? How does that feel from from your side?
Allison Elizondo 7:45
Well, I think the the time that I spent on the other side and and don’t get me wrong, I mean, I do analyst relations for for a company called SentinelOne. And the analyst relations versus the test lab experience are similar, but not the same. But the mindset is, is very similar.
Allison Elizondo 8:01
So coming from a test organization, having seen some of the mistakes that the vendors would make, and the things I’d need to kind of coach them on. I kind of have that toolbox in hand. And now that I’m sitting on the vendor side, I can help prepare our engineers and leadership to go into an engagement like that with the right kind of attitude, really the right the right mindset.
Allison Elizondo 8:26
And that’s helpful, because I’ve got that really kind of compassion for the vendors that I saw go through some interesting challenges with NSS. And now that I’m on the other side, I can kind of coach according to that, you know, that that context.
Simon Edwards 8:41
So if somebody came along and said to you, “I’m going to make a new security testing firm, give me some advice, what should I do to make you that potential client, kind of relax and trust me and participate properly?”
Allison Elizondo 8:56
Well, the first question is, who do you think is your client? And how are you monetizing? Right? So you kind of have to figure out in your head, who’s my customer? Is it the end user? Is it the vendor? It really should be the end user, if we’re really engaged in a virtuous cycle, we’re starting with the customers needs always. And that should be part of your mindset, as a vendor. This is about the customer. It’s not you at the end of the day.
Allison Elizondo 9:20
But I think that’s just a good way to kind of level set before you go into an engage, you know, to to kind of build out a test organization. Who’s my customer? Once you kind of figure that out that it needs to be, you know, the end-user versus the vendor. You figure out how you want to kind of monetize and build out that business.
Allison Elizondo 9:39
And then I would say, get a team together, call it your, I don’t know, vendor experience team, your VX team, because the vendors experience is very important, just as important as the, you know, the end users that you’re helping. You need to provide a good experience for the vendors. So make that a focus as well.
Simon Edwards 10:00
okay, so when you know, we’ve worked together before, and I think you’ve told me it wasn’t a horrible experience. What did we do right? What would you recommend other people copy? To make it a good vendor experience?
Allison Elizondo 10:13
Yeah, well, I think you guys do a great job of just being open and transparent. Even if, you know, let’s say there’s something about a test that you you haven’t quite perfected, you’re honest about that – “we’re working on that. Here’s, you know, here’s some ideas we have for tests coming up,” and you engage the vendor community in that conversation.
Allison Elizondo 10:32
I think it’s very important to make the vendors a part of that kind of journey, when it comes to the tests themselves. Obviously, you don’t want vendors building out your tests or influencing them so in a way that’s, you know, possibly gaming the system for their own benefit. But, you know, I think it’s important to kind of bring them along that journey, and you guys do a good job of that.
Simon Edwards 10:52
I was gonna ask, you know, because we do talk to everybody, we talked to the end customers, you know, the people that buy the stuff, but we are also heavily involved in the vendor community, does that not kind of make us inherently biased in some way? We’re kind of in the old boys’ club, you know,
Allison Elizondo 11:10
Not looking at it that way. I see it more as collaboration. And I think collaboration is very important. Because at the end of the day, aren’t we all fighting the same battle, like the same criminals. We’re trying to do the right thing together. And I think it’s important that we collaborate. You know, I think collaboration versus partnership is a little bit different.
Simon Edwards 11:33
I think it helps that the problem we’re all trying to solve is so hard, that no one company or person can do it on their own and I think even vendors collaborate with each other, because the challenge of cybersecurity is so insane.
Allison Elizondo 11:48
Absolutely, yeah, big problems can’t be solved in a silo, you know, you we have to work together.
Simon Edwards 11:54
And that’s true in many areas of life.
Allison Elizondo 11:55
Well, what about right now with, with COVID, you know, these different around the world, these organizations that are working together to help solve this giant problem that we all have?
Simon Edwards 12:05
Yeah, and you know, there’s still some people, lots of people who have got a big conspiracy theory about that haven’t they? They just don’t trust vaccines in the same way that we could do the most perfect unbiased test in the world. And the minute the report comes out, the forums are full of people saying, “Oh, it’s all corrupt and terrible!”
Allison Elizondo 12:22
Well, like, it’s, it’s something else. Yeah. And I and I find against that too, in my role, if that’s one of those little subtle things that you do have to address with like your sales teams, you don’t want them, you know, disparaging or or talking down about a test result or you know, things like that, it’s it’s, it’s important as well.
Simon Edwards 12:45
I know that there’s an organization that many people won’t have heard of called the Anti-Malware Testing Standards Organization or AMTSO – if I had my, my time, at the beginning, I would have named it differently. But they, they’ve got a testing standard, which is quite interesting. And it’s to do with transparency.
Simon Edwards 13:05
So they don’t tell you technically how to do a test. They tell you how to kind of behave around a test, which is to sort of… the way I summarize it is, say what you’re going to do, do it and then be prepared to prove it. But not very many testers follow that. Have you any thoughts on why that might be?
Allison Elizondo 13:28
Well, you know, I can speak from my own experiences at NSS labs, I think they’re they were, they were very hardcore into being the the hacker and thinking like an adversary, and they’re not going to be transparent, necessarily, and be forthcoming in their methods. “Hey, we’re going to attack you, here’s how we’re going to do it!”
Allison Elizondo 13:54
And so NSS Labs would sort of of the mindset of “Well, that’s not how hackers behave. So that’s how we’re kind of going to lean in to our own kind of approach is by being very closed.” Yeah.
Simon Edwards 14:08
Yeah, I think that’s a really interesting point. And I think it, it does illustrate how some of the testers have misunderstood the goals there. Because what what you can do is you can say, we are going to test the product, and we’re going to attack it. And we’re not going to tell you how we’re going to attack it. And actually, that’s a good enough description. You don’t have to outline everything that you’re going to do in detail.
Simon Edwards 14:30
And actually, one of the good things about the Standard is even a really bad test can pass because so long as you say what you’re going to do, even if it’s completely arbitrary and stupid, the fact that you’ve said it is enough. And the point is that then people can read that report and read the methodology and go, “this is stupid. I’m not gonna give any credibility to the results.”
Allison Elizondo 14:51
Well, yeah, and I think if we all follow a Standard that you know, a customer and user can also understand. If we’re not following standards, and let’s say someone’s running a test, that they’re giving no visibility on how they do all their testing whatsoever, why should the customer trust the results? You know, it has to be transparent. AMTSO helps to kind of keep everyone within within those healthy kind of guardrails.
Simon Edwards 15:19
Yeah, I think that’s right, you know, you’ve got, the vendors are always held in a bit of suspicion, you know, do they really do what they what they say they do? I think the same should be held against the testers. You know, why should we get away with a free pass and not to have to explain ourselves?
Allison Elizondo 15:35
I completely agree. You know, I think, for example, an MQ on test labs would be interesting, or some sort of, you know, comparison. But yeah, AMTSO does a great job of kind of keeping one on keeping everyone on the level. But yeah, it’s, it’s, it’s difficult. Why should a customer trust a test lab that is completely opaque in their methods. But they do! They trusted… an organization that was rather opaque in their methods for a long time.
Simon Edwards 16:05
I think there isn’t a lot of competition. I think that’s part of the problem.
Allison Elizondo 16:09
Hmm!
Simon Edwards 16:10
Not much choice!
Allison Elizondo 16:11
Absolutely. Well, it’s hard. I mean, it’s, I’m sure you could testify, it’s a difficult business to be in and, you know, you have to really be passionate.
Simon Edwards 16:22
It is and then you bring analysts into the mix,
Allison Elizondo 16:25
To me testing is that you know, to answer that question, okay, does it work? And then the analyst answers the question, does it work for me if I’m an enterprise?
Simon Edwards 16:34
Right, and that actually brings us quite nicely to, to how does security sell in the market? Because the analysts are really important. They sit right in the middle of the whole process, don’t they?
Allison Elizondo 16:46
Absolutely. Yeah. And they are trusted 100% by these these companies.
Simon Edwards 16:52
Yeah. And we talked to the main analysts, I won’t necessarily say their names. But essentially, they have opinions based on the deep research that they do, and the conversations they have with the vendors. But they don’t generally get hands on with the products. And so the main ones will use us to be kind of their, their external lab, if you like, and give them given data, and they really, really care about the details.
Allison Elizondo 17:16
Interesting. So how does that I guess, how does that work? How did the analysts connect with these test labs to kind of… do they vet them out? Like, what’s your experience there?
Simon Edwards 17:26
I think probably, it’s word of mouth initially. And then as as the AMTSO thing kicked off they started, certainly Gartner started, to assess the testers based on their transparency. And there was at one point, adherence to the AMTSO guidelines or Standard was one of the criteria. And SE Labs did well on that one, and nobody else really did.
Simon Edwards 17:53
But there were other things like, you know, do they publish methodologies? And how detailed are those methodologies, that kind of thing? Because, again, you can’t you can’t say that one test is necessarily better than the other, but it can be better described and conclusions can be more realistic, in some cases.
Simon Edwards 18:12
So, Allison, I’ve never worked for an analyst or particularly with analysts, how does it work? Do they talk to customers and vendors in certain ways? How does where does the money and the time flow?
Allison Elizondo 18:27
They do talk to customers, and that’s the a lot. And that’s why it’s so important that, you know, vendors understand that this is the way that analysts work. So customer who’s making a buying decision, who’s probably gone through your your sales cycle, and, you know, depending on where they are in their in their buying process, they’ll they’ll reach out to a Gartner or sometimes a Forrester.
Allison Elizondo 18:51
But oftentimes, it’s Gartner and ask to speak with an analyst and get their opinion on whoever the vendors are on their shortlist. And that’s called an analyst inquiry. And those are not inexpensive. Normally, this is a this is an engagement that a company can use if they’re already like a Gartner client. They’re not normally just one offs. So yeah, it’s a paid engagement. It’s called a inquiry and they have 30 minutes to talk to an analyst. And that analyst, depending on what the question is, will advise them accordingly, based on their, you know, their needs, their environment, their problems, all that good stuff.
Simon Edwards 19:29
So we can see that it’s not just as simple as reading some reviews and then spending millions on endpoint and network security. We’ve put these diagrams and notes in the show notes of this episode. But we have a concept of how cybersecurity sells the way that people think it does. And then with our kind of inside industry knowledge how we think it really does. So generally speaking, what people think happens is that vendors talk to the analysts.
Simon Edwards 19:58
As Allison just said, the analysts talk to the media, computer magazines, business magazines, websites, that kind of thing. And that information flows to the customer one way or another who then go and make a massive multi dollar purchase at a vendor. But it’s not quite as simple as that. And then reality is a real web of time and money, and misinformation and, and back briefings and all sorts of things flowing all the way around.
Simon Edwards 20:28
So when the customer does spend a big chunk of money with a security vendor, lots of other things have happened in the background. So for example, a pretty common situation would be a large bank is thinking about switching from one firewall vendor to another. So they don’t just listen to the two firewall vendors, they go and talk to other people.
Simon Edwards 20:48
They’ll talk to Gartner and Forrester. In fact, in many cases, when I’ve been involved in these things, their shortlist has been, specifically the Gartner quadrant, the Magic Quadrant where different products appear with different strengths and weaknesses. And if it’s in the top right hand side of the quadrant, then it’s considered to be good in all sorts of different ways. So what we find is they’ll say, well, we’re thinking of vendor x, and y. Can you recommend anything else, Simon and team? But it’s got to be in that top right quadrant.
Simon Edwards 21:19
So already the analysts are heavily affecting the buying decision. And then you have what most people call a POC or a ‘Proof Of Concept’, where the customer will talk to the different vendors involved, get hold of the products, the vendor will do some tests and show that it works really, really well. And then the buyer, the customer has to make a decision.
Simon Edwards 21:42
But they can’t really trust that because each vendor is only going to show their product in the best possible light. And at that stage, increasingly, what we’re finding is the customer themselves are coming to us, and asking us for like a second or third opinion.
Simon Edwards 21:58
So we will then do our own POC, a private test, using targeted attacks and all sorts of cool hackery type techniques. And we will explain to the customer how well the product works. And one of the benefits of that is we don’t just do a general test using default settings, we can use the bank’s own configuration. And in some cases, they may even have multiple products installed, in which case, we can test the product that they’re thinking of using against the one that they already do.
Simon Edwards 22:29
But there can be other other products in that security stack already there. So one antivirus might work really well for one person. But you might not even need half of its features, if they’ve already got some other products in place, and they’re not going to change that. So working directly for the customer is quite a new thing I think for for any test lab, because a lot of these guys already have their own in house labs where they can do their security testing. But cybersecurity testing is so hard. So specific, that actually involving a third party is quite useful.
Simon Edwards 23:06
And in some cases, we’ve come in and had a different conclusion to the in house team. And it’s been eye raising. And they’ve actually not only thrown away the product that we’re using, but in one case notoriously ask for a refund. So I think third party testing has a really big and increasingly important role in helping customers make those huge, buying decisions where it’s multiple millions of dollars. And that is one way. I mean, Allison you said earlier on about how do you monetize? That is one way where a tester can fund the research and the work that they do without taking money from the vendors directly, which can seem problematic.
Marc Briggs 23:47
How is the media involved?
Simon Edwards 23:51
The media generally listen, I think to the vendors. So the large security companies, the antivirus guys, whoever will go out and start talking about the latest threats, the latest viruses, that kind of thing. And the assumption is always well, if they’re talking about it, they must be able to protect against it as well. So the stories will be written about the latest threats. And there’ll be spokespeople from the media there. And I think the hope is it’s a thought leadership thing that when a CISO and his team or her team come to buy a large set of products. In the back of their minds. They’ve already heard Fred from vendor a talking about bad things. So they must be competent at dealing with that.
Simon Edwards 24:34
In the same way that you have threat research teams at vendors discovering all sorts of really cool stuff. And you think why are you writing about this? It kind of goes down to competition, why are they giving away their secrets? But these aren’t actually the people that work on the product. So if I was a firewall company, I would actually hire a bunch of weird security guys, and put them in a room and get them to write blog posts about what’s happening out there. But that information doesn’t always go into the product, that’s just to make us the vendor look good and switched on.
Simon Edwards 25:03
And we’ve got a really good threat intelligence kind of handle on the world. So don’t believe the media. What I found, I used to be a journalist for many years, 20 years, and I covered computer security. So I think I understand it reasonably well, hopefully, because I run a company that tests it. And when I see the media covering cybersecurity, and particularly the mainstream media, actually, it’s a bit wrong. Often, and that makes me worry that there are areas that I don’t understand this depth, where I trust the media, I’m definitely not anti media, you know, journalists are under a lot of pressure to get stuff out. And they do do their best to understand it. But I think with most subjects where there is depth, the subtleties always get lost. And sometimes the subtleties are really important,
Allison Elizondo 25:55
Gosh I agree. So often, you hear the media talking cybersecurity, and you’re almost like Eurgh! Cringing, because you can tell they’re just struggling to understand what’s coming out of their mouth.
Marc Briggs 26:05
We also see specific stories, because there’s so many because there’s so many different media outlets, available now with connections globally. And journalists under such a pressure to get articles out, we we’ve found that we’ll write an article, and then within 24 hours, that has multiplied exponentially around the world, because the journalists have just copied what we’re saying. And it doesn’t take. I mean, in our case, we’ve got, we backup all of our information with our test data. But it wouldn’t take much for just a interested amateur to write an article and make claims to get picked up in the same way. And then suddenly, you’re perpetuating misinformation. But unwittingly, as a journalist.
Simon Edwards 27:10
actually, that’s a really interesting point that when when all the vendors and the testers got together years ago to work out, how could we make testing better? One of the big enemies that was perceived were these random internet reviews often on YouTube, when no one really knew who these people were. Um, so they take some viruses, whatever they were, and scan them with three different antivirus products and say, well, Symantec is obviously rubbish, and Kaspersky is brilliant, or vice versa, whoever was behind those reviews.
Simon Edwards 27:40
So yeah, you’ve it’s that lack of transparency again, isn’t it? If you write some kind of weird article that’s not very well done about testing. And it gets picked up by journalist without double checking. And that’s a disaster. And luckily, we have started to notice that some media outlets are paying attention to certifications and, and methodologies and things. So there’s a big one in the UK called IDG. They do a lot of computer magazines over here. And they only or mainly trust our results because of our AMTSO affiliation. So no journalist is going to sit down and read an antivirus testing methodology. That’s, there’s too much to hope for. But having a badge if you like saying “these guys are open and transparent”. That definitely helped.
Allison Elizondo 28:29
Yeah, and again, it goes back to why test labs like SE Labs are so critical, because it’s easy to pull the wool over someone’s eyes if they don’t really know what you’re talking about, you know, if they just shake their head. “Yeah, yeah, that sounds about right.” So we need labs like yours to come in and actually kind of put those products through the wringer.
Simon Edwards 28:54
It sounds like security testing is a critical part of keeping everyone safe from cyber threats. But we’re testers or ex-testers in Allison’s case. So what are the real customers, the ones who spend literally trillions each year on cybersecurity, what do they think?
Simon Edwards 29:10
Our next guest is Nabil Khokhar, a specialist security adviser who’s worked with the largest companies in the world, helping them buy security products. Thanks for joining us Nabil. Could you tell the listeners what you do?
Nabil Khokhar 29:25
Yeah, I mean, so basically, from from my perspective, mean, DarkGuard, we provide boutique cyber security strategy and solutions to to organizations that require services. And what we found is that there’s a real specialist need for product and vendor selection. I think customers are getting to the point now where they’re able to see past snake oil and understand that Oh, you know, it’s not just about the branding.
Simon Edwards 30:02
Are you suggesting that cybersecurity has an element of dishonesty in its marketing?
Nabil Khokhar 30:08
I would say… I would say so. I would say so. And the reason for that is because I’ve personally been burnt at many companies, organizations, where as an employee, where we’ve done the analysis, we’ve gone and bought the reports, you know, we’ve bought the industry specialism guidance, and you put it in your environment, and it just simply doesn’t work.
Nabil Khokhar 30:32
So how would you deal with that? How do you go back to your C-level and say, yeah, we spent 50,000 evaluating this. And actually, you know, our analysis shows us that this isn’t what we should buy, even though this is what you want us to buy?
Simon Edwards 30:52
Why do you think? Or how do you think they come to the conclusions that they should buy product x?
Nabil Khokhar 31:00
Peer network, and in the pre-COVID days, the glamorous world of conferences, and events, where, you know, hundreds of millions, billions were spent on marketing. And we’ve seen a paradigm shift in the last year or so, where you just don’t have that ability to execute anymore. You know, now it’s, we speak over zoom, you give me an environment, and I play around with it with a pre sales person. And then I make a decision, which, to me has brought a lot more transparency and honesty, to the game.
Simon Edwards 31:44
So if someone is sponsoring a sports team that no longer gives the the heavy implication that their security products work properly.
Nabil Khokhar 31:53
Exactly. Yeah, exactly. Now, you know, I don’t need to spend, I don’t need to do the the dance and pony show now. It’s quite easy to get a cloud first platform, get given a demo account, log in, play with it and see what see what I can see.
Simon Edwards 32:12
So if you if you go to RSA conference, and you think in the back of your head, you know that the top three companies you want to maybe be involved with? Is it Can it really be down to who puts on the best party and has the most champagne? That could be enough to sway and go? Well, I think they’re probably all about, right. But they, that sales team were really nice to me.
Nabil Khokhar 32:33
What that showed was the notion around having the financial backing, so you think to yourself, Well, if you’re putting across the show, then maybe you’re going to put that much effort into customer success, you’re going to put that into research development. So it shows that the money is that the VCs are satisfied. So you know, you clinched your your series C Series D and and now you were three, 5 billion, whatever that is at a conference kind of had some kudos. Now we don’t think of it because we’re not wowed by the fireworks, the shows and Imagine Dragons coming up on stage. Now, now it’s “all I’ve got is the platform, I go into how it works.”
Simon Edwards 33:24
It’s really I mean for the Emperor to have apparently wearing clothes, you have to convince some VCs and then by proxy, everybody believes that you’ve done a good job.
Nabil Khokhar 33:34
Exactly. And if you look at it without calling out names, you know, a lot of the the organizations that we aspire towards utilizing or will have opinions on, they’re heavily backed VC outfits, right? To the point where they’re installed on the board, you know, you can’t go past even in today’s setting a web conference or a blog post without them calling out their investors. So so they influence it heavily. They really, really do influence it.
Simon Edwards 34:11
We I tried to get involved in the venture capital world. And one of the things I ran up against was they said, Well, we’ve already decided to back this horse. So we don’t want a tester coming along and telling us that the technology doesn’t work, because we’ve made that decision already. Can we trust the teams? We think that Fred will take this into a multi billion dollar company, we don’t really care if the technology does what it says it does.
Nabil Khokhar 34:36
Yeah. And the reason I feel for that is the reason for that note, or feedback is because a VC’s emphasis is on exponential growth, right? So if you’re dealing with anyone in cybersecurity, they know what the capabilities are and they know what they could potentially get it to. And, you know, some of them have a strategy. So some of them may already know that the earmarked for an acquisition or an IPO, or maybe to basically go back to private, you know.
Nabil Khokhar 35:15
If we call out some of the large private equity organizations out there that own, you know, 70% of the cybersecurity productions, today, there’s a strategy behind it, and the CEO or the product can influence that I’m afraid. I’ve seen that myself. Yeah, you know, we do we see advisory. So we get the same feedback, as you saw,
Simon Edwards 35:37
When we’ve worked together in the past, with a large, you know, very large, one of the largest organizations in the world. And, and yet, despite all of this, we still have customers saying, “we’re considering these three or four vendors, because they’re all in a quadrant of an analyst report.” So aren’t the analysts enough, you know, why do we need anything else?
Nabil Khokhar 36:02
Well, if you look back on our assignment, Simon, this was exactly the point, you know, we, we trusted the quadrants, we trusted the analysts view, then once we put it through your lab, we actually saw what it was really doing. And that’s not a fault on the product, it may not be a fault on the analysis, but it just wasn’t the right decision for us. So that decision basically helped us with our security posture, it helped us with our budget. And basically, ensured we didn’t purchase something to be essentially a beta tester, you know, in a large corporate environment.
Simon Edwards 36:43
And actually, that’s a really good point is that, you know, one size doesn’t fit all. And so an analyst can say this vendor is going in the right direction, the vendor can say this product does what we’re saying it does. But you as a customer might have a slightly different twist on how you want things to go.
Nabil Khokhar 36:59
Exactly. absolutely. Because we look at, we had a report from a lab, one of your previous competitors. And you know, it scored very, very highly. But for us in let’s say, the oil and gas space, this was highly specialist, and it couldn’t satisfy the use cases.
Simon Edwards 37:21
I remember having a discussion with lots of vendors about what customers want. And they had lots of preconceived ideas, which they try to impose on the way that we test. And then I remember when I spoke to you and your team a few years ago, your requirements were the exact opposite of what the vendors were saying “customers” wanted.
Nabil Khokhar 37:43
And the reason for that is this particular organization had a very forward thinking view on on cyber security, because it was an environment which was constantly attacked. Everyone, and anyone wanted to get in.
Simon Edwards 37:58
Huge amount of money at stake!
Nabil Khokhar 38:00
Huge amounts of money at stake. $220 billion company. So there were challenges. So our view was more, you know, bottom up. So start from the bottom layer. Look at it, look at how a product like that could work in all of our ecosystem. And not just the white collar worker. You know, we our scenario was my I would say, personas were much larger than white collar blue collar worker, you know, we had factories, we had industrial assets, you know, logistics, you know, how do you operate on the sea?
Simon Edwards 38:44
Right. And it felt like you were one of the world’s biggest small companies as well, because you had so many very small branch offices spread out all over the world.
Nabil Khokhar 38:53
Correct! Yeah. And, and this was one of the major problems that we identified when a simple requirement of reporting. So when we looked at the product with with your evaluation, as well, we found the very basic requirements for reporting could not be captured, we couldn’t execute those.
Nabil Khokhar 39:15
And the product that we had currently run, whilst it wasn’t perfect, it was giving us the insight into where we were going wrong and what we needed to do. And then coming up with AI and machine learning, say it’s going to fix all your problems. Well, you know, how am I supposed to take that and put it into a report and go back to the board with it?
Simon Edwards 39:37
Yes. we’ve just finished some work with a quite a well known bank, and they were looking at changing the incumbent solution to something else, or maybe keeping the incumbents. So it’s similar to what we’ve done in the past. And what we found was their incumbent wasn’t actually doing anything at all like they thought it was. So not only are they removing it, they’re actually asking for a rebate as well.
Nabil Khokhar 40:02
That’s interesting, because I’ve actually seen that trend with a few others, where that secret sauce has turned out to be just very, very simple in practice and doesn’t warrant the marketing or the price that was applied to it. So no, we’ve seen the same to be honest.
Simon Edwards 40:22
When you … I know that you’ve had lots of experience engaging with, for want of a better word, third-party testers, are you able to kind of think about what characteristics of those engagements separate the ones that you would want to deal with again, and the ones that perhaps average to poor?
Nabil Khokhar 40:41
I would say that the ones that offer us transparency in the testing process… I mean not not having to go into intricate details, to code, but understanding what you’re testing and the rationale for testing it. And then, the other one is a controversial point, it’s around the vendor relationships.
Nabil Khokhar 41:02
So we found that in the testing community, you know, a lot of name dropping occurs. And once you you’re engaged within the community, you’ll hear from the vendor side, and you’ll be thinking, “well, how much truth is in that report?” And that’s where, you know, sometimes I have a problem with this industry, or this part of the industry, where, you know, of course, everyone is in it for business, everyone is in it to make money. But same time, you know, there’s a, there’s a commitment and a duty to ensure that we all play as one, even if it’s vendor A versus vendor B, we’re all dealing with the same issues, right? Where cyber security got to a point of you know, near, well, with crisis levels, if you look at the Solar Winds attack.
Simon Edwards 41:59
Yeah, I mean, I’m thinking that I don’t know much about the pharmaceutical industry or even your world of natural resources. But in the cybersecurity industry, the problem is so great that there are conferences where people from competing companies get together and share techniques and tools and approaches. I think it’s almost insane because you think they’d be competing? And of course they do. But the problem is so massive that they have to get their heads together.
Nabil Khokhar 42:26
Yep. Yeah. And that case, in point is what I personally appreciate. Because when when you’re when you’re going to market for testing, the provider bashing, or the competitor bashing, you know, we don’t tolerate, it’s, you know, we play as one we’re coming to you because we see a particular competence, we want to work with it. But, you know, we’re not going to stand here and hear you criticize how another organization tests, because that that’s not we don’t want to go down that route,
Simon Edwards 43:00
So you’ve had some bad experiences, have you, working directly with testers? So without naming names, or being too specific, what are they done wrong? What’s got your back up?
Nabil Khokhar 43:12
So we found on several occasions that the recommendation for a particular testing organization came to us in collaboration with the sales cycle. And basically, it comes out that “Oh, look, we recommend this as an independent, fully independent thing. And go ahead, and you know, that’s us against it, we’ve got nothing to hide,” then what what transpires is three, six months later, you’ll be in a peer discussion. And you’ll find that, you know, your information was played against the other.
Nabil Khokhar 43:52
So basically, the report which, you know, may have been in draft was not, let’s say, fully approved by us as a customer. Some of that data was already shared with other people who are interested in a particular product. And we didn’t really appreciate it. That’s not what we signed up to do. It was not part of the NBA wasn’t part of the engagement. So we saw that as shady behavior. And then when we went on to do an additional further independent test, we found evidence that basically this was being configured against a particular standard to make this product look better than it was.
Simon Edwards 44:35
So the test was being biased in favor of a specific vendor.
Nabil Khokhar 44:39
Yeah. Absolutely. Yeah.
Simon Edwards 44:41
It doesn’t take much of a stretch to imagine there’s maybe some commercial activity.
Nabil Khokhar 44:45
There was yes, there was and to be honest, they did disclose that later and basically, you know, came out with a report on why it happened, how it happened, but, but at that time, the trust had diminished.
Simon Edwards 44:59
Yeah. I mean, you got bitten there because you discovered it. I mean, how would you, if you well, we work together. But you know, if you were like, starting again, how would you kind of work out who would be best to deal with? Who Can you can you trust from the off?
Nabil Khokhar 45:15
So I would say, not to be biased with you Simon, but my expectation would be the way that you’re quite public with your reports and your analysis techniques. You know, if we were going to market, I would expect them to behave and act in the same way that your organization does. So we see the full transparency.
Nabil Khokhar 45:38
Secondly, customer engagement. So we would like a form of I don’t know whether you call it an onboarding call or what, but a discovery call to maybe even get to meet some of your your testers see how they think. Because it’s, it’s apparent that this is becoming such an important aspect.
Nabil Khokhar 46:03
We know that cyber security spend has to go up. I’ve seen at least two or three organizations already this year, where they’ve gone to the board and asked for more money for protection. So the testing space is very, very critical. I think selecting the right partner is as important as selecting the right product.
Simon Edwards 46:26
When you come to buy a product, say you’re thinking of changing your firewalls or whatever, is it a POC it’s called? Is it proof of concept? What does POC stand for?
Nabil Khokhar 46:37
Yeah, proof of concept. So the idea would be that you take the box A from said vendor, and then basically bring it into your environment where you’ve mapped out the requirements, and you run the POC as if it was your own.
Simon Edwards 46:54
So when you do that, when you’re comparing a firewall or an endpoint product, why, you know, why isn’t that enough?
Nabil Khokhar 47:04
Because you want, it’s a relationship, it’s a journey. So another case, an example would be that if a product is too perfect from the outset, the lifecycle of it, you know, that you’re going to invest for three to five years on that? Does it? Can it really run this marathon? Or have they provided you everything on day one, and then there’s no improvement, there’s no method for customer success, there’s no method to turn a feature, take feedback to turn it into a feature that gives me to the wider community. That is very, very important. And, you know, we found that select few vendors have invested a lot of time, effort and money into this. And we’ve seen the benefits, we really have seen the benefits.
Simon Edwards 47:55
And when you run a POC, you are in the hands of the vendors involved, aren’t you seeing threats that are used almost certainly going to be ones that they can detect?
Nabil Khokhar 48:03
Yes, yes, that is one issue. However, you know, what we found is if you do a good job of formulating your use cases, without the vendor, to really work with your internal operational teams, and identify the hazardous situation, you know, what, what is that problem today? What Why do we need this? Why are we going out and buying this? saying, Oh, we need to enhance our security posture?
Nabil Khokhar 48:31
It’s not good enough, right? Pretty much every product will enhance your security posture, because it’s new by default. But how do you live with it? You know, how do you maintain it? How’s the relationship? And how structured is the vendor? You know, we’ve seen quite a lot of change with some of the large players, right, recently, you know, one started with them, and they were they, you know, we went from being a public company back to a private company back to a public company, you know, think of the rigmarole that that causes a customer.
Simon Edwards 49:05
Yes
Nabil Khokhar 49:05
Think of some of the large acquisitions, you know, people on paper, the product may have been great, but because the organization’s changed, it’s killed the supply chain, it’s killed the support model. And people are trying to get out.
Simon Edwards 49:21
I think, as well that some products do develop, but maybe the customers don’t notice. So you might have a really good endpoint product that is keeping pace with all the threats are the current trends in attacks. But someone comes along says, hey, there’s this new thing that’s exciting. And what you sometimes see are companies keep adding tools on top of tools on top of tools, rather than really understanding what they’ve got. And you’re training a team to work with specific vendors, firewalls, for example. You know, have a look at the features that they’ve got and are introducing rather than going well, there’s a next gen thing that claims to solve today’s problem because I You say they may not solve tomorrow’s problem and you end up with 13 different vendors in your infrastructure.
Nabil Khokhar 50:06
Absolutely, absolutely. That’s why the R&D element and customer success should be as equally important to functions of said product. It’s a package you’re buying a package, you’re not just buying the product.
Simon Edwards 50:26
Please subscribe. And if you enjoyed this episode, please send a link to just one of your close colleagues. If you want to join the DE:CODED community, and access private content, including our monthly executive briefings, apply at DecodedCyber.com/circle. And that’s it. Thank you for listening, and we hope to see you again soon.
Peek further behind the curtain with DE:CODED Circle.
If you would like access to exclusive, private content from the security testers at SE Labs, please consider applying to join DE:CODED Circle.
DE:CODED Circle is a moderated, vetted community built with the goal of sharing threat intelligence and business-focussed security knowledge to responsible peers.
Feedback
Please send your comments, questions and concerns to info@decodedcyber.com.
