Posted on 2020-09-30 by Matt Strahan in Industry
I’ve talked about how it’s almost impossible to validate penetration testing results. Are we done then? Doomed to be left in the dark by ineffective testing?
There are other ways where you could figure out whether or not a penetration tester is good or not. We’ve already talked about things to look for when choosing a penetration testing company. I’d like to be a bit darker in this blog post and put my Evilfirm hat back on. Let’s say you’re a penetration testing firm. How could you present as a good penetration testing company but still deliver shoddy work for cheap?
Side channels for telling whether a pentesting firm is good
I gave a good rant about how it’s difficult to tell whether or not you’re getting a good penetration test based on the results alone. In the real world, we’re not going just on the results though. What are some other indicators we could potentially use?
- Report quality: Whether it gives good advice and additional information vs being templated or directly from a vulnerability assessment tool.
- Additional consulting and interactions: Penetration testing should never be a black box where you say “GO!” You should be able to talk with the tester and from that you might get a feeling whether or not they know what they’re talking about.
- Reputation: Talk with other potential customers. Don’t just talk to their references since they’ll only select their happy customers for their references.
- Retrospective looks and comparisons: If you find a vulnerability later on then use that to judge. Maybe throw another pentesting firm at the same scope and compare (though that has its limitations).
- Certifications and memberships: I don’t think certifications like OSCP and CREST are the be all and end all but having them is perhaps better than not.
- Speaking engagements: If they’ve spoken at conferences then they probably have some passion about security.
- Transparent processes: I had to include this one. Let’s say, hypothetically, they have a handbook open to the world that says exactly how they expect their employees to do pentesting…
Sure those side channels are there, how would Evilfirm deal with them?
Let’s put our black hat back on and create a pentesting company that tries to fulfil the above issues at the lowest cost, while not caring about the quality of their work. After all, since companies can’t tell whether you’re doing a good job what’s the point in doing a good job?
Before we get onto that, what does it mean to do the job for cheap? It costs a lot to build a penetration testing team but that’s a sunk cost. Ultimately the day to day operational costs come down to two main things:
- Labour: The penetration tester is paid a wage. You also need to pay for any systems, tools, training, infrastructure, and insurance.
- Time: You need to put X man days against the test. Depending on the companies they might have targets of between 180 days and 220 days for each individual tester. This does not strictly need to match up to the number of days actually sold.
How can we cut down on those two things?
One good tester (and a whole lot of grads and outsourcing)
You need a figurehead of your team that you put in front of customers. This is the person whose bio you include in the proposal and you make sure they have all their certs and training. Outside of this they can do research and speaking engagements because they’re building your rep in the industry.
They don’t do the work, though, because they’re far too expensive for that. You get new, low paid people to try their hand at testing. Or you can just outsource it. For customers who care you can get your figurehead to do the main customer interactions and at least pretend they’re doing the work.
Overscope time, underscope price
It’s far more impressive to say you’re spending 15 days on something than 10. Especially when you can do all this work with such a low cost. The other testing companies have only scoped 10 days! They mustn’t be doing much.
Now, you don’t actually need to do 15 days, that sounds like too much time and costs too much. Just do 5 days or something like that.
No need for unnecessary expenses like tools and training
There are a lot of publicly available tools that the testers can get for free. There’s also a lot of information you can just get online now. Why do we actually pay for tools and training? Also that infrastructure isn’t really necessary is it?
If they want to go to conferences they can pay their own way. Definitely formal courses and certifications are their own responsibility!
Spend in marketing and sales
You don’t necessarily need to do a great job, but it pays to look like you do a great job. I’d advise to get a bit of the money you’re saving by not hiring skilled testers or training the testers you have and instead redirect that into marketing and sales.
In particular, make sure your report looks nice. It’ll distract from the bad content. You might also want to sponsor a few conferences and engage with some research institutions to release some whitepapers. That could be something your one good tester does in his spare time.
Is it a bad thing to cut down on quality?
You might find it strange for me to now argue why this all might not be such a bad thing. After all, Volkis has a clear quality differentiation strategy, trying to pursue high quality rather than low costs. We’re right up for lowering costs through automation and efficiency, but never at the expense of quality. Maybe though this is the wrong way to think about it. Maybe we should be going after low cost, low quality testing.
Those tactics above allow organisations to cut costs and provide cheaper tests. With the free market you’d expect the cost cutting to continue, allowing more penetration testing for the same amount.
And companies only have a certain amount they can spend on security. They want to do what the company is there to do, they don’t have the luxury to spend infinite amounts on their security. If firms can deliver more penetration testing for them, they might have greater coverage than they’d otherwise have. A bad test is better than no test isn’t it?
I’m not sure this is the right way to think about it. The problem with this way of thinking is that you don’t really need many vulnerabilities to get popped in the modern world. You really only need one. Penetration testers are defenders not attackers - the attackers only need a single hole in the wall, while the defenders need to make sure the entire wall is secure.
In my opinion a bad penetration tester is better than nothing, but is it better than an automated vulnerability assessment? Perhaps not. You’re definitely not getting the value that you’d get from someone skilled.
What can we do then?
In the next post I’ll be looking at other industries. Maybe we’ll be able to steal borrow some ideas.
About the author
Matthew Strahan is Co-Founder and Managing Director at Volkis. He has over a decade of dedicated cyber security experience, including penetration testing, governance, compliance, incident response, technical security and risk management. You can catch him on Twitter and LinkedIn.
If you need help with your security,
get in touch with Volkis.
Follow us on Twitter and
LinkedIn