Posted on 2023-05-23 by Matt Strahan in Industry
If you see penetration testing reports for two different systems, one with 10 vulnerabilities and one with 20, which system has worse security?
Unfortunately in this case, the answer is “I don’t know”. How many vulnerabilities does it take to hack a system? One is usually enough.
The useless headline of penetration testing reports
We constantly try to make our reporting better. As part of this, I will take every chance I can get to read other reports for ideas. This includes resources like Public pentest reports, a great list of real world public reports released by notable cyber security companies.
When I’m reading these reports I’m not focusing on the actual content. Often I’m more likely to be focusing on how the content is presented. Does it get the information across effectively? Is it visually striking? Is it concise?
Too often I see in these reports the headline statistic that reads something like “We found 4 high risk vulnerabilities, 3 medium risk vulnerabilities, and 8 low risk vulnerabilities”. Sometimes there’s what seems like a refusal to elaborate further.
What does this mean for how vulnerable the system is? Well if I’m an attacker, I don’t need three vulnerabilities that give me remote code execution, I only need one. If I’ve downloaded the user database, I don’t particularly care whether or not there’s two other ways of doing the exact same thing.
At best, the number of vulnerabilities may give a minor feeling of how long it may take to remediate, but even then it is a loose guide at best. Our Penetration Testing Engagement Guide says:
If you had two different systems, one with 5 high risk vulnerabilities and one with 1, which would you say is better? Based just off that information you can say that the number 5 is bigger than 1, but that 1 might be a design flaw that requires total reimplementation of the system. Simple numbers like that don’t really get across what the results were.
Are bigger numbers a better result?
I’ve heard the argument that a bigger report with more vulnerabilities is automatically better. You are getting more for your buck if you get a bigger report aren’t you?
There are a bunch of simple ways of enbiggening penetration testing reports that go from “trying to be helpful” to being outright cynical and potentially unethical. Let’s put on our most cynical and potentially unethical hat and go through some of them:
- Supplemental information: At the end of our test we have a report that’s only 100 pages long. We need to fill in content somehow. Well it’s a web application test, let’s put in all the methodologies on how we tested and a general guide on how to improve web application security that does not consider the system itself or development framework used.
- Tool output: Add in direct reports from vulnerability scanning tools, including false positives and irrelevant information. This can add another hundred pages.
- Overwhelmingly detailed and menial lists: It’s remarkably easy for us to find out detailed information about their systems. They should know how easy it is. Let’s put in all the info we can find in the report! This could include a list of open services and their banners, a list of pages and API endpoints in each web application, a list of DNS entries…the lists can go on and on.
- Vulnerabilities with no risk: Cryptographic vulnerabilities are a great instance of vulnerabilities that often can’t be exploited or carry no inherent risk. A brochureware site that may be vulnerable to a nation state style cryptographic brute forcing attack? Add it in. Bigger numbers of vulnerabilities are better.
- Separating out instances of vulnerability: You find 8 examples of cross site scripting in the report. Here you have a choice. Should I report this as one vulnerability or eight? Well, bigger numbers are better so let’s report every single example as a separate vulnerability!
- Worst case scenario: We don’t know that cross site scripting can be used to compromise an admin portal, leading to complete control over the system, which can then be leveraged to compromise the entire network, and then be used to ransomware the business until it shuts down. That said, we don’t know it can’t be used that way! Let’s just rate this as catastrophic just in case - after all the higher the risk the better the job we do! While we’re at it, who can really say what’s likely to happen?
- Advertising for products that may fix vulnerabilities: We can run malware on a host system? Let’s put a brochure for our EDR tool inside the report. They didn’t block access? Let’s advertise our SOC. Their security budget is our oyster.
The case for concise reports
I have a pretty simple rule for report quality:
A shorter report that gets across the same amount of information is a better report.
Actually that’s a long sentence, let’s shorten that:
Shorter is better.
when putting forward our report format we tried to make sure our reports were consise and actionable. This meant making decisions:
- Should we have charts with numbers of vulnerabilities? We decided no. While they looked fancy, they didn’t present information in a useful way.
- Does our report need methodologies? We decided yes as they can provide context and are often needed for compliance reasons. They’re always available, of course in our handbook.
- What appendices should we have? As few as possible.
A penetration testing report doesn’t exist to show off how good our skills are, to justify our existence, or to make the client think they’re getting good value for money. They exist to make the client more secure. The more effective we are at providing the information needed to improve the security of the client, the better the report.
The value of a consultant
This brings us to what you should get when you have a consultant review the security of your systems above what you would get with either an automated tool or a “hacker”. The consultant should give you the knowledge and higher level analysis to know what the results mean for your business. This would include:
- The potential risks and ramifications of the vulnerabilities, applied in the context of your specific organisation.
- The root causes of vulnerabilities, including potential gaps in policy and process.
- Where you should go from here. Should you cast a wider net? Should you address your exposure in the cloud? Maybe you need to address business level?
We have a huge emphasis within Volkis on not just being good hackers but being good consultants. Even if you get all the vulnerabilities, if those results aren’t presented in a way that ends up improving the security of the organisation, the test was a failure.
More on penetration testing reports
Alexei recently contributed to a blog post on a similar discussion around reporting tips. You can read it here. You can also refer to our engagement guide which discusses reports.
If you’re interested in penetration testing, get in touch with us.
About the author
Matthew Strahan is Co-Founder and Managing Director at Volkis. He has over a decade of dedicated cyber security experience, including penetration testing, governance, compliance, incident response, technical security and risk management. You can catch him on Twitter and LinkedIn.
If you need help with your security,
get in touch with Volkis.
Follow us on Twitter and
LinkedIn