Three crazy ideas for reforming the penetration testing industry

Posted on 2020-10-02 by Matt Strahan in Industry


In two posts I looked at how it’s almost impossible to validate penetration testing results and where an Evilfirm penetration testing firm might cut costs and invest.

As much as we like to think we’re unique, there are other industries that have exactly the same issues as we do. In other industries there’s the situation where you can’t really verify the results because you’re after the skills of the other party. Some do it badly (I still don’t quite trust my mechanic), but others have made great strides in solving this problem.

Could we potentially use some of the ideas from other industries to do things better?

Good writers borrow

Let’s look at how two other industries, the healthcare industry and the financial industry, tackle this same problem.

Doctor, doctor, give me the news

If you think about it, penetration testers are kind of like the GPs of IT systems. We knock around a bit, diagnose some issues, and give recommendations on how best to fix them. How does the medical profession then ensure that doctors do a good job?

Not just anyone can call themselves a “Doctor”. The word “Doctor” is protected, meaning you have to comply with a bunch of requirements. These are governed by the Medical Board of Australia and are quite extreme:

  • Tertiary education (a.k.a. a medical degree)
  • 47 weeks of full-time practice that covers a variety of practices, ratings and recommendations
  • Ongoing professional education
  • Recency of practice
  • Appropriate personal professional indemnity insurance
  • Comply with a code of conduct that includes risk management

Without these requirements, it is illegal for someone to call themselves a GP or medical doctor. The Health Practitioner Regulation National Law Act 2009 enforces this provision as well as the code of conduct and registration standards.

“Doctor” isn’t the only protected term in the healthcare industry. You can’t just go up and call yourself a “Surgeon” for instance no matter how much Surgeon Simulator you have played.

Show me the money

We sometimes forget that penetration testing isn’t just trying to break in, but is really an audit of a system for security issues. The finance industry has really similar potential issues in the annual audits of company accounts.

Registered auditors are governed by the Australian Securities and Investments Commission (ASIC). To be an auditor you have to have tertiary education, 3-5 years of professional experience, and have worked on at least 3 audits.

The auditors themselves will each take personal accountability over these audits. Each audit will include the personal signature of the auditor who was in charge of the financial audit. Audits both public and private are inspected by ASIC for quality. Audits that are found lacking in quality can result in the disqualification of the auditor, being unable to perform further audits. For example, on 20th February 2020 ASIC disqualified one auditor and imposed conditions on five others.

In addition to ASIC oversight, all financial audits of publicly listed companies are made public. I can go right now and download the annual report including the financial audit of banks, healthcare organisations, mining companies and tech if they’re publicly listed. If the audit ends up being wrong, there’s not only legal ramifications, but organisations won’t trust the auditors anymore. This can form a retroactive test if auditors missed financial fraud that was found in future audits.

Three crazy ideas for reforming the industry

These three ideas I’ll be putting forward aren’t new - other industries are already doing them. I’m not even sure they’re appropriate for us, but maybe they could get us thinking about how we might be able to do things better.

Protect the terms “Penetration Tester” and “Penetration Test”

Just like a Doctor has to have the right credentials and training before they can be called a Doctor, we could potentially protect the term “Penetration Tester” and “Penetration Test” so that only people with the right credentials can perform them.

What could be some of the requirements? This could be formal requirements such as degrees or certifications, industry experience or an apprenticeship or a collection of all three.

Additional requirements could be having appropriate insurance in place and signing a code of ethics.

Personal sign-off on penetration tests

A key control in the financial industry is that the firm doesn’t sign off on a financial audit, a single auditor does. This encourages personal responsibility and accountability.

You could very well have an experienced penetration tester do the work, but what if they’re just personally lazy or underperforming. With penetration testers working for a firm, there’s little personal accountability around penetration testing.

Could you have penetration testers personally sign-off on each pentest they do?

This is a tricky idea. You can’t, after all, expect penetration testers to find absolutely everything given how hard and variable the job can be. That said, neither can you expect doctors to diagnose every ailment there could potentially be on the first visit to the medical centre.

There are potential cases, though, where there may be true negligence. The tester might just not do the job, or might have said “yeah I’m an expert in this tech!” when they’re hopelessly unskilled. Personal accountability can mean the personal negligence of a penetration tester could lead to personal impact.

This personal accountability might also just lead to pushback if firms try those tactics around underscoping and using unskilled personnel. It would be quite hard to get people to work for you if working for you is a personal risk to them.

All penetration testing reports become public

This is probably the craziest of the lot. What if all penetration testing reports become public after a certain period of time?

Surely not, people could get them and start hacking the application! Well you wouldn’t make them public straight away, you’d make sure that you fix all the security issues identified. If there are additional recommendations you could include your responses as to why you have or have not chosen to implement those additional recommendations. With all the vulnerabilities fixed and additional recommendations implemented, there’s unlikely to be additional risk there.

For potential customers of penetration testing firms, however, they’ll suddenly be able to look at the past record of those firms. They could see whether the firm has failed to miss issues that should have been caught, or whether appropriate best practice recommendations could be set. The records of penetration testing firms might just be good indications of the quality of work and might just raise the bar for the entire industry.

It will probably raise the bar for organisations as well in their own security. Suddenly their dirty laundry might just become public instead of being shoved under the table. People will be able to make sensible decisions around the security of their information.

I believe it would also be beneficial for the entire industry to have transparency around security issues that are found and fixed. Each new penetration testing report could become a learning point for the entire industry to be safer and more secure.

The maturity of the industry

These are just a few controls that other industries have but information security doesn’t yet have. Infosec is a relatively new industry, having been around for only a few decades. Compare that to the thousands of years that the financial industry and the healthcare industry has been around and you see that we’re barely toddlers.

Maybe by using a few of their ideas we could potentially raise the bar in our cyber security.


About the author

Matthew Strahan is Co-Founder and Managing Director at Volkis. He has over a decade of dedicated cyber security experience, including penetration testing, governance, compliance, incident response, technical security and risk management. You can catch him on Twitter and LinkedIn.

If you need help with your security, get in touch with Volkis.
Follow us on Twitter and LinkedIn