Finding a qualified penetration testing provider is sometimes like trying to find the treasure at the end of a rainbow. Whether it be capabilities, price, methodology, expertise, partnership…etc.. etc. Trying to find the ‘right' combination of evaluation criteria is not a walk in the park. While there are a number of evaluation criteria to consider beyond this, here is a shortlist of some to ponder.
This Is By No Means A Full And Complete List
Seems obvious, but that's not always the case. Your penetration test provider should keep you informed every single step of the way with scheduled touchpoint conference calls along with some sort of secure online project management portal to enable the right amount of communication. The project management portal should illustrate the phases of the pen test, which phase your project sits in, percentage completed and allow you to easily (electronically) communicate with the consultants and project manager(s). One of the most prominent of its capabilities must be informing you/your team when a new finding has been discovered (app, system, estimated impact, estimated criticality, minor finding info), preferably in near real-time. This will enable your team to respond quickly and accordingly if only to be aware of the findings being discovered to ensure there aren't any surprises when the final report is presented. This can go an extremely long way if your provider allows you to remediate and re-testing during the initial testing period. The primary takeaway here is that constant communication is key to a successful engagement — we see it happen daily.
Unfortunately, there is no silver bullet for teasing this out of a provider, but starting out by asking for a detailed testing methodology can help identify this. Methodologies that align with security frameworks like OWASP, OSSTMM, PTES, WASC, NIST help in this effort. Look for indicators of the firm mentioning keywords in their process or marketing material that may indicate this, such as "manual," "deep-dive," "advanced" penetration testing. Next, ask for a sample/sanitized report and examine the complexity of the findings identified. Do they look too basic like something only a Nessus report might show? Some companies may even offer an estimated ratio of automated testing to manual testing. For example, RedTeam estimates each project averages about 20% automated and about 80% manual, deep-dive, advanced pen-testing (aka our 80/20 rule).
Not many pen tests are done in a 100% pure manual fashion. Of course, there are always outlier situations. Under normal circumstances, there is generally some level of automated tool use, including but certainly not limited to: nmap, Nessus, Nexpose, etc. The winning balance lies in the implementation of manual-to-automation TTPs by the provider. That is to say, automation should be employed by the provider where manual techniques are too inefficient (ie: manually port knocking 65,535 ports for each host is not efficient use of manual time) and manual techniques should be used where best suited for human decision-making (ie: complex/correlated attack plans, kill chaining, post-exploitation TTPs, etc). A quality pen test should involve a lean more heavily toward manual testing TTPS than automated tactics (80/20 rule).
Ask your potential provider what happens when the final report is delivered. Unfortunately, most providers move on to the next customer before the ink dries on your final report and you never hear from them again. However, clients know all too well this is when the real legwork begins by way of carrying out remediation work and implementing compensating controls. Often times, this work can take weeks or months depending upon the number and complexity of findings. You want to be sure your pen test provider will be there to help your team during your remediation process and ensure the team fully comprehends the findings (impact, likelihood, criticality) and is on the right track toward remediation. For example, RedTeam clients often contact us for remediation guidance well after report delivery/presentation and we strongly encourage them to do so. RedTeam provides remediation assistance at no additional fee because we feel the true value of our service cannot be truly realized until we've helped close findings, not just report them.
Once a vulnerability has been remediated by your team, ask your pen test provider if they will re-test your findings to validate that your remediation efforts removed the vulnerability. Some providers offer this while others simply don't. Those that do re-testing will almost always charge an additional fee. Some providers will only re-test if you complete your remediation efforts within a certain time window and will only re-test High findings. Having a time window restriction is a bit like gambling, so be sure you have enough resources pooled up to handle the job within the allotted time. Needless to say, those that only re-test certain severities of findings (ie: Highs or Highs and Mediums) aren't giving you the full picture. Low findings matter too. If they didn't, there wouldn't be a severity for it. So be sure you're okay with only getting certain severities re-tested.
In addition to our free remediation assistance, RedTeam Security provides re-testing for all severities without a time window restriction all at no additional fee.
In which formats does your provider deliver pen test results? Most firms adopt Adobe PDF for report delivery and others may use HTML or Powerpoint. These formats are good for presenting the information, but not always the best option for managing what goes on AFTER the report presentation. What's more, you might be using a software platform for managing risks, such as Archer or Lockpath. A pen test report with even a handful of findings can be a tedious case of copy/paste from a PDF to your Archer or Lockpath app. In these situations, you should expect your provider to deliver pen test results in a format that's more manageable for you, such as XML or CSV, for ingestion into your favorite GRC app. Even if you don't have GRC app, your provider should at least provide results in Excel format enabling an optimized way to manage your findings. In addition to PDF, RedTeam provides report output in XML and CSV as well as tracking remediation statuses to ensure your findings can be effectively managed.
The lure of free checks when you sign up for a bank account doesn't carry the same weight as it did years ago. Times have changed and customer expectations have multiplied tenfold. Therefore having technical expertise, such as industry certs and experience, is automatically expected. That's not to say it is unimportant — it's very important! However, in the customer's eyes security expertise makes an impact only when it is significant and extraordinary. For example, when the provider publishes books, teaches security classes, discovers vulnerabilities (CVE-2010-2028), speaks at security conferences, publishes security tools, is featured on national TV news or is recognized in other extraordinary ways.
First and foremost, this isn't an all-inclusive list — it's merely a starting point for hunting for a new pen test provider or re-evaluating your current provider. We hope that you take this information forward and learn how to realize more value for the dollars you spend on cyber security services.