Why manual testing is worth the extra buck: weighing the pros and cons

Clients often ask for our guidance on whether they need to do a manual penetration test for a specific application or if automated testing suffices. Here I try to briefly describe some of the advantages and disadvantages of each method and explain our preferred approach.

Generally, there are two approaches to application penetration testing:

  • Automated testing, and
  • Manual testing

Security testing engagements may involve either of the above, or commonly a combination of both, depending on the project scope, goals, and available time and budget.

Automated testing normally takes less time and resources to cover a certain scope, however, there are major advantages to manual testing that justifies spending the extended resources and are worth highlighting here.

Firstly, manual testing can cover certain security aspects that automatic tools simply are not designed to address. Logical, multi-step and interaction based security flaws are some examples. Our research shows that even the best automated analysis tools validate less than 50% of security requirements.

Additionally, manual testing can lead to discovery of more complex vulnerabilities. Humans can run a creative combination of attacks to discover a vulnerability out of reach of the automatic test tools.

Finally, automatic tools are only as capable in finding vulnerabilities as the reach of their vulnerability database and rulesets. Vulnerabilities and attack vectors constantly evolve and it normally takes a matter of weeks to months before the tools ruleset get updated. Therefore, relying only on tool-based detection may lead to leaving the most recent vulnerabilities undiscovered.

In conclusion, we recommend leveraging a combination of automatic and manual techniques. Assessments can start by some automated testing to cover a broader scope at a low depth, narrow down the project scope, and pick the low hanging fruit. Testers can then follow-up with a manual testing phase to dig deeper in the areas of most interest or risk.

Previous Article
CMMI for Application Security — Four high-impact implementation considerations
CMMI for Application Security — Four high-impact implementation considerations

Ponemon Institute and Security Innovation recently made public the results of a research study they did to ...

Next Article
Why remediating assessment results might be harmful to your business
Why remediating assessment results might be harmful to your business

Let’s say you’ve just had a pen test or security scan performed on your application. You review the list of...