How do I measure the success of my training program?

October 31, 2013

An information security training program is crucial for ensuring and maintaining a good security posture; in order to effectively manage this program you have to be able to measure it. This article introduces a concept recommended by NIST in their Special Publication 800–16, for evaluating training effectiveness.

Measuring the overall success of a training program is broken down by NIST into the following four (4) distinct but interrelated measurements. As such, success or failure on one objective influences the outcome of other objectives.

  1. Learner satisfaction
  2. Learning/ Teaching effectiveness.
  3. Job performance effectiveness.
  4. Program effectiveness.

Determining the overall effectiveness of a training program takes into account employee and employer objectives, the delivery vehicle used for training delivery, the usefulness of the training and the ROI for the organization among other options. To obtain these measurements require planning in advance as measurements must be taken before and after, and integrated into the training program.

The following tools are typically used for assessing each training performance objective:

i. Learner satisfaction: This is typically extracted from the use of a post training questionnaire. It collects the learner’s subjective assessment of the training material and delivery vehicle used.

ii. Learning and teaching effectiveness: Observed by measuring participant’s knowledge before and after the training, and by the evaluator being mindful of the overall performance of participants.

iii. Job performance effectiveness: Supervisors should assess and observe employee’s performance before and after the training.

iv. Program effectiveness: The organization should link training outcomes to strategic goals and calculate ROI for training initiatives.

The NIST SP800–16 guideline outlines training requirements for information security within an organization and provides insight into the evaluation process. For evaluation, it highlights measurement tools and provides samples that can be used to get you started with evaluating your training program and its outcomes.

The following list are typical goals for training, each should be measured using all four (4) measurements discussed above in order to ascertain the overall success of a training program:

i. Influencing staff behavior, for example training on selecting a memorable secure password.

ii. Increase technical knowledge, for example training on mobile in-security.

iii. Facilitating the adoption of new processes and procedures, for example adapting an SDLC or a risk based approach for security.

iv. Raising awareness and getting buy-in for specific changes.

v. Increasing performance.

vi. To reduce the occurrence of defects in a process.

For more information please see: NIST Special Publication 800–16, Information Technology Security Training Requirements: Information Technology Security Training Requirements.

About the Guest Blogger:

Andre Harricharran joined Security Compass as a Security Consultant with a Bachelor of Computer Science and a Master’s degree in Information Technology Security from the University Of Ontario Institute Of Technology. Andre has extensive knowledge in technology roadmapping, pentesting, network security and disaster recovery. With 13 + years in the information technology field, Andre brings a tremendous amount of experience to every project he is working on

Previous Article
Business Logic Pitfalls in Trading Applications (Blog Series) — 1
Business Logic Pitfalls in Trading Applications (Blog Series) — 1

Business logic vulnerabilities have always been the elusive unicorn of the application security world that ...

Next Article
5 Common Windows Hardening Misconfigurations
5 Common Windows Hardening Misconfigurations

Over numerous Windows configuration review engagements that we have performed for our clients, we observed ...

Learn how you can use SD Elements to integrate security into software development.

Watch Video