All That Glitters Is Not Gold: On the Effectiveness of Cyber Security Qualifications

The problem of the shortage of skills in cybersecurity is generally considered to be solvable through industry professionalization, competency requirements and training programs. Competency gained through training is often expressed in terms of qualifications. For cybersecurity, the methods used to assess competency vary between qualifications. The importance and variety of assessment in qualifications merits an analysis of the effectiveness of these differing examinations in meeting industry needs.

Knowles et al. reviewed 74 industry-focussed cybersecurity qualifications examinations to see which methods of assessment they use. They found there were five distinct testing methods: “Multiple Choice Examinations”; Narrative Form Examinations”; “Oral Examinations”; “Virtual Lab Examinations” and “Employment History and Qualifications Reviews.” The researchers surveyed 153 industry stakeholders in a variety of roles to see how effective they considered these testing methods to be at measuring cybersecurity competence. The survey enquired about perceptions of which of the tests were considered to be effective and which combinations of testing were economical. Multiple Choice Examinations were dominant, with 60 of 74 of the examined qualifications making use of this method. Furthermore, 36 of the qualification programs used Multiple Choice Examinations as the sole form of assessment. Virtual Lab Examinations and Employment History and Qualification Review were the next most common methods, appearing in 21 and 20 of the 74 programs, respectively. However, these methods were much less often used as the sole form of assessment than Multiple Choice Examination.

It is important to note the examinations within each method vary in how intensive and exacting they are. Virtual Lab Examinations ranged from short focused single task oriented assessments to 72 hour intensive examinations. Likewise, Employment History and Qualification Reviews could range from simply requiring a minimum number of years of industry experience to needing specific qualifications or types of experience.

Survey respondents did not seem to perceive Multiple Choice Examinations as very effective. Nearly half (45.5%) of respondents rating perceiving it as a fair or poor competency assessment method. Respondents rated both Employment History and Qualification Reviews and Virtual Lab Examinations more highly with over 45% marking them as “good” or better.

Respondents considered the combination of Oral Examinations and Employment History and Qualification Reviews to be the most cost-effective combination for cybersecurity competency assessment. Each of these methods also often appeared in combinations considered to be cost-effective. The second highest rated assessment groupings of Multiple Choice Examinations together with Employment History and Qualification Reviews combines assessments considered as opposites in efficiency. This was also the most common combination used in practice, with 13 of the 30 qualifications with composite examinations using this combination.

The least effective assessment method appears to be Multiple Choice Examinations; however, this method is required for 81% of qualifications and is the only assessment method in 47% of qualifications. A large proportion of current cyber security industry qualifications use methods perceived as being ineffective. Reappraising the examination methods that industry qualifications use could help in understanding the balance between testing cost and effectiveness. Understanding the appropriateness of testing and the relative costs for cybersecurity could help in underpinning future competency assurance programs.

You might also like

Keywords