Sabermetrics for Cyber: Collecting and Analyzing User Activity Data from Ephemeral Exercises

Abstract

The term sabermetrics was coined in the 1970s by members of the Society for American Baseball Research (SABR) to describe how baseball teams use advanced analytics to evaluate talent and maximize performance both offensively and defensively. Sabermetrics transformed professional baseball through its data-driven approach, enabling teams to devise new tactics and strategies for improving individual and overall team performance. The concept of sabermetrics or advanced analytics can also be applied to the cybersecurity domain to improve performance, both offensively and defensively, and to better evaluate talent. To do this, data is needed. Cybersecurity exercises are well suited for providing this data because they are designed to develop critical technical skills in controlled, simulated environments that closely mirror real-world threats. However, preserving data for ephemeral cybersecurity exercises can be challenging because these environments are temporary, and when they are torn down, log data is lost unless deliberate actions are taken to retain the data for future use. This includes all information regarding the actions participants took in the exercise. x`Recognizing that important information can be gleaned by analyzing this data, the Software Engineering Institute (SEI) at Carnegie Mellon University developed a capability to capture a high-fidelity record of user activities during cybersecurity exercises. This paper discusses the motivation behind this development, the insights that can be gained from the collected data, and how the SEI configures exercises used in cybersecurity competitions to collect and store user activity data for future detailed analysis.