A Hybrid Machine Learning Approach for Red Team Log Analysis

Abstract

Red teaming is a common cybersecurity practice that simulates real-world adversarial cyber operations on defended systems to identify vulnerabilities. Current red team tools often have limited logging capabilities, resulting in insufficient analysis that prevents red teams from receiving real-time feedback and insights after operations. The application of machine learning for automating the analysis of red team operations is severely constrained by the scarcity of labeled, real-world log data. This research addresses this challenge by exploring the potential of using synthetic data to train attack-detection models for Cobalt Strike logs. We systematically evaluate three different training approaches for analyzing Cobalt Strike operational logs: synthetic-only, real-world-only, and a hybrid approach that combines both data types. Our methodology employs a comprehensive feature engineering pipeline that includes both programmatic log generation for creating large-scale structured data and large language model techniques for introducing variety and edge cases. We transform each log file into a high-dimensional vector that includes event types, command verbs, temporal activity patterns, and mappings to the MITRE ATT&CK knowledge base. Random Forest classification models are trained using this feature set to distinguish between successful and failed attack scenarios. By rigorously testing each training approach against a manually labeled ground-truth set of 112 authentic Cobalt Strike logs, we quantify the performance and limitations of each strategy. Our main contribution is demonstrating that a hybrid training strategy achieves 94% accuracy, greatly surpassing synthetic-only models (56%) and real-world-only models (79%). This combined approach effectively addresses both the domain gap in synthetic data and the data scarcity in small, real-world datasets. The hybrid model learns attack diversity from over 30,000 synthetic scenarios while grounding understanding in the authentic structural patterns of real logs, providing a 15 percentage-point improvement over real-data-only approaches. This research offers a practical framework for enhancing limited real-world cybersecurity datasets by strategically integrating synthetic data, enabling immediate use in Department of Defense red team operations and wider cybersecurity machine learning applications.