Evaluating Zero-Shot Chatgpt Performance on Predicting CVE Data From Vulnerability Descriptions





AI, ChatGPT, CVE, ML, NVD, vulnerability management


Vulnerability management is a critical industry activity driven by compliance and regulations aiming to allocate best-fitted resources to address vulnerabilities efficiently. The increasing number of vulnerabilities reported and discovered by a diverse community results in varying quality of the reports and differing perspectives. To tackle this, machine learning (ML) has shown promise in automating vulnerability assessments. While some existing ML approaches have demonstrated feasibility, there is room for improvement. Additionally, gaps remain in the literature to understand how the specific terminology used in vulnerability databases and reports influences ML interpretation. Large Language Model (LLM) systems, such as ChatGPT, are praised for their versatility and high applicability to any domain. However, how well or poorly a state-of-the-art LLM system performs on existing vulnerability datasets at a large scale and across different scoring metrics needs to be clarified or well-researched. This paper aims to close several such gaps and present a more precise and comprehensive picture of how ChatGPT performs on predicting vulnerability metrics based on NVD's CVE vulnerability database. We analyze the responses from ChatGPT on a set of 113,228 (~50% out of all NVD vulnerabilities) CVE vulnerability descriptions and measure its performance against NVD-CVE as ground truth. We measure and analyze the predictions for several vulnerabilities in metadata and calculate performance statistics.

Author Biographies

Andrei Costin, University of Jyvaskyla

Dr. Andrei Costin received his PhD from EURECOM/TelecomParisTech and is currently a Senior Lecturer/Assistant Professor of Cybersecurity with the University of Jyväskylä (Central Finland), Jyväskylä, Finland, with a particular focus on system security, IoT/firmware cybersecurity, avionics/space/aerospace security, and some aspects of Digital Privacy.

Hannu Turtiainen, University of Jyvaskyla

Hannu Turtiainen received the M.Sc. degree in cybersecurity from the University of Jyväskylä, Jyväskylä, in 2020, where he is currently pursuing the Ph.D. degree in software and communication technology. His research topic is Machine Learning and Artificial Intelligence in the Cybersecurity and Digital Privacy field.

Narges Yousefnezhad, Binare Oy

Narges Yousefnezhad is currently a Postdoc researcher in Binare Oy (binare.io), a cybersecurity spinoff from University of Jyvaskyla. Prior to joining Binare Oy, the obtained her PhD from Aalto University in Helsinki, where her PhD research topic was on Secure IoT Systems in Product Lifecycle Information Management.

Vadim Bogulean, Binare Oy

Vadim Bogulean is the CTO and co-founder of Binare Oy (binare.io), a cybersecurity spin-off from University of Jyvaskyla. He has more than 20 years of professional and industrial experience around the world, in companies related to embedded devices, manufacturing and factory robotics, medicla devices, telecomunication and telecom billing.

Timo Hämäläinen, University of Jyvaskyla

Prof. Timo Hämäläinen has over 25 years of research and teaching experience related to computer networks. He has more than 200 internationally peer reviewed publications and he has supervised 36 Ph.D theses. His current research interests include wireless/wired network resource management (IoT, SDN, NFV) and network security.