AI Safety Under Uncertainty: Hallucinations and Unpredictable Failures

Abstract

With the rise of AI-supported tools across mission-critical workflows in medicine, finance, commerce, education, and cybersecurity, their errors and incorrect decisions can pose safety risks. In our broader study of safety challenges in AI applications, we identify and analyze various safety concerns related to AI-supported tools, including hidden dangers in AI-generated content, the misuse of AI-supported tools for cyberattacks, and their societal impacts. In this particular work-in-progress paper, we focus on our analyses of unpredictable failures and hallucinations in AI-supported systems. We analyze how generative AI models can produce fluent, convincing, yet misleading results, and how they can contaminate mission-critical applications, such as healthcare/medical decision-making, software development, cybersecurity, privacy/data governance, and workflows with agentic AI-supported systems. We survey use cases across domains to articulate the vulnerabilities. We discuss how agentic AI-supported workflows can amplify small errors into significant damage by automatically feeding erroneous outputs and actions. We also propose practical countermeasures to minimize the hallucinations and unpredictable failures with continuous monitoring and evaluation. In our future work, we will integrate this analysis with other areas of AI safety to develop a comprehensive framework and strategies.