Enacting Explainability: Knowledge Practices Around Generative AI in a Consulting Firm

Abstract

As generative AI becomes embedded in knowledge-intensive work, organizations face a central paradox: AI systems produce influential outputs while their inner workings remain opaque. Although explainability research has largely focused on technical methods, less attention has been given to how organizational actors practically make sense of and govern AI-generated knowledge. This study adopts a knowledge-practice perspective to examine how explainability is enacted in everyday work within a mid-sized consulting firm. The study is based on a qualitative case design involving nine semi-structured interviews with AI practitioners, clients, and an AI expert. Rather than treating explainability as a technical property, the analysis explores how consultants interpret, validate, and operationalize AI outputs in the absence of formal governance structures. The findings show that explainability is enacted as a situated and role-dependent knowledge practice. First, employees assess outputs pragmatically, focusing on whether they “make sense” in context rather than seeking insight into model internals. Second, prompting emerges as a form of tacit expertise developed through trial-and-error learning, functioning as an informal mechanism for influencing and interpreting outputs. Third, actors rely on local validation routines such as re-running prompts and cross-checking information to manage uncertainty and maintain trust. Finally, the absence of internal guidelines results in individualized and uneven practices, indicating that knowledge about AI use is created but not institutionalized. From a Knowledge Management (KM) perspective, the study contributes by conceptualizing explainability as an emergent knowledge practice shaped by situated sensemaking, tacit skill development, and informal governance. It extends knowing-in-practice research by illustrating how organizations cope with opaque digital systems when formal knowledge infrastructures lag behind technological adoption. For KM practice, the findings highlight the need for lightweight governance mechanisms, shared prompting norms and role-adapted guidelines that support collective knowledge development around AI use.