AI for Knowledge-Driven Business Decisions: Capabilities and Constraints

Abstract

This paper presents experimental research evaluating the capability of generative artificial intelligence (GenAI) chatbots to assist business executives in their decision-making processes. With the growing prominence of large language models such as ChatGPT and Perplexity, there is significant interest in their potential for knowledge management (KM) and decision support, but empirical evidence is still lacking. This study conducted over 210 experiments using approximately 900 prompts to assess the performance of ChatGPT and Perplexity across various business decision tasks, including marketing analysis, work shift optimization, sentiment analysis, review of company financial and strategic documentation, and product marketing evaluation. The outputs generated by the chatbots were compared with traditional decision support methods, human expert analysis, and benchmarking data. The results indicate that GenAI chatbots can streamline processes by collecting, interpreting, and synthesizing data into actionable insights. Nevertheless, limitations were identified in handling complex file formats, response variability based on context framing, and the necessity for human validation of outputs. A comparative analysis of ChatGPT and Perplexity revealed distinct differences: ChatGPT demonstrated overall reliability but struggled with tabular data, whereas Perplexity provided more substantive yet occasionally less accurate responses. The findings suggest that GenAI chatbots could enhance decision efficiency through rapid analysis, but a collaborative human-AI model remains advisable to validate outputs and iteratively refine queries. Our study underscores the potential of GenAI for KM in guiding decisions while highlighting the need for further research on evaluation methods, integration with business intelligence tools, and the development of user guidelines for the responsible adoption of this technology.