Beyond the Black Box: Unraveling the Role of Explainability in Human-Artificial Intelligence Collaboration

Explainable Artificial Intelligence (AI) models have been proposed to mitigate overreliance and underreliance on AI, which reduce the effectiveness of human-AI collaborative tools. Yet, empirical evidence is mixed, and the impact of explainable AI on a decision-maker (DM)'s cognitive load and fatigue is often ignored. This paper offers a theoretical perspective on these issues. We develop an analytical model that incorporates the defining features of human and machine intelligence, capturing the limited but flexible nature of human cognition with imperfect machine recommendations. Crucially, we represent how AI-based explanations influence the DM's belief in the algorithm's predictive quality. Our results indicate that explainable AI has varying effects depending on the level of explainability it provides. While low explainability levels have no impact on decision accuracy and reliance levels, they lessen the cognitive burden of the DM. In contrast, higher explainability levels enhance accuracy by improving overreliance but at the expense of increased underreliance. Further, the relative impact of explainability (c.f. a black-box system) is higher when the DM is more cognitively constrained, the decision task is sufficiently complex or when the stakes are lower. Importantly, higher explainability levels can escalate the DM's cognitive burden and hence overall processing time and fatigue, precisely when explanations are most needed, i.e. when the DM is pressed for time to complete a complex task and doubts the machine's quality. Our study elicits comprehensive effects of explainability on decision outcomes and cognitive effort, enhancing our understanding of designing effective human-AI systems in diverse decision-making environments.