From ally to algorithm: How disagreement shapes users’ engagement with AI

Advances in AI language capabilities have led people to use it for conversations about relationships, social dilemmas, and moral questions. However, AI chatbots are sycophantic; they tend to affirm what the user says, exacerbating confirmation bias and leading to overconfidence or distorted perceptions of reality. Based on these concerns, developers started addressing sycophancy problems, creating models more likely to disagree when necessary. Here, we investigated what happens when AI models disagree with users in conversations. Participants were presented with one of three hypothetical dilemmas in which they formed an opinion and explained their reasoning to an AI model. They were randomly assigned to an agree condition, in which the AI model was instructed to support the user’s reasoning, or a disagree condition, in which the AI model was instructed to provide refutations. Finally, participants rated AI’s capacity to understand human emotions, confidence in their decisions, and willingness to use AI again. Participants rated AI as less capable of understanding human emotions and as more machine-like in the disagree condition compared to the agree condition. Participants also weighed the AI’s opinion asymmetrically, increasing confidence when the model agreed but not decreasing it when it disagreed. Finally, they reported greater willingness to use AI to discuss issues when it agreed with them. These findings suggest that people’s perception of AI depends on whether it agrees with them, and that they may discount disagreement by weighing its advice less. Hence, creating human–AI interactions may require more than reducing sycophancy.