Interacting with fallible AI: Is distrust helpful when receiving AI misclassifications?

Due to the application of Artificial Intelligence (AI) in high-risk domains like law or medicine, trustworthy AI and trust in AI are of increasing scientific and public relevance. A typical conception, for example in the context of medical diagnosis, is that a knowledgeable user receives AIgenerated classification as advice. Research to improve such interactions often aims to foster the user’s trust, which in turn should improve the combined human-AI performance. Given that AI models can err, we argue that the possibility to critically review, thus to distrust, an AI decision is an equally interesting target of research. We created two image classification scenarios in which the participants received mock-up AI advice. The quality of the advice decreases for a phase of the experiment. We studied the task performance, trust and distrust of the participants, and tested whether an instruction to remain skeptical and review each piece of advice led to a better performance compared to a neutral condition. Our results indicate that this instruction does not improve but rather worsens the participants’ performance. Repeated single-item self-report of trust and distrust shows an increase in trust and a decrease in distrust after the drop in the AI’s classification quality, with no difference between the two instructions. Furthermore, via a Bayesian Signal Detection Theory analysis, we provide a procedure to assess appropriate reliance in detail, by quantifying whether the problems of under- and over-reliance have been mitigated. We discuss implications of our results for the usage of disclaimers before interacting with AI, as prominently used in current LLM-based chatbots, and for trust and distrust research.