When “yeah” means “not quite”: Multimodal detection of backchannels expressing incomplete understanding

Multimodal backchannels are fundamental to conversational grounding and understanding. However, backchannels do not always transparently reflect the interlocutor's actual cognitive state. “Incongruent backchannels”, where the observable feedback implies understanding although there is no genuine understanding, are a potential source of ambiguity in any interaction. Using acoustic, head movement, and discourse-related data from 45 naturalistic dyadic interactions, we investigate whether incongruent and congruent backchannels show systematically different properties using a classification task. Results show that these backchannels are indeed separable based on multimodal features. Incongruent backchannels are typically characterised by more neutral head movement configurations and lower acoustic dynamism, while discursive cues strongly influence the classification. Overall, the findings suggest a relatively reduced effort in the signalling of incongruent backchannels.