Characteristics of nonverbal behavior - Lazarov, Stefan Teodorov; Tchappi, Igor; Grimminger, Angela (Social Explainable AI)

Retrospective video recall for analyzing cognitive processes in naturalistic explanations - Lazarov, Stefan Teodorov; Schaffer, Michael; Gladow, Viviane; Buschmeier, Hendrik; Buhl, Heike M.; Grimminger, Angela
In this paper, we discuss the application of retrospective video recall for the assessment of cognitive processes in explanatory interactions, such as understanding and mental models. Our purpose is to reflect on the benefits and limitations of video recall compared to another self-report method, ‘thinking-aloud’. To do so, we reveal empirical results from the application of video recall in three interdisciplinary research projects that applied the method for the qualitative and quantitative assessment of cognitive and behavioral phenomena in everyday explanations. In all three projects, video recall was applied as a post-hoc procedure following the recording of dyadic face-to-face explanations of board games. The design of the video recall procedure differed between individual projects because they pursued different research objectives – that is the investigation of (1) an interlocutor's multimodal signals of understanding, (2) the change in assumptions about an interlocutor's dispositional and situational knowledge, and (3) the differentiated assessment of an interlocutor's developing understanding of domain knowledge aspects by distinguishing between mechanistic and functional explanatory stances. By discussing the benefits and the limitations of each procedure, this article provides critical reflections on video recall as a versatile research method applied for the analysis of human multimodal behavior in interaction and cognitive processing.

How are gaze aversions and mutual gaze related to the topical development of dyadic explanatory interactions? - Lazarov, Stefan Teodorov; Grimminger, Angela (Journal of Nonverbal Behavior)
Abstract Gaze behavior, being continuously accessible to interlocutors in face-to-face interactions, serves as a cue for managing turn-taking, regulating the duration of topical sequences, and supporting cognitive processing in various everyday conversational contexts. The present study seeks to enhance the understanding of the relation between two forms of interactive gaze behavior – gaze aversions and mutual gaze – and the topical development in the explanatory discourse. To do so, we analyzed 24 dyadic board game explanations in which one explainer subsequently explained a board game to three different explainees while the board game was physically absent from the shared space. The main objective of the present study was to investigate the relation of gaze aversions and mutual gaze to the topical development of explanations. For this, based on previous research (Lazarov et al., 2024; Rossano, 2012) we hypothesized that (1) gaze aversions are more likely to be associated with topic changes than topic continuations, and that (2) mutual gaze is more likely to be associated with topic continuations than topic changes. In addition, we explored how the two forms of gaze behavior are related to the interlocutor who initiates a topic change or continuation. Our proportional analysis using a Generalized linear mixed effects model revealed that gaze aversions are related to topic changes initiated by both interlocutors. In contrast, the analysis did not reveal a significant relation between mutual gaze and topic continuations, which could be explained by the feedback elicitation function of mutual gaze at the end of speakers’ utterances (Bavelas et al., 2002; Brône et al., 2017; Kendon, 1967) while monitoring the addressees’ understanding (Clark & Krych, 2004) and the complexity of the analyzed fixed and random effects.

When “yeah” means “not quite”: Multimodal detection of backchannels expressing incomplete understanding - Türk, Olcay; Lazarov, Stefan Teodorov; Wang, Yu; Grimminger, Angela; Buschmeier, Hendrik; Wagner, Petra (Proceedings of INTERSPEECH 2026)
Multimodal backchannels are fundamental to conversational grounding and understanding. However, backchannels do not always transparently reflect the interlocutor's actual cognitive state. “Incongruent backchannels”, where the observable feedback implies understanding although there is no genuine understanding, are a potential source of ambiguity in any interaction. Using acoustic, head movement, and discourse-related data from 45 naturalistic dyadic interactions, we investigate whether incongruent and congruent backchannels show systematically different properties using a classification task. Results show that these backchannels are indeed separable based on multimodal features. Incongruent backchannels are typically characterised by more neutral head movement configurations and lower acoustic dynamism, while discursive cues strongly influence the classification. Overall, the findings suggest a relatively reduced effort in the signalling of incongruent backchannels.

MUNDEX Annotations - Buschmeier, Hendrik; Grimminger, Angela; Wagner, Petra; Lazarov, Stefan Teodorov; Türk, Olcay; Wang, Yu

The reflection of interactional monitoring in the dynamics of verbal and nonverbal forms of explaining - Lazarov, Stefan Teodorov
Erklärungen spielen eine zentrale Rolle in alltäglichen persönlichen Gesprächen, indem sie den Wissensaustausch fördern, Ideen klären und das Verständnis unterstützen. In solchen Gesprächen versuchen die Erklärenden (d. h. die sachkundigere Person), das Verständnis der Explainees (d. h. die Person, die eine Erklärung erhält) durch Interaktionsprozesse wie Monitoring, Scaffolding und gemeinsame Konstruktion zu verbessern (Buschmeier et al., 2023; Rohlfing et al., 2021). Während gemeinsame Konstruktionen aus dem bidirektionalen (non-)verbalen Austausch zwischen den Gesprächspartnern entstehen, bezeichnet Scaffolding den Prozess, durch den die Erklärenden eine Erklärung anpassen, indem sie unterschiedliche Verhaltensweisen als Reaktion auf das Verhalten der Explainees einsetzen, welches deren kognitive Verarbeitung signalisiert (Wood et al., 1976). Monitoring bezeichnet einen kontinuierlichen Prozess, in dem die Gesprächspartner auf Wahrnehmungssignale wie (non-)verbale Verhaltensweisen achten, um Hinweise auf (Miss-)Verständnisse zu erkennen und zu interpretieren (Clark & Krych, 2004).In der vorliegenden Arbeit berichte ich über fünf Studien zu dyadischen Erklärungen zwischen Menschen und diskutiere anhand empirischer Befunde die Interaktionsdynamiken, die bestimmten Formen verbalen und nonverbalen Erklärungsverhaltens zugrunde liegen. Zu diesem Zweck analysierte ich in den vorgestellten Studien Daten aus zwei Videokorpora zu verschiedenen Bereichen alltäglicher Erklärungen, beispielsweise medizinischen Erklärungen und Brettspielerklärungen. Das Korpus zu medizinischen Erklärungen umfasst elf naturalistische Interaktionen zwischen Ärzten und Bezugspersonen über eine bevorstehende chirurgische Operation von Kindern. Das Korpus zu Brettspielerklärungen besteht aus 87 dyadischen Brettspielerklärungen, von denen eine Teilstichprobe von 24 Interaktionen in den vorgestellten Studien spezifisch untersucht wurde.Um das verbale Erklärungsverhalten zu untersuchen, analysierte ich in zwei Studien, die sich mit medizinischen Erklärungen und Brettspielerklärungen befassten, den Zusammenhang zwischen Themenwechseln in Erklärungen und dem multimodalen Verhalten der Explainees, das von den Erklärenden beobachtet wurde. Die Analyse medizinischer Erklärungen legt nahe, dass der Wechsel von Elaborationen zu neuen Themen mit dem multimodalen Verhalten der Explainees einhergeht, welches Blickabwendung, Kopfnicken und verbale Rückkopplung umfasst. Der Wechsel zu Elaborationen hingegen ist mit einer anhaltenden Blickrichtung verbunden, unabhängig davon, ob zusätzliche Signale vorhanden sind oder nicht (Lazarov et al., 2024). Eine nachfolgende Studie zu Brettspielerklärungen (Lazarov & Grimminger, in Begutachtung) erweiterte diese Analyse durch die Einbeziehung des Blickverhaltens der Erklärenden. Die Studie untersuchte den Zusammenhang zwischen gegenseitigem Blickkontakt und Blickabwendung mit der Einleitung neuer Themen. Die Ergebnisse bestätigten die Ergebnisse aus dem medizinischen Kontext: Blickabwendungen der Explainees gehen Themenwechseln häufiger voraus als gegenseitiger Blickkontakt, was mit früheren Forschungsergebnissen von Rossano (2012, 2013) übereinstimmt. Die Analyse untersuchte zudem den Zusammenhang zwischen dem / der Gesprächspartner:in, der / die Blickabwendungen initiierte, und dem / der Gesprächspartner:in, der / die Themenwechsel initiierte.Um das nonverbale Erklärungsverhalten zu untersuchen, analysierte ich die Verwendung von sprachbegleitenden Gesten von verschiedenen Erklärenden in drei Studien zu Brettspielerklärungen, in denen das zu erklärende Objekt im gemeinsamen Referenzraum physisch nicht vorhanden war. Obwohl diese Abwesenheit ein fortwährendes Bedürfnis nach der Etablierung gemeinsamer imaginärer Räume impliziert (Kang et al., 2015; Kinalzik & Heller, 2021), beispielsweise durch kontinuierliches Zeigen auf unsichtbare Orte, zeigte die Studie von Lazarov & Grimminger (2025), dass Gestenikonizität und zeitliche Hervorhebung auch in Themen zu Objekteigenschaften, Handlungsprozessen und bedingten Regeln variabel auftreten.Motiviert durch die kontinuierliche Verwendung von der gestischen Deixis während der physischen Abwesenheit des Explanandums erforschten die letzten zwei Studien den kognitiven Mechanismus der Anpassung des Gebrauchs deiktischer Gesten in Bezug auf die Beobachtung des Verständnisses der Explainees. Unabhängig davon, ob die Erklärenden das Verständnis der Explainees in einer retrospektiven Video Recall Aufgabe interpretierten (Lazarov & Grimminger, 2024a) oder die verbalen Verständnissignale der Explainees wahrnahmen (Lazarov & Grimminger, 2024b), zeigten die Analysen, dass die Häufigkeit der gestischen Deixis während der Erklärungsphase, in der das Explanandum nicht im gemeinsamen Raum vorhanden war, stabil blieb. Zu den Ergebnissen der in dieser Dissertation präsentierten Studien diskutiere ich, wie die kontinuierliche Beobachtung des Feedbackverhaltens der Explainees Anpassungen sowohl im verbalen als auch im nonverbalen Erklärungsverhalten erläutern kann. Darüber hinaus verdeutlichen die von mir präsentierten Studien das Ausmaß der individuellen Variation innerhalb und zwischen den Erklärenden, von denen jeder / jede mit drei verschiedenen Explainees interagierte.

Forms of Understanding for XAI-Explanations - Buschmeier, Hendrik; Buhl, Heike M.; Kern, Friederike; Grimminger, Angela; Beierling, Helen; Fisher, Josephine Beryl; Groß, André; Horwath, Ilona; Klowait, Nils; Lazarov, Stefan Teodorov; Lenke, Michael; Lohmer, Vivien; et al. (Cognitive Systems Research)
Explainability has become an important topic in computer science and artificial intelligence, leading to a subfield called Explainable Artificial Intelligence (XAI). The goal of providing or seeking explanations is to achieve (better) ‘understanding’ on the part of the explainee. However, what it means to ‘understand’ is still not clearly defined, and the concept itself is rarely the subject of scientific investigation. This conceptual article aims to present a model of forms of understanding for XAI-explanations and beyond. From an interdisciplinary perspective bringing together computer science, linguistics, sociology, philosophy and psychology, a definition of understanding and its forms, assessment, and dynamics during the process of giving everyday explanations are explored. Two types of understanding are considered as possible outcomes of explanations, namely enabledness, ‘knowing how’ to do or decide something, and comprehension, ‘knowing that’ – both in different degrees (from shallow to deep). Explanations regularly start with shallow understanding in a specific domain and can lead to deep comprehension and enabledness of the explanandum, which we see as a prerequisite for human users to gain agency. In this process, the increase of comprehension and enabledness are highly interdependent. Against the background of this systematization, special challenges of understanding in XAI are discussed.
Keywords: understanding; explaining; explanations; explainable; AI; interdisciplinarity; comprehension; enabledness; agency

Acoustic detection of false positive backchannels of understanding in explanations - Türk, Olcay; Lazarov, Stefan Teodorov; Buschmeier, Hendrik; Wagner, Petra; Grimminger, Angela (LingCologne 2025 – Book of Abstracts)

Annotation Schemes Project A02 "Monitoring the understanding of explanations" - Lazarov, Stefan Teodorov; Türk, Olcay; Grimminger, Angela; Wagner, Petra; Buschmeier, Hendrik

Variations in explainers’ gesture deixis in explanations related to the monitoring of explainees’ understanding - Lazarov, Stefan Teodorov; Grimminger, Angela (Proceedings of the Annual Meeting of the Cognitive Science Society)
In this study on the use of gesture deixis during explanations, a sample of 24 videorecorded dyadic interactions of a board game explanation was analyzed. The relation between the use of gesture deixis by different explainers and their interpretation of explainees' understanding was investigated. In addition, we describe explainers' intra-individual variations related to their interactions with three different explainees consecutively. While we did not find a relation between interpretations of explainees' complete understanding and a decrease in explainers' use of gesture deixis, we demonstrated that the overall use of gesture deixis is related to the process of interactional monitoring and the attendance of a different explainee.
Keywords: explanation; gesture deixis; monitoring; understanding

Changes in the topical structure of explanations are related to explainees’ multimodal behaviour - Lazarov, Stefan Teodorov; Biermeier, Kai; Grimminger, Angela (Interaction Studies)
Everyday explanations are interactive processes with the aim to provide a less knowledgeable person with reasonable information about other people, objects, or events. Because explanations are interactive communicative processes, the topical structure of an explanation may vary dynamically depending on the immediate feedback of the explainee. In this paper, we analyse topical transitions in medical explanations organised by different physicians (explainers) related to different forms of multimodal behaviour of caregivers (explainees) attending an explanation about the procedures of an upcoming surgery of a child. The analyses reveal that explainees’ multimodal behaviour with gaze shifts (and particularly gaze aversion) can predict a transition from an elaborated topic to a new one, whereas explainees’ forms of multimodal behaviour with static gaze cannot be related to changes of the topical structure.
Keywords: explanations; multimodal behaviour; elaborations; conditional probabilities

Predictability of understanding in explanatory interactions based on multimodal cues - Türk, Olcay; Lazarov, Stefan Teodorov; Wang, Yu; Buschmeier, Hendrik; Grimminger, Angela; Wagner, Petra (Proceedings of the 26th ACM International Conference on Multimodal Interaction)
In explanatory interactions, explainees are expected to continuously provide feedback to explainers by signaling whether they understand an ongoing explanation. The study presented in this paper is based on the hypothesis that explainees use a set of multimodal cues, including vocalizations, facial expressions, and movements of the torso, head, and hands, to do so. We test this hypothesis by building a random forest classifier based on a multimodal corpus of dyadic explanations (21 explainers and explainees), in which windows of understanding or non-understanding were identified by participants in a retrospective video recall task. Results show that sequences of understanding can indeed be differentiated from those of non-understanding, and that a diverse set of predictors covering a wide range of modalities contributes to this classification. Due to data sparsity and a high degree of individual variation, the generalizability of our results is currently limited, but they support our hypothesis of the relevance of multimodal display in explanatory interactions.

Approaches of Assessing Understanding Using Video-Recall Data - Lazarov, Stefan Teodorov; Schaffer, Michael; Ronoh, Erick Kiprop