Scaffolding the human partner by contrastive guidance in an explanatory human-robot dialogue - Groß, A.; Singh, Amit; Banh, Ngoc Chi; Richter, B.; Scharlau, Ingrid; Rohlfing, Katharina J.; Wrede, B. (Frontiers in Robotics and AI)
Explanation has been identified as an important capability for AI-based systems, but research on systematic strategies for achieving understanding in interaction with such systems is still sparse. Negation is a linguistic strategy that is often used in explanations. It creates a contrast space between the affirmed and the negated item that enriches explaining processes with additional contextual information. While negation in human speech has been shown to lead to higher processing costs and worse task performance in terms of recall or action execution when used in isolation, it can decrease processing costs when used in context. So far, it has not been considered as a guiding strategy for explanations in human-robot interaction. We conducted an empirical study to investigate the use of negation as a guiding strategy in explanatory human-robot dialogue, in which a virtual robot explains tasks and possible actions to a human explainee to solve them in terms of gestures on a touchscreen. Our results show that negation vs. affirmation 1) increases processing costs measured as reaction time and 2) increases several aspects of task performance. While there was no significant effect of negation on the number of initially correctly executed gestures, we found a significantly lower number of attempts—measured as breaks in the finger movement data before the correct gesture was carried out—when being instructed through a negation. We further found that the gestures significantly resembled the presented prototype gesture more following an instruction with a negation as opposed to an affirmation. Also, the participants rated the benefit of contrastive vs. affirmative explanations significantly higher. Repeating the instructions decreased the effects of negation, yielding similar processing costs and task performance measures for negation and affirmation after several iterations. We discuss our results with respect to possible effects of negation on linguistic processing of explanations and limitations of our study.
Keywords: HRI; XAI; negation; understanding; explaining; touch interaction; gesture

When do individuals choose care robots over a human caregiver? Insights from a laboratory experiment on choices under uncertainty - Hoppe, Julia Amelie; Tuisku, Outi; Johansson-Pajala, Rose-Marie; Pekkarinen, Satu; Hennala, Lea; Gustafsson, Christine; Melkas, Helinä; Thommes, Kirsten (Computers in Human Behavior Reports)
Keywords: Artificial Intelligence; Cognitive Neuroscience; Computer Science Applications; Human-Computer Interaction; Applied Psychology; Neuroscience (miscellaneous)

The Broken Leg of Algorithm Appreciation: An Experimental Study on the Effect of Unobserved Variables on Advice Utilization - Leffrang, Dirk (Wirtschaftsinformatik Conference)
Despite the widespread use of machine learning algorithms, their effectiveness is limited by a phenomenon known as algorithm aversion. Recent research concluded that unobserved variables can cause algorithm aversion. However, the impact of an unobserved variable on algorithm aversion remains unclear. Previous studies focused on situations where humans had more variables available than algorithms. We extend this research by conducting an online experiment with 94 participants, systematically varying the number of observable variables to the advisor and the advisor type. Surprisingly, our results did not confirm that an unobserved variable had a negative effect on advice-taking. Instead, we found a positive impact in an algorithm appreciation scenario. This study provides new insights into the paradoxical behavior in which people weigh advice more despite having fewer variables, as they correct for the advisor's errors. Practitioners should consider this behavior when designing algorithms and account for user correction behavior.
Keywords: Algorithm aversion; Data; Decision-making; Advice-taking; Human-Computer Interaction

EEG Correlates of Distractions and Hesitations in Human–Robot Interaction: A LabLinking Pilot Study - Richter, Birte; Putze, Felix; Ivucic, Gabriel; Brandt, Mara; Schütze, Christian; Reisenhofer, Rafael; Wrede, Britta; Schultz, Tanja (Multimodal Technologies and Interaction)
In this paper, we investigate the effect of distractions and hesitations as a scaffolding strategy. Recent research points to the potential beneficial effects of a speaker’s hesitations on the listeners’ comprehension of utterances, although results from studies on this issue indicate that humans do not make strategic use of them. The role of hesitations and their communicative function in human-human interaction is a much-discussed topic in current research. To better understand the underlying cognitive processes, we developed a human–robot interaction (HRI) setup that allows the measurement of the electroencephalogram (EEG) signals of a human participant while interacting with a robot. We thereby address the research question of whether we find effects on single-trial EEG based on the distraction and the corresponding robot’s hesitation scaffolding strategy. To carry out the experiments, we leverage our LabLinking method, which enables interdisciplinary joint research between remote labs. This study could not have been conducted without LabLinking, as the two involved labs needed to combine their individual expertise and equipment to achieve the goal together. The results of our study indicate that the EEG correlates in the distracted condition are different from the baseline condition without distractions. Furthermore, we could differentiate the EEG correlates of distraction with and without a hesitation scaffolding strategy. This proof-of-concept study shows that LabLinking makes it possible to conduct collaborative HRI studies in remote laboratories and lays the first foundation for more in-depth research into robotic scaffolding strategies.
Keywords: Computer Networks and Communications; Computer Science Applications; Human-Computer Interaction; Neuroscience (miscellaneous)

On the Multimodal Resolution of a Search Sequence in Virtual Reality - Klowait, Nils (Human Behavior and Emerging Technologies)
In virtual reality (VR), participants may not always have hands, bodies, eyes, or even voices—using VR helmets and two controllers, participants control an avatar through virtual worlds that do not necessarily obey familiar laws of physics; moreover, the avatar’s bodily characteristics may not neatly match our bodies in the physical world. Despite these limitations and specificities, humans get things done through collaboration and the creative use of the environment. While multiuser interactive VR is attracting greater numbers of participants, there are currently few attempts to analyze the in situ interaction systematically. This paper proposes a video-analytic detail-oriented methodological framework for studying virtual reality interaction. Using multimodal conversation analysis, the paper investigates a nonverbal, embodied, two-person interaction: two players in a survival game strive to gesturally resolve a misunderstanding regarding an in-game mechanic—however, both of their microphones are turned off for the duration of play. The players’ inability to resort to complex language to resolve this issue results in a dense sequence of back-and-forth activity involving gestures, object manipulation, gaze, and body work. Most crucially, timing and modified repetitions of previously produced actions turn out to be the key to overcome both technical and communicative challenges. The paper analyzes these action sequences, demonstrates how they generate intended outcomes, and proposes a vocabulary to speak about these types of interaction more generally. The findings demonstrate the viability of multimodal analysis of VR interaction, shed light on unique challenges of analyzing interaction in virtual reality, and generate broader methodological insights about the study of nonverbal action.
Keywords: Human-Computer Interaction; General Social Sciences; Social Psychology; Virtual Reality : Multimodality; Nonverbal Interaction; Search Sequence; Gesture; Co-Operative Action; Goodwin; Ethnomethodology

Perception of Society’s Trust in Care Robots by Public Opinion Leaders - Hoppe, Julia Amelie; Melkas, Helinä; Pekkarinen, Satu; Tuisku, Outi; Hennala, Lea; Johansson-Pajala, Rose-Marie; Gustafsson, Christine; Thommes, Kirsten (International Journal of Human–Computer Interaction)
Keywords: Computer Science Applications; Human-Computer Interaction; Human Factors and Ergonomics

Perception of Society’s Trust in Care Robots by Public Opinion Leaders - Hoppe, Julia Amelie; Melkas, Helinä; Pekkarinen, Satu; Tuisku, Outi; Hennala, Lea; Johansson-Pajala, Rose-Marie; Gustafsson, Christine; Thommes, Kirsten (International Journal of Human–Computer Interaction)
Keywords: Computer Science Applications; Human-Computer Interaction; Human Factors and Ergonomics

Seizing the Opportunity for Automation—How Traffic Density Determines Truck Drivers' Use of Cruise Control - Hoffmann, Christin; Thommes, Kirsten (IEEE Transactions on Human-Machine Systems)
Keywords: Artificial Intelligence; Computer Networks and Communications; Computer Science Applications; Human-Computer Interaction; Signal Processing; Control and Systems Engineering; Human Factors and Ergonomics

Assistant nurses and orientation to care robot use in three European countries - Tuisku, Outi; Johansson-Pajala, Rose-Marie; Hoppe, Julia Amelie; Pekkarinen, Satu; Hennala, Lea; Thommes, Kirsten; Gustafsson, Christine; Melkas, Helinä (Behaviour & Information Technology)
Keywords: Human-Computer Interaction; General Social Sciences; Arts and Humanities (miscellaneous); Developmental and Educational Psychology

The Investigator’s Dilemma - A Review of Social Media Analytics Research Ethics in Information Systems - Marx, Julian; Mirbabaie, Milad (Australasian Journal of Information Systems)
Social media have become not only integral parts of our private and professional lives, but also an indispensable source of data for empirical research across a variety of academic disciplines. Applying a Social Media Analytics (SMA) methodology, however, imposes heavy ethical challenges on researchers. Scholars in the Information Systems (IS) discipline must deal with a patchwork of ethical frameworks, regulations, and (missing) institutional support. To initiate a debate on how to develop a common understanding of SMA research ethics, this paper compiles a scoping review of extant literature and suggests a research agenda for IS scholarship on this matter. The review yields a total of eight fundamental principles of ethical SMA research, which provide a starting point to guiding individual researchers towards more ethical conduct. At the same time, this work unearths a multitude of intricate dilemmas that are currently unresolved. The findings of this review will encourage IS scholarship to find its own voice in the debate about social media research ethics.
Keywords: Information Systems and Management; Human-Computer Interaction; Business; Management and Accounting (miscellaneous); Information Systems

When do individuals choose care robots over a human caregiver? Insights from a laboratory experiment on choices under uncertainty - Hoppe, Julia Amelie; Tuisku, Outi; Johansson-Pajala, Rose-Marie; Pekkarinen, Satu; Hennala, Lea; Gustafsson, Christine; Melkas, Helinä; Thommes, Kirsten (Computers in Human Behavior Reports)
Keywords: Artificial Intelligence; Cognitive Neuroscience; Computer Science Applications; Human-Computer Interaction; Applied Psychology; Neuroscience (miscellaneous)

When do individuals choose care robots over a human caregiver? Insights from a laboratory experiment on choices under uncertainty - Hoppe, Julia Amelie; Tuisku, Outi; Johansson-Pajala, Rose-Marie; Pekkarinen, Satu; Hennala, Lea; Gustafsson, Christine; Melkas, Helinä; Thommes, Kirsten (Computers in Human Behavior Reports)
Keywords: Artificial Intelligence; Cognitive Neuroscience; Computer Science Applications; Human-Computer Interaction; Applied Psychology; Neuroscience (miscellaneous)

When do individuals choose care robots over a human caregiver? Insights from a laboratory experiment on choices under uncertainty - Hoppe, Julia A.; Tuisku, Outi; Johansson-Pajala, Rose-Marie; Pekkarinen, Satu; Hennala, Lea; Gustafsson, Christine; Melkas, Helinä; Thommes, Kirsten (Computers in Human Behavior Reports)
Keywords: Artificial Intelligence; Cognitive Neuroscience; Computer Science Applications; Human-Computer Interaction; Applied Psychology; Neuroscience (miscellaneous)

Perception of Society’s Trust in Care Robots by Public Opinion Leaders - Hoppe, Julia Amelie; Melkas, Helinä; Pekkarinen, Satu; Tuisku, Outi; Hennala, Lea; Johansson-Pajala, Rose-Marie; Gustafsson, Christine; Thommes, Kirsten (International Journal of Human–Computer Interaction)
Keywords: Computer Science Applications; Human-Computer Interaction; Human Factors and Ergonomics

Assistant nurses and orientation to care robot use in three European countries - Tuisku, Outi; Johansson-Pajala, Rose-Marie; Hoppe, Julia Amelie; Pekkarinen, Satu; Hennala, Lea; Thommes, Kirsten; Gustafsson, Christine; Melkas, Helinä (Behaviour & Information Technology)
Keywords: Human-Computer Interaction; General Social Sciences; Arts and Humanities (miscellaneous); Developmental and Educational Psychology

User Involvement in Training Smart Home Agents - Sieger, Leonie Nora; Hermann, Julia; Schomäcker, Astrid; Heindorf, Stefan; Meske, Christian; Hey, Celine-Chiara; Doğangün, Ayşegül (International Conference on Human-Agent Interaction)
Smart home systems contain plenty of features that enhance wellbeing in everyday life through artificial intelligence (AI). However, many users feel insecure because they do not understand the AI’s functionality and do not feel they are in control of it. Combining technical, psychological and philosophical views on AI, we rethink smart homes as interactive systems where users can partake in an intelligent agent’s learning. Parallel to the goals of explainable AI (XAI), we explored the possibility of user involvement in supervised learning of the smart home to have a first approach to improve acceptance, support subjective understanding and increase perceived control. In this work, we conducted two studies: In an online pre-study, we asked participants about their attitude towards teaching AI via a questionnaire. In the main study, we performed a Wizard of Oz laboratory experiment with human participants, where participants spent time in a prototypical smart home and taught activity recognition to the intelligent agent through supervised learning based on the user’s behaviour. We found that involvement in the AI’s learning phase enhanced the users’ feeling of control, perceived understanding and perceived usefulness of AI in general. The participants reported positive attitudes towards training a smart home AI and found the process understandable and controllable. We suggest that involving the user in the learning phase could lead to better personalisation and increased understanding and control by users of intelligent agents for smart home automation.
Keywords: human-agent interaction; smart homes; supervised learning; participation

Digitale Räume geben und nehmen: Unterrichtsinteraktion in DSH-Kursen während der COVID-19-Pandemie - Drumm, Sandra; Müller, Mareike; Stenzel, Nadja (Informationen Deutsch als Fremdsprache)
Das Auftreten der COVID-19-Pandemie stellt Fremdsprachenkurse vielerorts vor Herausforderungen. Unter Zuhilfenahme diverser digitaler Tools werden nicht nur Lernmaterialien online geteilt, sondern auch die Interaktion zwischen Lehrenden und Lernenden sowie der Lernenden untereinander in den virtuellen Raum verlagert. Qualitative Interviews mit den Beteiligten erfassen, wie diese mit den Herausforderungen videogestützten Sprachunterrichts umgehen und welche Strategien sie wählen, um Sprachenlernen zu ermöglichen. Die Ergebnisse zeigen auf, wo seitens der Kursorganisation und -durchführung Handlungsbedarf besteht. ----- The rise of the COVID-19 pandemic challenges the teaching and learning of foreign languages at many institutions. The implementation of various digital tools aids not only the online sharing of learning materials, but also shifts teacher-learner and learner-learner interaction to the virtual space. Via qualitative interviews, this study examines how both teachers and learners handle the challenges of language instruction based on videoconferences, and what strategies they employ to enable language learning. The results highlight areas in need of improvement in terms of course organization and facilitation.
Keywords: German language courses at university; interaction; digital space; language learning/teaching via videoconference

Is Reading Comprehension Taken for Granted? An Analysis of Austrian Textbooks in Fourth and Sixth Grade - Seifert, Susanne (Technology, Knowledge and Learning)
Abstract The range of teaching materials now available is becoming increasingly diverse. Despite this, however, the use and influence of textbooks in teaching still remains very high. When instructing reading comprehension, teachers often use textbooks as the basis for teaching in language lessons. Establishing a good match between textbooks and the skills to be acquired is therefore essential. In this paper, I investigate whether textbooks used in Austrian schools can adequately support the teaching of reading comprehension skills. Since reading comprehension is the basis for acquiring knowledge in all subjects, science textbooks are examined in addition to (German) language lesson textbooks. Thus, the content pages of four language textbooks and four science textbooks for fourth and sixth grade were analysed in terms of five different categories, i.e. general structural setup, learning goals, text types, text structures, and activities. The results reveal clear variations with respect to learning goals in language textbooks. For example, the extent to which reading comprehension is addressed ranges from 13.64 to 69.70%, depending on the book used. Although not addressed as a learning goal in the science textbooks, reading comprehension is often presupposed, especially in sixth grade. While the instruction of reading comprehension ought to entail coverage of reading strategies, this is often neglected, or only dealt with indirectly. Given the diversity of textbooks analysed, it seems all the more important to stress that teachers should: 1) clarify the goals and teaching strategies of a book before using it, 2) become aware of strategies that support the development of students' reading comprehension, and 3) use textbooks as a complementary (and not sole) tool to support reading comprehension in all subjects.
Keywords: Computer Science Applications; Human-Computer Interaction; Education; Mathematics (miscellaneous)

Ethical Considerations of Applying Robots in Kindergarten Settings: Towards an Approach from a Macroperspective - Tolksdorf, Nils F.; Siebert, Scarlet; Zorn, Isabel; Horwath, Ilona; Rohlfing, Katharina J. (International Journal of Social Robotics)
In child–robot interaction (cHRI) research, many studies pursue the goal to develop interactive systems that can be applied in everyday settings. For early education, increasingly, the setting of a kindergarten is targeted. However, when cHRI and research are brought into a kindergarten, a range of ethical and related procedural aspects have to be considered and dealt with.While ethical models elaborated within other human–robot interaction settings, e.g., assisted living contexts, can provide some important indicators for relevant issues, we argue that it is important to start developing a systematic approach to identify and tackle those ethical issues which rise with cHRI in kindergarten settings on a more global level and address the impact of the technology from a macroperspective beyond the effects on the individual. Based on our experience in conducting studies with children in general and pedagogical considerations on the role of the institution of kindergarten in specific, in this paper,we enfold some relevant aspects that have barely been addressed in an explicit way in current cHRI research. Four areas are analyzed and key ethical issues are identified in each area: (1) the institutional setting of a kindergarten, (2) children as a vulnerable group, (3) the caregivers’ role, and (4) pedagogical concepts. With our considerations, we aim at (i) broadening the methodology of the current studies within the area of cHRI, (ii) revalidate it based on our comprehensive empirical experience with research in kindergarten settings, both laboratory and real-world contexts, and (iii) provide a framework for the development of a more systematic approach to address the ethical issues in cHRI research within kindergarten settings.
Keywords: Robot ethics · Kindergarten settings · Child–robot interaction · Early childhood education

Depth camera based statistical shape fitting approach for the creation of an individualized lower body biomechanical model: validity and reliability - Taetz, Bertram; Teufl, Wolfgang; Weidmann, Alexander; Pietschmann, Juliane; Jöllenbeck, Thomas; Bleser, Gabriele (Computer Methods in Biomechanics and Biomedical Engineering)
Keywords: Computer Science Applications; Human-Computer Interaction; Biomedical Engineering; General Medicine; Bioengineering

Reciprocation Effort Games - Polevoy, Gleb; de Weerdt, M.M. (Proceedings of the 29th Benelux Conference on Artificial Intelligence)
Keywords: interaction; reciprocation; contribute; shared effort; curbing; convergence; threshold; Nash equilibrium; social welfare; efficiency; price of anarchy; price of stability

The quest for appropriate models of human-likeness: anthropomorphism in media equation research - Klowait, Nils (AI & SOCIETY)
Keywords: Artificial Intelligence; Human-Computer Interaction; Philosophy

The Convergence of Reciprocation - Polevoy, Gleb; de Weerdt, Mathijs; Jonker, Catholijn (Proceedings of the 2016 International Conference on Autonomous Agents and Multiagent Systems)
Keywords: agent's influence; behavior; convergence; perron-frobenius; reciprocal interaction; repeated reciprocation

Embodied Language Learning and Cognitive Bootstrapping: Methods and Design Principles - Lyon, Caroline; Nehaniv, Chrystopher L.; Saunders, Joe; Belpaeme, Tony; Bisio, Ambra; Fischer, Kerstin; Forster, Frank; Lehmann, Hagen; Metta, Giorgio; Mohan, Vishwanathan; Morse, Anthony; Nolfi, Stefano; et al. (International Journal of Advanced Robotic Systems)
Co-development of action, conceptualization and social interaction mutually scaffold and support each other within a virtuous feedback cycle in the development of human language in children. Within this framework, the purpose of this article is to bring together diverse but complementary accounts of research methods that jointly contribute to our understanding of cognitive development and in particular, language acquisition in robots. Thus, we include research pertaining to developmental robotics, cognitive science, psychology, linguistics and neuroscience, as well as practical computer science and engineering. The different studies are not at this stage all connected into a cohesive whole; rather, they are presented to illuminate the need for multiple different approaches that complement each other in the pursuit of understanding cognitive development in robots. Extensive experiments involving the humanoid robot iCub are reported, while human learning relevant to developmental robotics has also contributed useful results. Disparate approaches are brought together via common underlying design principles. Without claiming to model human language acquisition directly, we are nonetheless inspired by analogous development in humans and consequently, our investigations include the parallel co-development of action, conceptualization and social interaction. Though these different approaches need to ultimately be integrated into a coherent, unified body of knowledge, progress is currently also being made by pursuing individual methods.
Keywords: Robot Language; Human Robot Interaction; HRI; Developmental Robotics; Cognitive Bootstrapping; Statistical Learning

Alignment to the Actions of a Robot - Vollmer, Anna-Lisa; Rohlfing, Katharina; Wrede, Britta; Cangelosi, Angelo (International Journal of Social Robotics)
Alignment is a phenomenon observed in human conversation: Dialog partners' behavior converges in many respects. Such alignment has been proposed to be automatic and the basis for communicating successfully. Recent research on human-computer dialog promotes a mediated communicative design account of alignment according to which the extent of alignment is influenced by interlocutors' beliefs about each other. Our work aims at adding to these findings in two ways. (a) Our work investigates alignment of manual actions, instead of lexical choice. (b) Participants interact with the iCub humanoid robot, instead of an artificial computer dialog system. Our results confirm that alignment also takes place in the domain of actions. We were not able to replicate the results of the original study in general in this setting, but in accordance with its findings, participants with a high questionnaire score for emotional stability and participants who are familiar with robots align their actions more to a robot they believe to be basic than to one they believe to be advanced. Regarding alignment over the course of an interaction, the extent of alignment seems to remain constant, when participants believe the robot to be advanced, but it increases over time, when participants believe the robot to be a basic version.
Keywords: learning; Human-robot interaction; Alignment; Robot social; Action understanding

Improving Human Interaction in Crowdsensing - Polevoy, Gleb; de Weerdt, Mathijs M. (Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems)
Keywords: dynamics; emotion modeling; negotiation; network interaction; shared effort game

Co-development of manner and path concepts in language, action, and eye-gaze behavior - Lohan, Katrin S.; Griffiths, Sascha; Sciutti, Alessandra; Partmann, Tim C.; Rohlfing, Katharina (Topics in Cognitive Science)
In order for artificial intelligent systems to interact naturally with human users, they need to be able to learn from human instructions when actions should be imitated. Human tutoring will typically consist of action demonstrations accompanied by speech. In the following, the characteristics of human tutoring during action demonstration will be examined. A special focus will be put on the distinction between two kinds of motion events: path-oriented actions and manner-oriented actions. Such a distinction is inspired by the literature pertaining to cognitive linguistics, which indicates that the human conceptual system can distinguish these two distinct types of motion. These two kinds of actions are described in language by more path-oriented or more manner-oriented utterances. In path-oriented utterances, the source, trajectory, or goal is emphasized, whereas in manner-oriented utterances the medium, velocity, or means of motion are highlighted. We examined a video corpus of adult-child interactions comprised of three age groups of children-pre-lexical, early lexical, and lexical-and two different tasks, one emphasizing manner more strongly and one emphasizing path more strongly. We analyzed the language and motion of the caregiver and the gazing behavior of the child to highlight the differences between the tutoring and the acquisition of the manner and path concepts. The results suggest that age is an important factor in the development of these action categories. The analysis of this corpus has also been exploited to develop an intelligent robotic behavior -the tutoring spotter system-able to emulate children's behaviors in a tutoring situation, with the aim of evoking in human subjects a natural and effective behavior in teaching to a robot. The findings related to the development of manner and path concepts have been used to implement new effective feedback strategies in the tutoring spotter system, which should provide improvements in human-robot interaction.
Keywords: Imitation; Tutoring; Adult-child interaction; Human-robot interaction; Semantics; Teachable robots

Tutoring in adult-child-interaction: On the loop of the tutor's action modification and the recipient's gaze - Pitsch, Karola; Vollmer, Anna-Lisa; Rohlfing, Katharina; Fritsch, Jannik; Wrede, Britta (Interaction Studies)
Research of tutoring in parent-infant interaction has shown that tutors - when presenting some action - modify both their verbal and manual performance for the learner (‘motherese’, ‘motionese’). Investigating the sources and effects of the tutors’ action modifications, we suggest an interactional account of ‘motionese’. Using video-data from a semi-experimental study in which parents taught their 8 to 11 month old infants how to nest a set of differently sized cups, we found that the tutors’ action modifications (in particular: high arches) functioned as an orienting device to guide the infant’s visual attention (gaze). Action modification and the recipient’s gaze can be seen to have a reciprocal sequential relationship and to constitute a constant loop of mutual adjustments. Implications are discussed for developmental research and for robotic ‘Social Learning’. We argue that a robot system could use on-line feedback strategies (e.g. gaze) to pro-actively shape a tutor’s action presentation as it emerges.
Keywords: conversation analysis; interactional coordination; adult-child-interaction; feedback; gaze; quantification; social learning; motionese; tutoring

Infants' online perception of give-and-take interactions - Elsner, Claudia; Bakker, Marta; Rohlfing, Katharina; Gredebäck, Gustaf (Journal of Experimental Child Psychology)
This research investigated infants’ online perception of give-me gestures during observation of a social interaction. In the first experiment, goal-directed eye movements of 12-month-olds were recorded as they observed a give-and-take interaction in which an object is passed from one individual to another. Infants’ gaze shifts from the passing hand to the receiving hand were significantly faster when the receiving hand formed a give-me gesture relative to when it was presented as an inverted hand shape. Experiment 2 revealed that infants’ goal-directed gaze shifts were not based on different affordances of the two receiving hands. Two additional control experiments further demonstrated that differences in infants’ online gaze behavior were not mediated by an attentional preference for the give-me gesture. Together, our findings provide evidence that properties of social action goals influence infants’ online gaze during action observation. The current studies demonstrate that infants have expectations about well-formed object transfer actions between social agents. We suggest that 12-month-olds are sensitive to social goals within the context of give-and-take interactions while observing from a third-party perspective.
Keywords: Give-me gesture; Infant; Anticipation; Eye movement; Gesture; Social interaction

Studying the Co-Construction of Interaction Protocols in Collaborative Tasks with Humans - Vollmer, Anna-Lisa; Grizou, Jonathan; Lopes, Manuel; Rohlfing, Katharina; Oudeyer, Pierre-Yves (2014 Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics)
Keywords: interaction; communication; co-construction; interaction protocols

Designing interaction routines in service networks: A modularity and social construction-based approach - Becker, Jörg; Beverungen, Daniel; Knackstedt, Ralf; Matzner, Martin; Müller, Oliver; Pöppelbuss, Jens (Scandinavian Journal of Information Systems)
Keywords: Business process management; Conceptual modeling; Interaction routines; Modular design; Service networks; Social construction

How to Assess and Study the Cultural Dimension of Social Interactions in Higher Education Institutions (HEI) - Gebhardt, Anja; Jenert, Tobias
Keywords: learning culture; social interaction

Generation and evaluation of communicative robot gesture - Salem, Maha; Kopp, Stefan; Wachsmuth, Ipke; Rohlfing, Katharina; Joublin, Frank (International Journal of Social Robotics, Special Issue on Expectations, Intentions, and Actions)
How is communicative gesture behavior in robots perceived by humans? Although gesture is crucial in social interaction, this research question is still largely unexplored in the field of social robotics. Thus, the main objective of the present work is to investigate how gestural machine behaviors can be used to design more natural communication in social robots. The chosen approach is twofold. Firstly, the technical challenges encountered when implementing a speech-gesture generation model on a robotic platform are tackled. We present a framework that enables the humanoid robot to flexibly produce synthetic speech and co-verbal hand and arm gestures at run-time, while not being limited to a predefined repertoire of motor actions. Secondly, the achieved flexibility in robot gesture is exploited in controlled experiments. To gain a deeper understanding of how communicative robot gesture might impact and shape human perception and evaluation of human-robot interaction, we conducted a between-subjects experimental study using the humanoid robot in a joint task scenario. We manipulated the non-verbal behaviors of the robot in three experimental conditions, so that it would refer to objects by utilizing either (1) unimodal (i.e., speech only) utterances, (2) congruent multimodal (i.e., semantically matching speech and gesture) or (3) incongruent multimodal (i.e., semantically non-matching speech and gesture) utterances. Our findings reveal that the robot is evaluated more positively when non-verbal behaviors such as hand and arm gestures are displayed along with speech, even if they do not semantically match the spoken utterance.
Keywords: Social Human-Robot Interaction; Multimodal Interaction and Conversational Skills; Robot Companions and Social Robots; Non-verbal Cues and Expressiveness

Generation and evaluation of communicative robot gesture - Salem, Maha; Kopp, Stefan; Wachsmuth, Ipke; Rohlfing, Katharina; Joublin, Frank (International Journal of Social Robotics, Special Issue on Expectations, Intentions, and Actions)
How is communicative gesture behavior in robots perceived by humans? Although gesture is crucial in social interaction, this research question is still largely unexplored in the field of social robotics. Thus, the main objective of the present work is to investigate how gestural machine behaviors can be used to design more natural communication in social robots. The chosen approach is twofold. Firstly, the technical challenges encountered when implementing a speech-gesture generation model on a robotic platform are tackled. We present a framework that enables the humanoid robot to flexibly produce synthetic speech and co-verbal hand and arm gestures at run-time, while not being limited to a predefined repertoire of motor actions. Secondly, the achieved flexibility in robot gesture is exploited in controlled experiments. To gain a deeper understanding of how communicative robot gesture might impact and shape human perception and evaluation of human-robot interaction, we conducted a between-subjects experimental study using the humanoid robot in a joint task scenario. We manipulated the non-verbal behaviors of the robot in three experimental conditions, so that it would refer to objects by utilizing either (1) unimodal (i.e., speech only) utterances, (2) congruent multimodal (i.e., semantically matching speech and gesture) or (3) incongruent multimodal (i.e., semantically non-matching speech and gesture) utterances. Our findings reveal that the robot is evaluated more positively when non-verbal behaviors such as hand and arm gestures are displayed along with speech, even if they do not semantically match the spoken utterance.
Keywords: Social Human-Robot Interaction; Multimodal Interaction and Conversational Skills; Robot Companions and Social Robots; Non-verbal Cues and Expressiveness

SCM - A Simple, Modular and Flexible Customer Interaction Management System - Schuster, Jörg; Lee, Yeong Su; Kobothanassi, Despina; Bargel, Matthias; Geierhos, Michaela (International Conference on Information Society (i-Society 2011))
SCM is a simple, modular and flexible system for web monitoring and customer interaction management. In our view, its main advantages are the following: It is completely web based. It combines all technologies, data, software agents and human agents involved in the monitoring and customer interaction process. It can be used for messages written in any natural language. Although the prototype of SCM is designed for classifying and processing messages about mobile-phone related problems in social networks, SCM can easily be adapted to other text types such as discussion board posts, blogs or emails. Unlike comparable systems, SCM uses linguistic technologies to classify messages and recognize paraphrases of product names. For two reasons, product name paraphrasing plays a major role in SCM: First, product names typically have many, sometimes hundreds or thousands of intralingual paraphrases. Secondly, product names have interlingual paraphrases: The same products are often called or spelt differently in different countries and/or languages. By mapping product name variants to an international canonical form, SCM allows for answering questions like Which statements are made about this mobile phone in which languages/in which social networks/in which countries/...? The SCM product name paraphrasing engine is designed in such a way that standard variants are assigned automatically, regular variants are assigned semiautomatically and idiosyncratic variants can be added manually. With this and similar features we try to realize our philosophy of simplicity, modularity and flexibility: Whatever can be done automatically is done automatically. But manual intervention is always possible and easy and it does not conflict in any way with the automatic functions of SCM.
Keywords: Social Media Business Integration; Contact Center Application Support; Monitoring Social Conversations; Social Customer Interaction Management; Monitoring; Software Agents

Customer Interaction 2.0: Adopting Social Media as Customer Service Channel - Geierhos, Michaela (Journal of Advances in Information Technology)
Since customers first share their problems with a social networking community before directly addressing a company, social networking sites such as Facebook, Twitter, MySpace or Foursquare will be the interface between customer and company. For this reason, it is assumed that social networks will evolve into a common communication channel – not only between individuals but also between customers and companies. However, social networking has not yet been integrated into customer interaction management (CIM) tools. In general, a CIM application is used by the agents in a contact centre while communicating with the customers. Such systems handle communication across multiple different channels, such as e-mail, telephone, Instant Messaging, letter etc. What we do now is to integrate social networking into CIM applications by adding another communication channel. This allows the company to follow general trends in customer opinions on the Internet, but also record two-sided communication for customer service management and the company’s response will be delivered through the customer’s preferred social networking site.
Keywords: Social Media Business Integration; Multichannel Customer Interaction Management; Contact Centre Application Support

Mindful tutors: Linguistic choice and action demonstration in speech to infants and a simulated robot - Fischer, Kerstin; Foth, Kilian; Rohlfing, Katharina; Wrede, Britta (Interaction Studies)
It has been proposed that the design of robots might benefit from interactions that are similar to caregiver–child interactions, which is tailored to children’s respective capacities to a high degree. However, so far little is known about how people adapt their tutoring behaviour to robots and whether robots can evoke input that is similar to child-directed interaction. The paper presents detailed analyses of speakers’ linguistic and non-linguistic behaviour, such as action demonstration, in two comparable situations: In one experiment, parents described and explained to their nonverbal infants the use of certain everyday objects; in the other experiment, participants tutored a simulated robot on the same objects. The results, which show considerable differences between the two situations on almost all measures, are discussed in the light of the computer-as-social-actor paradigm and the register hypothesis.
Keywords: human–robot interaction (HRI); social communication; register theory; motionese; robotese; child-directed speech (CDS); motherese; mindless transfer; computers-as-social-actors

Language Does Something: Body Action and Language in Maternal Input to Three-Month-Olds - Nomikou, Iris; Rohlfing, Katharina (IEEE Transactions on Autonomous Mental Development)
Keywords: acoustic packaging; mother-child interaction; social learning; multimodal grounding in input; ecology of interactions; synchrony

Effects of gesture on the perception of psychological anthropomorphism: A case study with a humanoid robot - Salem, Maha; Eyssel, Friederike Anne; Rohlfing, Katharina; Kopp, Stefan; Joublin, F. (Social Robotics)
Previous work has shown that gestural behaviors affect anthropomorphic inferences about artificial communicators such as virtual agents. In an experiment with a humanoid robot, we investigated to what extent gesture would affect anthropomorphic inferences about the robot. Particularly, we examined the effects of the robot's hand and arm gestures on the attribution of typically human traits, likability of the robot, shared reality, and future contact intentions after interacting with the robot. For this, we manipulated the non-verbal behaviors of the humanoid robot in three experimental conditions: (1) no gesture, (2) congruent gesture, and (3) incongruent gesture. We hypothesized higher ratings on all dependent measures in the two gesture (vs. no gesture) conditions. The results confirm our predictions: when the robot used gestures during interaction, it was anthropomorphized more, participants perceived it as more likable, reported greater shared reality with it, and showed increased future contact intentions than when the robot gave instructions without using gestures. Surprisingly, this effect was particularly pronounced when the robot's gestures were partly incongruent with speech. These findings show that communicative non-verbal behaviors in robotic systems affect both anthropomorphic perceptions and the mental models humans form of a humanoid robot during interaction.
Keywords: Multimodal Interaction and Conversational Skills; Anthropomorphism; Non-verbal Cues and Expressiveness

Using Prominence Detection to Generate Acoustic Feedback in Tutoring Scenarios - Schillingmann, Lars; Wagner, Petra; Munier, Christian; Wrede, Britta; Rohlfing, Katharina (Interspeech 2011 (12th Annual Conference of the International Speech Communication Association))
Robots interacting with humans need to understand actions and make use of language in social interactions. Research on infant development has shown that language helps the learner to structure visual observations of action. This acoustic information typically in the form of narration overlaps with action sequences and provides infants with a bottom-up guide to ﬁnd structure within them. This concept has been introduced as acoustic packaging by Hirsh-Pasek and Golinkoff. We developed and integrated a prominence detection module in our acoustic packaging system to detect semantically relevant information linguistically highlighted by the tutor. Evaluation results on speech data from adult-infant interactions show a signiﬁcant agreement with human raters. Furthermore a ﬁrst approach based on acoustic packages which uses the prominence detection results to generate acoustic feedback is presented. Index Terms: prominence, multimodal action segmentation, human robot interaction, feedback
Keywords: Feedback; Human Robot Interaction; Prominence; Multimodal Action Segmentation

Acoustic Packaging and the Learning of Words - Schillingmann, Lars; Wagner, Petra; Munier, Christian; Wrede, Britta; Rohlfing, Katharina
Keywords: Prominence; Multimodal Action Segmentation; Feedback; Color Saliency; Human Robot Interaction

Effects of gesture on the perception of psychological anthropomorphism: A case study with a humanoid robot - Salem, Maha; Eyssel, Friederike Anne; Rohlfing, Katharina; Kopp, Stefan; Joublin, F. (Social Robotics)
Previous work has shown that gestural behaviors affect anthropomorphic inferences about artificial communicators such as virtual agents. In an experiment with a humanoid robot, we investigated to what extent gesture would affect anthropomorphic inferences about the robot. Particularly, we examined the effects of the robot's hand and arm gestures on the attribution of typically human traits, likability of the robot, shared reality, and future contact intentions after interacting with the robot. For this, we manipulated the non-verbal behaviors of the humanoid robot in three experimental conditions: (1) no gesture, (2) congruent gesture, and (3) incongruent gesture. We hypothesized higher ratings on all dependent measures in the two gesture (vs. no gesture) conditions. The results confirm our predictions: when the robot used gestures during interaction, it was anthropomorphized more, participants perceived it as more likable, reported greater shared reality with it, and showed increased future contact intentions than when the robot gave instructions without using gestures. Surprisingly, this effect was particularly pronounced when the robot's gestures were partly incongruent with speech. These findings show that communicative non-verbal behaviors in robotic systems affect both anthropomorphic perceptions and the mental models humans form of a humanoid robot during interaction.
Keywords: Multimodal Interaction and Conversational Skills; Anthropomorphism; Non-verbal Cues and Expressiveness

Developing feedback: How children of different age contribute to a tutoring interaction with adults - Vollmer, Anna-Lisa; Pitsch, Karola; Lohan, Katrin Solveig; Fritsch, Jannik; Rohlfing, Katharina; Wrede, Britta (Development and Learning (ICDL), 2010 IEEE 9th International Conference on Development and Learning)
Keywords: tutoring interaction; social interaction; video signal processing; robot systems; paediatrics; neurophysiology; Learning; infant; feedback; biology computing; cognitive capabilities; cognition; children

On the loop of action modification and the recipient's gaze in adult-child interaction - Pitsch, Karola; Vollmer, Anna-Lisa; Fritsch, Jannik; Wrede, Britta; Rohlfing, Katharina; Sagerer, Gerhard (Gesture and Speech in Interaction)
Learning is a social endeavor, in which the learner generally receives support from his/her social partner(s). In developmental research – even though tutors/adults behavior modifications in their speech, gestures and motions have been extensively studied, studies barely consider the recipient’s (i.e. the child’s) perspective in the analysis of the adult’s presentation, In addition, the variability in parental behavior, i.e. the fact that not every parent modifies her/his behavior in the same way, found less fine-grained analysis. In contrast, in this paper, we assume an interactional perspective investigating the loop between the tutor’s and the learner’s actions. With this approach, we aim both at discovering the levels and features of variability and at achieving a better understanding of how they come about within the course of the interaction. For our analysis, we used a combination of (1) qualitative investigation derived from ethnomethodological Conversation Analysis (CA), (2) semi-automatic computational 2D hand tracking and (3) a mathematically based visualization of the data. Our analysis reveals that tutors not only shape their demonstrations differently with regard to the intended recipient per se (adult-directed vs. child-directed), but most importantly that the learner’s feedback during the presentation is consequential for the concrete ways in which the presentation is carried out.
Keywords: gaze; gesture; Multimodal; adult-child interaction

People modify their tutoring behavior in robot-directed interaction for action learning - Vollmer, Anna-Lisa; Lohan, Katrin Solveig; Fischer, Kerstin; Nagai, Yukie; Pitsch, Karola; Fritsch, Jannik; Rohlfing, Katharina; Wrede, Britta (Development and Learning, 2009. ICDL 2009. IEEE 8th International Conference on Development and Learning)
In developmental research, tutoring behavior has been identified as scaffolding infants' learning processes. It has been defined in terms of child-directed speech (Motherese), child-directed motion (Motionese), and contingency. In the field of developmental robotics, research often assumes that in human-robot interaction (HRI), robots are treated similar to infants, because their immature cognitive capabilities benefit from this behavior. However, according to our knowledge, it has barely been studied whether this is true and how exactly humans alter their behavior towards a robotic interaction partner. In this paper, we present results concerning the acceptance of a robotic agent in a social learning scenario obtained via comparison to adults and 8-11 months old infants in equal conditions. These results constitute an important empirical basis for making use of tutoring behavior in social robotics. In our study, we performed a detailed multimodal analysis of HRI in a tutoring situation using the example of a robot simulation equipped with a bottom-up saliency-based attention model. Our results reveal significant differences in hand movement velocity, motion pauses, range of motion, and eye gaze suggesting that for example adults decrease their hand movement velocity in an Adult-Child Interaction (ACI), opposed to an Adult-Adult Interaction (AAI) and this decrease is even higher in the Adult-Robot Interaction (ARI). We also found important differences between ACI and ARI in how the behavior is modified over time as the interaction unfolds. These findings indicate the necessity of integrating top-down feedback structures into a bottom-up system for robots to be fully accepted as interaction partners.
Keywords: robot simulation; hand movement velocity; robotic interaction partner; robotic agent; robot-directed interaction; multimodal analysis; Motionese; Motherese; intelligent tutoring systems; immature cognitive capability; human computer interaction; eye gaze; child-directed speech; child-directed motion; bottom-up system; bottom-up saliency-based attention model; adult-robot interaction; adult-child interaction; adult-adult interaction; human-robot interaction; action learning; social learning scenario; social robotics; software agents; top-down feedback structures; tutoring behavior

Systemic interaction analysis (SInA) in HRI - Lohse, Manja; Hanheide, Marc; Rohlfing, Katharina; Sagerer, Gerhard (Proceedings of the 4th ACM/IEEE international conference on Human robot interaction - HRI '09)
Keywords: SINA; human robot interaction; biron

Assessment of a Multimodal Interaction and Rendering System against Established Design Principles - Schäfer, Robbie; Müller, Wolfgang (Journal on Multimodal User Interfaces)
To provide user interfaces for a rich set of devices and interaction modalities, we follow a model-based development methodology. We devised an architecture which deploys user interfaces specified as dialogue models with abstract interaction objects and allows context-based adaptations by means of an external transcoding process. For the validation of the applicability of this methodology for developing usable multimodal multi-device systems, we present two case studies based on proof-of-concept implementations and assessed them with a large set of established design principles and different types of modality cooperation.
Keywords: Interaction architecture Abstract interaction objects Dialogue model Transformations Multimodality Multi-device Design principles

“Try something else!” — When users change their discursive behavior in human-robot interaction - Lohse, Manja; Rohlfing, Katharina; Wrede, Britta; Sagerer, Gerhard
This paper investigates the influence of feedback provided by an autonomous robot (BIRON) on users’ discursive behavior. A user study is described during which users show objects to the robot. The results of the experiment indicate, that the robot’s verbal feedback utterances cause the humans to adapt their own way of speaking. The changes in users’ verbal behavior are due to their beliefs about the robots knowledge and abilities. In this paper they are identified and grouped. Moreover, the data implies variations in user behavior regarding gestures. Unlike speech, the robot was not able to give feedback with gestures. Due to the lack of feedback, users did not seem to have a consistent mental representation of the robot’s abilities to recognize gestures. As a result, changes between different gestures are interpreted to be unconscious variations accompanying speech.
Keywords: discursive behavior; autonomous robot; BIRON; man-machine systems; robot abilities; robot knowledge; user gestures; robot verbal feedback utterance; speech processing; user verbal behavior; service robots; human-robot interaction; human computer interaction; gesture recognition

Temporal processes in prime-mask interaction: Assessing perceptual consequences of masked information. - Scharlau, Ingrid (Advances in Cognitive Psychology)
Visual backward masking is frequently used to study the temporal dynamics of visual perception. These dynamics may include the temporal features of conscious percepts, as suggested, for instance, by the asynchronous-updating model (Neumann, 1982) and perceptual-retouch theory (Bachmann, 1994). These models predict that the perceptual latency of a visual backward mask is shorter than that of a like reference stimulus that was not preceded by a masked stimulus. The prediction has been confirmed by studies using temporal-order judgments: For certain asynchronies between mask and reference stimulus, temporal-order reversals are quite frequent (e.g. Scharlau, & Neumann, 2003a). However, it may be argued that these reversals were due to a response bias in favour of the mask rather than true temporal-perceptual effects. I introduce two measures for assessing latency effects that (1) are not prone to such a response bias, (2) allow to quantify the latency gain, and (3) extend the perceptual e
Keywords: temporal processes; prime mask interaction; perceptual consequences; masked information; visual backward masking; visual perception; Temporal Lobe; Visual Masking; Visual Perception; Consequence

Dialog Modelling for Multiple Devices and Multiple Interaction Modalities - Schäfer, Robbie; Bleul, Steffen; Müller, Wolfgang (Proceedings of the 5th International Workshop on Task Models and Diagrams for User Interface Design (TAMODIA'2006))
Today a large variety of mobile interaction devices such as PDAs and mobile phones enforce the development of a wide range of user interfaces for each platform. The complexity even grows, when multiple interaction devices are used to perform the same task and when different modalities have to be supported. We introduce a new dialog model for the abstraction of concrete user interfaces with a separate advanced control layer for the integration of different modalities. In this context, we present the Dialog and Interface Specification Language (DISL), which comes with a proof-of-concept implementation.
Keywords: User Interface Interaction Manager Output Device Multimodal Interface Interaction Object