[{"status":"public","abstract":[{"lang":"eng","text":"Multilingual picture books are considered to have great potential for language reflection and language learning, partly due to their multimodality, in which writing(s), language(s) and image(s) interact in the construction of\r\nmeaning. What possibilities multilingual picture books offer for grammatical learning is still a research desidera-\r\ntum. In a multi-perspective qualitative analysis that brings together categories from picture book and typography\r\nresearch as well as graphemics and grammar, three variants of multilingual picture books are analysed. The results show that all of them – to varying degrees – open up possibilities for the further development of multiliteracies and offer potential that could be used in classroom practice for both implicit and explicit grammatical learning.\r\n"},{"text":"Mehrsprachigen Bilderbüchern wird ein hohes Potential für sprachreflexives und sprachliches Lernen zugespro-\r\nchen, auch aufgrund ihrer Multimodalität, bei der Schrift(en), Sprache(n) und Bild(er) bei der Bedeutungskonstruk-\r\ntion zusammenspielen. Welche Möglichkeiten Bilderbücher, bei denen Mehrsprachigkeit auf Multimodalität trifft, für grammatisches Lernen bieten, stellt noch immer ein Forschungsdesiderat dar. In einer mehrperspektivisch ausgerichteten qualitativen Analyse, die Kategorien aus der Bilderbuch- und Typographieforschung sowie Graphe-\r\nmatik und Grammatik zusammenführt, werden drei Varianten mehrsprachiger Bilderbücher (additiv, parallel und\r\nintegrativ mehrsprachige) untersucht. Die Ergebnisse zeigen, dass alle drei – zum Teil in unterschiedlichem Maße – Möglichkeiten für die Weiterentwicklung von Multiliteracies (Multiliteralität und Multiliterarität) eröffnen und Anregungspotentiale bieten, die in der Unterrichtspraxis – sowohl für implizites als auch explizites – multimodales grammatisches Lernen genutzt werden könnten.\r\n","lang":"ger"}],"editor":[{"first_name":"Sabrina","full_name":"Geyer, Sabrina","last_name":"Geyer"},{"first_name":"Valentina","last_name":"Cristante","full_name":"Cristante, Valentina"}],"publication":"Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven","type":"book_chapter","language":[{"iso":"ger"}],"keyword":["grammar","multimodality","multiliteracies","picture book","typography","Grammatik","Multimodalität","Multiliteracies","Bilderbuch","Typographie"],"user_id":"23088","_id":"64840","page":"190 - 207","citation":{"apa":"Topalovic, E., &#38; Härtel, K. (2026). Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild. In S. Geyer &#38; V. Cristante (Eds.), <i>Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven</i> (pp. 190–207). <a href=\"https://doi.org/10.46586/SLLD.462\">https://doi.org/10.46586/SLLD.462</a>","mla":"Topalovic, Elvira, and Kira Härtel. “Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild.” <i>Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven</i>, edited by Sabrina Geyer and Valentina Cristante, 2026, pp. 190–207, doi:<a href=\"https://doi.org/10.46586/SLLD.462\">https://doi.org/10.46586/SLLD.462</a>.","short":"E. Topalovic, K. Härtel, in: S. Geyer, V. Cristante (Eds.), Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven, 2026, pp. 190–207.","bibtex":"@inbook{Topalovic_Härtel_2026, title={Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild}, DOI={<a href=\"https://doi.org/10.46586/SLLD.462\">https://doi.org/10.46586/SLLD.462</a>}, booktitle={Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven}, author={Topalovic, Elvira and Härtel, Kira}, editor={Geyer, Sabrina and Cristante, Valentina}, year={2026}, pages={190–207} }","chicago":"Topalovic, Elvira, and Kira Härtel. “Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild.” In <i>Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven</i>, edited by Sabrina Geyer and Valentina Cristante, 190–207, 2026. <a href=\"https://doi.org/10.46586/SLLD.462\">https://doi.org/10.46586/SLLD.462</a>.","ieee":"E. Topalovic and K. Härtel, “Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild,” in <i>Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven</i>, S. Geyer and V. Cristante, Eds. 2026, pp. 190–207.","ama":"Topalovic E, Härtel K. Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild. In: Geyer S, Cristante V, eds. <i>Grammatikdidaktik und Mehrsprachigkeit: Theoretische und empirische Perspektiven</i>. ; 2026:190-207. doi:<a href=\"https://doi.org/10.46586/SLLD.462\">https://doi.org/10.46586/SLLD.462</a>"},"year":"2026","quality_controlled":"1","publication_status":"published","doi":"https://doi.org/10.46586/SLLD.462","main_file_link":[{"url":"https://omp.ub.rub.de/index.php/SLLD/catalog/view/462/379/2391","open_access":"1"}],"title":"Grammatik und Multimodalität im mehrsprachigen Bilderbuch. Qualitative Analyse an der Schnittstelle von Schrift, Sprache und Bild","date_created":"2026-03-05T09:13:53Z","author":[{"full_name":"Topalovic, Elvira","id":"23088","last_name":"Topalovic","orcid":"0009-0002-7576-3558","first_name":"Elvira"},{"first_name":"Kira","last_name":"Härtel","full_name":"Härtel, Kira","id":"23198"}],"oa":"1","date_updated":"2026-03-05T09:29:27Z"},{"citation":{"bibtex":"@article{Klowait_2023, title={On the Multimodal Resolution of a Search Sequence in Virtual Reality}, volume={2023}, DOI={<a href=\"https://doi.org/10.1155/2023/8417012\">10.1155/2023/8417012</a>}, journal={Human Behavior and Emerging Technologies}, publisher={Hindawi Limited}, author={Klowait, Nils}, year={2023}, pages={1–15} }","short":"N. Klowait, Human Behavior and Emerging Technologies 2023 (2023) 1–15.","mla":"Klowait, Nils. “On the Multimodal Resolution of a Search Sequence in Virtual Reality.” <i>Human Behavior and Emerging Technologies</i>, vol. 2023, Hindawi Limited, 2023, pp. 1–15, doi:<a href=\"https://doi.org/10.1155/2023/8417012\">10.1155/2023/8417012</a>.","apa":"Klowait, N. (2023). On the Multimodal Resolution of a Search Sequence in Virtual Reality. <i>Human Behavior and Emerging Technologies</i>, <i>2023</i>, 1–15. <a href=\"https://doi.org/10.1155/2023/8417012\">https://doi.org/10.1155/2023/8417012</a>","chicago":"Klowait, Nils. “On the Multimodal Resolution of a Search Sequence in Virtual Reality.” <i>Human Behavior and Emerging Technologies</i> 2023 (2023): 1–15. <a href=\"https://doi.org/10.1155/2023/8417012\">https://doi.org/10.1155/2023/8417012</a>.","ieee":"N. Klowait, “On the Multimodal Resolution of a Search Sequence in Virtual Reality,” <i>Human Behavior and Emerging Technologies</i>, vol. 2023, pp. 1–15, 2023, doi: <a href=\"https://doi.org/10.1155/2023/8417012\">10.1155/2023/8417012</a>.","ama":"Klowait N. On the Multimodal Resolution of a Search Sequence in Virtual Reality. <i>Human Behavior and Emerging Technologies</i>. 2023;2023:1-15. doi:<a href=\"https://doi.org/10.1155/2023/8417012\">10.1155/2023/8417012</a>"},"intvolume":"      2023","page":"1-15","publication_status":"published","publication_identifier":{"issn":["2578-1863"]},"has_accepted_license":"1","main_file_link":[{"open_access":"1","url":"https://doi.org/10.1155/2023/8417012"}],"doi":"10.1155/2023/8417012","oa":"1","date_updated":"2024-03-26T09:40:53Z","author":[{"first_name":"Nils","id":"98454","full_name":"Klowait, Nils","orcid":"0000-0002-7347-099X","last_name":"Klowait"}],"volume":2023,"status":"public","type":"journal_article","article_type":"original","file_date_updated":"2023-04-06T11:00:01Z","funded_apc":"1","project":[{"name":"TRR 318 - Ö: TRR 318 - Project Area Ö","_id":"119"}],"_id":"43437","user_id":"98454","department":[{"_id":"9"}],"year":"2023","quality_controlled":"1","title":"On the Multimodal Resolution of a Search Sequence in Virtual Reality","publisher":"Hindawi Limited","date_created":"2023-04-06T10:57:28Z","abstract":[{"text":"<jats:p>In virtual reality (VR), participants may not always have hands, bodies, eyes, or even voices—using VR helmets and two controllers, participants control an avatar through virtual worlds that do not necessarily obey familiar laws of physics; moreover, the avatar’s bodily characteristics may not neatly match our bodies in the physical world. Despite these limitations and specificities, humans get things done through collaboration and the creative use of the environment. While multiuser interactive VR is attracting greater numbers of participants, there are currently few attempts to analyze the in situ interaction systematically. This paper proposes a video-analytic detail-oriented methodological framework for studying virtual reality interaction. Using multimodal conversation analysis, the paper investigates a nonverbal, embodied, two-person interaction: two players in a survival game strive to gesturally resolve a misunderstanding regarding an in-game mechanic—however, both of their microphones are turned off for the duration of play. The players’ inability to resort to complex language to resolve this issue results in a dense sequence of back-and-forth activity involving gestures, object manipulation, gaze, and body work. Most crucially, timing and modified repetitions of previously produced actions turn out to be the key to overcome both technical and communicative challenges. The paper analyzes these action sequences, demonstrates how they generate intended outcomes, and proposes a vocabulary to speak about these types of interaction more generally. The findings demonstrate the viability of multimodal analysis of VR interaction, shed light on unique challenges of analyzing interaction in virtual reality, and generate broader methodological insights about the study of nonverbal action.</jats:p>","lang":"eng"}],"file":[{"content_type":"application/pdf","relation":"main_file","success":1,"date_created":"2023-04-06T11:00:01Z","creator":"nklowait","date_updated":"2023-04-06T11:00:01Z","file_id":"43438","file_name":"Klowait_2023a.pdf","access_level":"closed","file_size":2877385}],"publication":"Human Behavior and Emerging Technologies","ddc":["300"],"keyword":["Human-Computer Interaction","General Social Sciences","Social Psychology","Virtual Reality : Multimodality","Nonverbal Interaction","Search Sequence","Gesture","Co-Operative Action","Goodwin","Ethnomethodology"],"language":[{"iso":"eng"}]},{"year":"2022","title":"BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems","publisher":"Springer International Publishing","date_created":"2023-11-14T15:58:58Z","abstract":[{"lang":"eng","text":"In multimodal multi-objective optimization (MMMOO), the focus is not solely on convergence in objective space, but rather also on explicitly ensuring diversity in decision space. We illustrate why commonly used diversity measures are not entirely appropriate for this task and propose a sophisticated basin-based evaluation (BBE) method. Also, BBE variants are developed, capturing the anytime behavior of algorithms. The set of BBE measures is tested by means of an algorithm configuration study. We show that these new measures also transfer properties of the well-established hypervolume (HV) indicator to the domain of MMMOO, thus also accounting for objective space convergence. Moreover, we advance MMMOO research by providing insights into the multimodal performance of the considered algorithms. Specifically, algorithms exploiting local structures are shown to outperform classical evolutionary multi-objective optimizers regarding the BBE variants and respective trade-off with HV."}],"publication":"Parallel Problem Solving from Nature (PPSN XVII)","keyword":["Anytime behavior","Benchmarking","Continuous optimization","Multi-objective optimization","Multimodality","Performance metric"],"language":[{"iso":"eng"}],"place":"Cham","citation":{"apa":"Heins, J., Rook, J., Schäpermeier, L., Kerschke, P., Bossek, J., &#38; Trautmann, H. (2022). BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems. In G. Rudolph, A. V. Kononova, H. Aguirre, P. Kerschke, G. Ochoa, &#38; T. Tusar (Eds.), <i>Parallel Problem Solving from Nature (PPSN XVII)</i> (pp. 192–206). Springer International Publishing. <a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">https://doi.org/10.1007/978-3-031-14714-2_14</a>","bibtex":"@inproceedings{Heins_Rook_Schäpermeier_Kerschke_Bossek_Trautmann_2022, place={Cham}, series={Lecture Notes in Computer Science}, title={BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems}, DOI={<a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">10.1007/978-3-031-14714-2_14</a>}, booktitle={Parallel Problem Solving from Nature (PPSN XVII)}, publisher={Springer International Publishing}, author={Heins, Jonathan and Rook, Jeroen and Schäpermeier, Lennart and Kerschke, Pascal and Bossek, Jakob and Trautmann, Heike}, editor={Rudolph, Günter and Kononova, Anna V. and Aguirre, Hernán and Kerschke, Pascal and Ochoa, Gabriela and Tusar, Tea}, year={2022}, pages={192–206}, collection={Lecture Notes in Computer Science} }","short":"J. Heins, J. Rook, L. Schäpermeier, P. Kerschke, J. Bossek, H. Trautmann, in: G. Rudolph, A.V. Kononova, H. Aguirre, P. Kerschke, G. Ochoa, T. Tusar (Eds.), Parallel Problem Solving from Nature (PPSN XVII), Springer International Publishing, Cham, 2022, pp. 192–206.","mla":"Heins, Jonathan, et al. “BBE: Basin-Based Evaluation of Multimodal Multi-Objective Optimization Problems.” <i>Parallel Problem Solving from Nature (PPSN XVII)</i>, edited by Günter Rudolph et al., Springer International Publishing, 2022, pp. 192–206, doi:<a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">10.1007/978-3-031-14714-2_14</a>.","ama":"Heins J, Rook J, Schäpermeier L, Kerschke P, Bossek J, Trautmann H. BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems. In: Rudolph G, Kononova AV, Aguirre H, Kerschke P, Ochoa G, Tusar T, eds. <i>Parallel Problem Solving from Nature (PPSN XVII)</i>. Lecture Notes in Computer Science. Springer International Publishing; 2022:192–206. doi:<a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">10.1007/978-3-031-14714-2_14</a>","chicago":"Heins, Jonathan, Jeroen Rook, Lennart Schäpermeier, Pascal Kerschke, Jakob Bossek, and Heike Trautmann. “BBE: Basin-Based Evaluation of Multimodal Multi-Objective Optimization Problems.” In <i>Parallel Problem Solving from Nature (PPSN XVII)</i>, edited by Günter Rudolph, Anna V. Kononova, Hernán Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tusar, 192–206. Lecture Notes in Computer Science. Cham: Springer International Publishing, 2022. <a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">https://doi.org/10.1007/978-3-031-14714-2_14</a>.","ieee":"J. Heins, J. Rook, L. Schäpermeier, P. Kerschke, J. Bossek, and H. Trautmann, “BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems,” in <i>Parallel Problem Solving from Nature (PPSN XVII)</i>, 2022, pp. 192–206, doi: <a href=\"https://doi.org/10.1007/978-3-031-14714-2_14\">10.1007/978-3-031-14714-2_14</a>."},"page":"192–206","publication_identifier":{"isbn":["978-3-031-14714-2"]},"doi":"10.1007/978-3-031-14714-2_14","date_updated":"2023-12-13T10:47:50Z","author":[{"full_name":"Heins, Jonathan","last_name":"Heins","first_name":"Jonathan"},{"last_name":"Rook","full_name":"Rook, Jeroen","first_name":"Jeroen"},{"full_name":"Schäpermeier, Lennart","last_name":"Schäpermeier","first_name":"Lennart"},{"first_name":"Pascal","last_name":"Kerschke","full_name":"Kerschke, Pascal"},{"full_name":"Bossek, Jakob","id":"102979","last_name":"Bossek","orcid":"0000-0002-4121-4668","first_name":"Jakob"},{"first_name":"Heike","last_name":"Trautmann","full_name":"Trautmann, Heike"}],"editor":[{"first_name":"Günter","full_name":"Rudolph, Günter","last_name":"Rudolph"},{"first_name":"Anna V.","full_name":"Kononova, Anna V.","last_name":"Kononova"},{"full_name":"Aguirre, Hernán","last_name":"Aguirre","first_name":"Hernán"},{"first_name":"Pascal","last_name":"Kerschke","full_name":"Kerschke, Pascal"},{"last_name":"Ochoa","full_name":"Ochoa, Gabriela","first_name":"Gabriela"},{"first_name":"Tea","full_name":"Tusar, Tea","last_name":"Tusar"}],"status":"public","type":"conference","extern":"1","_id":"48882","series_title":"Lecture Notes in Computer Science","user_id":"102979","department":[{"_id":"819"}]},{"status":"public","abstract":[{"text":"To provide user interfaces for a rich set of devices and interaction modalities, we follow a model-based development methodology. We devised an architecture which deploys user interfaces specified as dialogue models with abstract interaction objects and allows context-based adaptations by means of an external transcoding process. For the validation of the applicability of this methodology for developing usable multimodal multi-device systems, we present two case studies based on proof-of-concept implementations and assessed them with a large set of established design principles and different types of modality cooperation.","lang":"eng"}],"type":"journal_article","publication":"Journal on Multimodal User Interfaces","language":[{"iso":"eng"}],"keyword":["Interaction architecture     Abstract interaction objects     Dialogue model     Transformations     Multimodality     Multi-device     Design principles"],"user_id":"5786","department":[{"_id":"672"}],"_id":"34564","citation":{"ieee":"R. Schäfer and W. Müller, “Assessment of a Multimodal Interaction and Rendering System against Established Design Principles,” <i>Journal on Multimodal User Interfaces</i>, vol. 2, no. 1, pp. 25–41, 2008, doi: <a href=\"https://doi.org/10.1007/s12193-008-0003-3\">10.1007/s12193-008-0003-3</a>.","chicago":"Schäfer, Robbie, and Wolfgang Müller. “Assessment of a Multimodal Interaction and Rendering System against Established Design Principles.” <i>Journal on Multimodal User Interfaces</i> 2, no. 1 (2008): 25–41. <a href=\"https://doi.org/10.1007/s12193-008-0003-3\">https://doi.org/10.1007/s12193-008-0003-3</a>.","ama":"Schäfer R, Müller W. Assessment of a Multimodal Interaction and Rendering System against Established Design Principles. <i>Journal on Multimodal User Interfaces</i>. 2008;2(1):25-41. doi:<a href=\"https://doi.org/10.1007/s12193-008-0003-3\">10.1007/s12193-008-0003-3</a>","mla":"Schäfer, Robbie, and Wolfgang Müller. “Assessment of a Multimodal Interaction and Rendering System against Established Design Principles.” <i>Journal on Multimodal User Interfaces</i>, vol. 2, no. 1, Springer-Verlag, 2008, pp. 25–41, doi:<a href=\"https://doi.org/10.1007/s12193-008-0003-3\">10.1007/s12193-008-0003-3</a>.","short":"R. Schäfer, W. Müller, Journal on Multimodal User Interfaces 2 (2008) 25–41.","bibtex":"@article{Schäfer_Müller_2008, title={Assessment of a Multimodal Interaction and Rendering System against Established Design Principles}, volume={2}, DOI={<a href=\"https://doi.org/10.1007/s12193-008-0003-3\">10.1007/s12193-008-0003-3</a>}, number={1}, journal={Journal on Multimodal User Interfaces}, publisher={Springer-Verlag}, author={Schäfer, Robbie and Müller, Wolfgang}, year={2008}, pages={25–41} }","apa":"Schäfer, R., &#38; Müller, W. (2008). Assessment of a Multimodal Interaction and Rendering System against Established Design Principles. <i>Journal on Multimodal User Interfaces</i>, <i>2</i>(1), 25–41. <a href=\"https://doi.org/10.1007/s12193-008-0003-3\">https://doi.org/10.1007/s12193-008-0003-3</a>"},"page":"25-41","intvolume":"         2","year":"2008","issue":"1","doi":"10.1007/s12193-008-0003-3","title":"Assessment of a Multimodal Interaction and Rendering System against Established Design Principles","date_created":"2022-12-19T12:22:50Z","author":[{"last_name":"Schäfer","full_name":"Schäfer, Robbie","first_name":"Robbie"},{"first_name":"Wolfgang","last_name":"Müller","id":"16243","full_name":"Müller, Wolfgang"}],"volume":2,"date_updated":"2022-12-19T12:23:05Z","publisher":"Springer-Verlag"}]