Quote to explain: Using multimodal metalinguistic markers to explain large language models’ understanding capabilities

Starting from the assumption that LLMs are systems bearing only formal but not functional linguistic competence, this short paper explores how the understanding capabilities of LLMs could be implicitly explained based on a “pause and refect” strategy. Specifically, we propose to include a virtual embodied agent in human interactions with LLM-based chatbots. The agent will use air quotes as multimodal metalinguistic markers to explicitly point to those parts of the LLM’s output that are relevant to explaining the LLM’s meaning understanding capabilities. At the same time, by scafolding users to perceive the output as ‘mentioned language’ inferred from a metalinguistic function of multimodal markers, the agent implicitly explains how the meaning of the output should be understood. In this proposal, users will actively participate in the co-construction of the implicit explanation by providing feedback and deciding when and to what extent the agent’s scafold (e.g., the air quotes) is used.