---
_id: '61290'
abstract:
- lang: eng
  text: ffective computing often relies on audiovisual data to identify affective
    states from non-verbal signals, such as facial expressions and vocal cues. Since
    automatic affect recognition can be used in sensitive applications, such as healthcare
    and education, it is crucial to understand how models arrive at their decisions.
    Interpretability of machine learning models is the goal of the emerging research
    area of Explainable AI (explainable AI (XAI)). This scoping review aims to survey
    the field of audiovisual affective machine learning to identify how XAI is applied
    in this domain. We first provide an overview of XAI concepts relevant to affective
    computing. Next, following the recommended PRISMA guidelines, we perform a literature
    search in the ACM, IEEE, Web of Science and PubMed databases. After systematically
    reviewing 1190 articles, a final set of 65 papers is included in our analysis.
    We quantitatively summarize the scope, methods and evaluation of the XAI techniques
    used in the identified papers. Our findings show encouraging developments for
    using XAI to explain models in audiovisual affective computing, yet only a limited
    set of methods are used in the reviewed works. Following a critical discussion,
    we provide recommendations for incorporating interpretability in future work for
    affective machine learnin
article_type: review
author:
- first_name: David
  full_name: Johnson, David
  id: '97208'
  last_name: Johnson
- first_name: Olya
  full_name: Hakobyan, Olya
  last_name: Hakobyan
- first_name: Jonas
  full_name: Paletschek, Jonas
  id: '98941'
  last_name: Paletschek
- first_name: Hanna
  full_name: Drimalla, Hanna
  last_name: Drimalla
citation:
  ama: 'Johnson D, Hakobyan O, Paletschek J, Drimalla H. Explainable AI for Audio
    and Visual Affective Computing: A Scoping Review. <i>IEEE Transactions on Affective
    Computing</i>. 2024;16(2):518-536. doi:<a href="https://doi.org/10.1109/taffc.2024.3505269">10.1109/taffc.2024.3505269</a>'
  apa: 'Johnson, D., Hakobyan, O., Paletschek, J., &#38; Drimalla, H. (2024). Explainable
    AI for Audio and Visual Affective Computing: A Scoping Review. <i>IEEE Transactions
    on Affective Computing</i>, <i>16</i>(2), 518–536. <a href="https://doi.org/10.1109/taffc.2024.3505269">https://doi.org/10.1109/taffc.2024.3505269</a>'
  bibtex: '@article{Johnson_Hakobyan_Paletschek_Drimalla_2024, title={Explainable
    AI for Audio and Visual Affective Computing: A Scoping Review}, volume={16}, DOI={<a
    href="https://doi.org/10.1109/taffc.2024.3505269">10.1109/taffc.2024.3505269</a>},
    number={2}, journal={IEEE Transactions on Affective Computing}, publisher={Institute
    of Electrical and Electronics Engineers (IEEE)}, author={Johnson, David and Hakobyan,
    Olya and Paletschek, Jonas and Drimalla, Hanna}, year={2024}, pages={518–536}
    }'
  chicago: 'Johnson, David, Olya Hakobyan, Jonas Paletschek, and Hanna Drimalla. “Explainable
    AI for Audio and Visual Affective Computing: A Scoping Review.” <i>IEEE Transactions
    on Affective Computing</i> 16, no. 2 (2024): 518–36. <a href="https://doi.org/10.1109/taffc.2024.3505269">https://doi.org/10.1109/taffc.2024.3505269</a>.'
  ieee: 'D. Johnson, O. Hakobyan, J. Paletschek, and H. Drimalla, “Explainable AI
    for Audio and Visual Affective Computing: A Scoping Review,” <i>IEEE Transactions
    on Affective Computing</i>, vol. 16, no. 2, pp. 518–536, 2024, doi: <a href="https://doi.org/10.1109/taffc.2024.3505269">10.1109/taffc.2024.3505269</a>.'
  mla: 'Johnson, David, et al. “Explainable AI for Audio and Visual Affective Computing:
    A Scoping Review.” <i>IEEE Transactions on Affective Computing</i>, vol. 16, no.
    2, Institute of Electrical and Electronics Engineers (IEEE), 2024, pp. 518–36,
    doi:<a href="https://doi.org/10.1109/taffc.2024.3505269">10.1109/taffc.2024.3505269</a>.'
  short: D. Johnson, O. Hakobyan, J. Paletschek, H. Drimalla, IEEE Transactions on
    Affective Computing 16 (2024) 518–536.
date_created: 2025-09-16T07:24:07Z
date_updated: 2025-09-16T08:02:23Z
ddc:
- '000'
department:
- _id: '660'
doi: 10.1109/taffc.2024.3505269
file:
- access_level: closed
  content_type: application/pdf
  creator: johnson
  date_created: 2025-09-16T07:34:27Z
  date_updated: 2025-09-16T07:34:27Z
  file_id: '61291'
  file_name: Explainable_AI_for_Audio_and_Visual_Affective_Computing_A_Scoping_Review.pdf
  file_size: 3252812
  relation: main_file
  success: 1
file_date_updated: 2025-09-16T07:34:27Z
has_accepted_license: '1'
intvolume: '        16'
issue: '2'
language:
- iso: eng
page: 518-536
project:
- _id: '110'
  name: TRR 318 - Project Area A
- _id: '1204'
  name: TRR 318 - Teilprojekt IRG BI
- _id: '1200'
  name: TRR 318 - Teilprojekt A6 - Inklusive Ko-Konstruktion sozialer Signale des
    Verstehens
publication: IEEE Transactions on Affective Computing
publication_identifier:
  issn:
  - 1949-3045
  - 2371-9850
publication_status: published
publisher: Institute of Electrical and Electronics Engineers (IEEE)
status: public
title: 'Explainable AI for Audio and Visual Affective Computing: A Scoping Review'
type: journal_article
user_id: '97208'
volume: 16
year: '2024'
...
---
_id: '51372'
abstract:
- lang: eng
  text: Machine learning is frequently used in affective computing, but presents challenges
    due the opacity of state-of-the-art machine learning methods. Because of the impact
    affective machine learning systems may have on an individual's life, it is important
    that models be made transparent to detect and mitigate biased decision making.
    In this regard, affective machine learning could benefit from the recent advancements
    in explainable artificial intelligence (XAI) research. We perform a structured
    literature review to examine the use of interpretability in the context of affective
    machine learning. We focus on studies using audio, visual, or audiovisual data
    for model training and identified 29 research articles. Our findings show an emergence
    of the use of interpretability methods in the last five years. However, their
    use is currently limited regarding the range of methods used, the depth of evaluations,
    and the consideration of use-cases. We outline the main gaps in the research and
    provide recommendations for researchers that aim to implement interpretable methods
    for affective machine learning.
author:
- first_name: 'David '
  full_name: 'Johnson, David '
  last_name: Johnson
- first_name: Olya
  full_name: Hakobyan, Olya
  last_name: Hakobyan
- first_name: Hanna
  full_name: Drimalla, Hanna
  last_name: Drimalla
citation:
  ama: 'Johnson D, Hakobyan O, Drimalla H. Towards Interpretability in Audio and Visual
    Affective Machine Learning: A Review. Published online 2023.'
  apa: 'Johnson, D., Hakobyan, O., &#38; Drimalla, H. (2023). <i>Towards Interpretability
    in Audio and Visual Affective Machine Learning: A Review</i>.'
  bibtex: '@article{Johnson_Hakobyan_Drimalla_2023, title={Towards Interpretability
    in Audio and Visual Affective Machine Learning: A Review}, author={Johnson, David  and
    Hakobyan, Olya and Drimalla, Hanna}, year={2023} }'
  chicago: 'Johnson, David , Olya Hakobyan, and Hanna Drimalla. “Towards Interpretability
    in Audio and Visual Affective Machine Learning: A Review,” 2023.'
  ieee: 'D. Johnson, O. Hakobyan, and H. Drimalla, “Towards Interpretability in Audio
    and Visual Affective Machine Learning: A Review.” 2023.'
  mla: 'Johnson, David, et al. <i>Towards Interpretability in Audio and Visual Affective
    Machine Learning: A Review</i>. 2023.'
  short: D. Johnson, O. Hakobyan, H. Drimalla, (2023).
date_created: 2024-02-18T10:52:36Z
date_updated: 2024-02-26T08:43:01Z
department:
- _id: '660'
language:
- iso: eng
project:
- _id: '110'
  name: 'TRR 318 - A: TRR 318 - Project Area A'
status: public
title: 'Towards Interpretability in Audio and Visual Affective Machine Learning: A
  Review'
type: preprint
user_id: '54779'
year: '2023'
...