---
_id: '22156'
abstract:
- lang: eng
  text: Word embedding models reflect bias towards genders, ethnicities, and other
    social groups present in the underlying training data. Metrics such as ECT, RNSB,
    and WEAT quantify bias in these models based on predefined word lists representing
    social groups and bias-conveying concepts. How suitable these lists actually are
    to reveal bias - let alone the bias metrics in general - remains unclear, though.
    In this paper, we study how to assess the quality of bias metrics for word embedding
    models. In particular, we present a generic method, Bias Silhouette Analysis (BSA),
    that quantifies the accuracy and robustness of such a metric and of the word lists
    used. Given a biased and an unbiased reference embedding model, BSA applies the
    metric systematically for several subsets of the lists to the models. The variance
    and rate of convergence of the bias values of each model then entail the robustness
    of the word lists, whereas the distance between the models' values gives indications
    of the general accuracy of the metric with the word lists. We demonstrate the
    behavior of BSA on two standard embedding models for the three mentioned metrics
    with several word lists from existing research.
author:
- first_name: Maximilian
  full_name: Spliethöver, Maximilian
  id: '84035'
  last_name: Spliethöver
  orcid: 0000-0003-4364-1409
- first_name: Henning
  full_name: Wachsmuth, Henning
  id: '3900'
  last_name: Wachsmuth
citation:
  ama: 'Spliethöver M, Wachsmuth H. Bias Silhouette Analysis: Towards Assessing the
    Quality of Bias Metrics for Word Embedding Models. In: <i>Proceedings of the Thirtieth
    International Joint Conference on Artificial Intelligence, IJCAI-21</i>. ; 2021:552-559.
    doi:<a href="https://doi.org/10.24963/ijcai.2021/77">10.24963/ijcai.2021/77</a>'
  apa: 'Spliethöver, M., &#38; Wachsmuth, H. (2021). Bias Silhouette Analysis: Towards
    Assessing the Quality of Bias Metrics for Word Embedding Models. <i>Proceedings
    of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21</i>,
    552–559. <a href="https://doi.org/10.24963/ijcai.2021/77">https://doi.org/10.24963/ijcai.2021/77</a>'
  bibtex: '@inproceedings{Spliethöver_Wachsmuth_2021, title={Bias Silhouette Analysis:
    Towards Assessing the Quality of Bias Metrics for Word Embedding Models}, DOI={<a
    href="https://doi.org/10.24963/ijcai.2021/77">10.24963/ijcai.2021/77</a>}, booktitle={Proceedings
    of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21},
    author={Spliethöver, Maximilian and Wachsmuth, Henning}, year={2021}, pages={552–559}
    }'
  chicago: 'Spliethöver, Maximilian, and Henning Wachsmuth. “Bias Silhouette Analysis:
    Towards Assessing the Quality of Bias Metrics for Word Embedding Models.” In <i>Proceedings
    of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21</i>,
    552–59, 2021. <a href="https://doi.org/10.24963/ijcai.2021/77">https://doi.org/10.24963/ijcai.2021/77</a>.'
  ieee: 'M. Spliethöver and H. Wachsmuth, “Bias Silhouette Analysis: Towards Assessing
    the Quality of Bias Metrics for Word Embedding Models,” in <i>Proceedings of the
    Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21</i>,
    Online, 2021, pp. 552–559, doi: <a href="https://doi.org/10.24963/ijcai.2021/77">10.24963/ijcai.2021/77</a>.'
  mla: 'Spliethöver, Maximilian, and Henning Wachsmuth. “Bias Silhouette Analysis:
    Towards Assessing the Quality of Bias Metrics for Word Embedding Models.” <i>Proceedings
    of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21</i>,
    2021, pp. 552–59, doi:<a href="https://doi.org/10.24963/ijcai.2021/77">10.24963/ijcai.2021/77</a>.'
  short: 'M. Spliethöver, H. Wachsmuth, in: Proceedings of the Thirtieth International
    Joint Conference on Artificial Intelligence, IJCAI-21, 2021, pp. 552–559.'
conference:
  end_date: 2021-08-26
  location: Online
  name: 30th International Joint Conference on Artificial Intelligence (IJCAI-21)
  start_date: 2021-08-19
date_created: 2021-05-11T23:13:26Z
date_updated: 2022-01-06T06:55:28Z
department:
- _id: '600'
doi: 10.24963/ijcai.2021/77
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://www.ijcai.org/proceedings/2021/77
oa: '1'
page: 552-559
publication: Proceedings of the Thirtieth International Joint Conference on Artificial
  Intelligence, IJCAI-21
quality_controlled: '1'
status: public
title: 'Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for
  Word Embedding Models'
type: conference
user_id: '82920'
year: '2021'
...