Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models

M. Spliethöver, H. Wachsmuth, in: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 2021, pp. 552–559.

Conference Paper | English
Abstract
Word embedding models reflect bias towards genders, ethnicities, and other social groups present in the underlying training data. Metrics such as ECT, RNSB, and WEAT quantify bias in these models based on predefined word lists representing social groups and bias-conveying concepts. How suitable these lists actually are to reveal bias - let alone the bias metrics in general - remains unclear, though. In this paper, we study how to assess the quality of bias metrics for word embedding models. In particular, we present a generic method, Bias Silhouette Analysis (BSA), that quantifies the accuracy and robustness of such a metric and of the word lists used. Given a biased and an unbiased reference embedding model, BSA applies the metric systematically for several subsets of the lists to the models. The variance and rate of convergence of the bias values of each model then entail the robustness of the word lists, whereas the distance between the models' values gives indications of the general accuracy of the metric with the word lists. We demonstrate the behavior of BSA on two standard embedding models for the three mentioned metrics with several word lists from existing research.
Publishing Year
Proceedings Title
Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21
Page
552-559
Conference
30th International Joint Conference on Artificial Intelligence (IJCAI-21)
Conference Location
Online
Conference Date
2021-08-19 – 2021-08-26
LibreCat-ID

Cite this

Spliethöver M, Wachsmuth H. Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. ; 2021:552-559. doi:10.24963/ijcai.2021/77
Spliethöver, M., & Wachsmuth, H. (2021). Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 552–559. https://doi.org/10.24963/ijcai.2021/77
@inproceedings{Spliethöver_Wachsmuth_2021, title={Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models}, DOI={10.24963/ijcai.2021/77}, booktitle={Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21}, author={Spliethöver, Maximilian and Wachsmuth, Henning}, year={2021}, pages={552–559} }
Spliethöver, Maximilian, and Henning Wachsmuth. “Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models.” In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 552–59, 2021. https://doi.org/10.24963/ijcai.2021/77.
M. Spliethöver and H. Wachsmuth, “Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models,” in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, Online, 2021, pp. 552–559, doi: 10.24963/ijcai.2021/77.
Spliethöver, Maximilian, and Henning Wachsmuth. “Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models.” Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 2021, pp. 552–59, doi:10.24963/ijcai.2021/77.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
Restricted Closed Access

Export

Marked Publications

Open Data LibreCat

Search this title in

Google Scholar