---
_id: '53959'
abstract:
- lang: eng
  text: In light of the growing interest in type inference research for Python, both
    researchers and practitioners require a standardized process to assess the performance
    of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive
    micro-benchmarking framework for evaluating type inference tools. TypeEvalPy contains
    154 code snippets with 845 type annotations across 18 categories that target various
    Python features. The framework manages the execution of containerized tools, transforms
    inferred types into a standardized format, and produces meaningful metrics for
    assessment. Through our analysis, we compare the performance of six type inference
    tools, highlighting their strengths and limitations. Our findings provide a foundation
    for further research and optimization in the domain of Python type inference.
author:
- first_name: Ashwin Prasad
  full_name: Shivarpatna Venkatesh, Ashwin Prasad
  id: '66637'
  last_name: Shivarpatna Venkatesh
- first_name: Samkutty
  full_name: Sabu, Samkutty
  last_name: Sabu
- first_name: Jiawei
  full_name: Wang, Jiawei
  last_name: Wang
- first_name: Amir M.
  full_name: Mir, Amir M.
  last_name: Mir
- first_name: Li
  full_name: Li, Li
  last_name: Li
- first_name: Eric
  full_name: Bodden, Eric
  id: '59256'
  last_name: Bodden
  orcid: 0000-0003-3470-3647
citation:
  ama: 'Shivarpatna Venkatesh AP, Sabu S, Wang J, Mir AM, Li L, Bodden E. TypeEvalPy:
    A Micro-benchmarking Framework for Python Type Inference  Tools. In: <i>Proceedings
    of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion
    Proceedings</i>. ICSE-Companion 24. Association for Computing Machinery; 2024:49-53.
    doi:<a href="https://doi.org/10.1145/3639478.3640033">10.1145/3639478.3640033</a>'
  apa: 'Shivarpatna Venkatesh, A. P., Sabu, S., Wang, J., Mir, A. M., Li, L., &#38;
    Bodden, E. (2024). TypeEvalPy: A Micro-benchmarking Framework for Python Type
    Inference  Tools. <i>Proceedings of the 2024 IEEE/ACM 46th International Conference
    on Software Engineering: Companion Proceedings</i>, 49–53. <a href="https://doi.org/10.1145/3639478.3640033">https://doi.org/10.1145/3639478.3640033</a>'
  bibtex: '@inproceedings{Shivarpatna Venkatesh_Sabu_Wang_Mir_Li_Bodden_2024, place={New
    York, NY, USA}, series={ICSE-Companion 24}, title={TypeEvalPy: A Micro-benchmarking
    Framework for Python Type Inference  Tools}, DOI={<a href="https://doi.org/10.1145/3639478.3640033">10.1145/3639478.3640033</a>},
    booktitle={Proceedings of the 2024 IEEE/ACM 46th International Conference on Software
    Engineering: Companion Proceedings}, publisher={Association for Computing Machinery},
    author={Shivarpatna Venkatesh, Ashwin Prasad and Sabu, Samkutty and Wang, Jiawei
    and Mir, Amir M. and Li, Li and Bodden, Eric}, year={2024}, pages={49–53}, collection={ICSE-Companion
    24} }'
  chicago: 'Shivarpatna Venkatesh, Ashwin Prasad, Samkutty Sabu, Jiawei Wang, Amir
    M. Mir, Li Li, and Eric Bodden. “TypeEvalPy: A Micro-Benchmarking Framework for
    Python Type Inference  Tools.” In <i>Proceedings of the 2024 IEEE/ACM 46th International
    Conference on Software Engineering: Companion Proceedings</i>, 49–53. ICSE-Companion
    24. New York, NY, USA: Association for Computing Machinery, 2024. <a href="https://doi.org/10.1145/3639478.3640033">https://doi.org/10.1145/3639478.3640033</a>.'
  ieee: 'A. P. Shivarpatna Venkatesh, S. Sabu, J. Wang, A. M. Mir, L. Li, and E. Bodden,
    “TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference  Tools,”
    in <i>Proceedings of the 2024 IEEE/ACM 46th International Conference on Software
    Engineering: Companion Proceedings</i>, Lisbon, Portugal, 2024, pp. 49–53, doi:
    <a href="https://doi.org/10.1145/3639478.3640033">10.1145/3639478.3640033</a>.'
  mla: 'Shivarpatna Venkatesh, Ashwin Prasad, et al. “TypeEvalPy: A Micro-Benchmarking
    Framework for Python Type Inference  Tools.” <i>Proceedings of the 2024 IEEE/ACM
    46th International Conference on Software Engineering: Companion Proceedings</i>,
    Association for Computing Machinery, 2024, pp. 49–53, doi:<a href="https://doi.org/10.1145/3639478.3640033">10.1145/3639478.3640033</a>.'
  short: 'A.P. Shivarpatna Venkatesh, S. Sabu, J. Wang, A.M. Mir, L. Li, E. Bodden,
    in: Proceedings of the 2024 IEEE/ACM 46th International Conference on Software
    Engineering: Companion Proceedings, Association for Computing Machinery, New York,
    NY, USA, 2024, pp. 49–53.'
conference:
  location: Lisbon, Portugal
date_created: 2024-05-06T11:49:22Z
date_updated: 2024-08-05T07:49:33Z
department:
- _id: '76'
doi: 10.1145/3639478.3640033
external_id:
  arxiv:
  - '2312.16882'
language:
- iso: eng
page: 49-53
place: New York, NY, USA
publication: 'Proceedings of the 2024 IEEE/ACM 46th International Conference on Software
  Engineering: Companion Proceedings'
publication_identifier:
  isbn:
  - '9798400705021'
publisher: Association for Computing Machinery
series_title: ICSE-Companion 24
status: public
title: 'TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference  Tools'
type: conference
user_id: '15249'
year: '2024'
...