---
_id: '52662'
abstract:
- lang: eng
  text: Static analysis tools support developers in detecting potential coding issues,
    such as bugs or vulnerabilities. Research emphasizes technical challenges of such
    tools but also mentions severe usability shortcomings. These shortcomings hinder
    the adoption of static analysis tools, and user dissatisfaction may even lead
    to tool abandonment. To comprehensively assess the state of the art, we present
    the first systematic usability evaluation of a wide range of static analysis tools.
    We derived a set of 36 relevant criteria from the literature and used them to
    evaluate a total of 46 static analysis tools complying with our inclusion and
    exclusion criteria - a representative set of mainly non-proprietary tools. The
    evaluation against the usability criteria in a multiple-raters approach shows
    that two thirds of the considered tools off er poor warning messages, while about
    three-quarters provide hardly any fix support. Furthermore, the integration of
    user knowledge is strongly neglected, which could be used for instance, to improve
    handling of false positives. Finally, issues regarding workflow integration and
    specialized user interfaces are revealed. These findings should prove useful in
    guiding and focusing further research and development in user experience for static
    code analyses.
author:
- first_name: Marcus
  full_name: Nachtigall, Marcus
  id: '41213'
  last_name: Nachtigall
- first_name: Michael
  full_name: Schlichtig, Michael
  id: '32312'
  last_name: Schlichtig
  orcid: 0000-0001-6600-6171
- first_name: Eric
  full_name: Bodden, Eric
  id: '59256'
  last_name: Bodden
  orcid: 0000-0003-3470-3647
citation:
  ama: 'Nachtigall M, Schlichtig M, Bodden E. Evaluation of Usability Criteria Addressed
    by Static Analysis Tools on a Large Scale. In: <i>Software Engineering 2023</i>.
    Gesellschaft für Informatik e.V.; 2023:95–96.'
  apa: Nachtigall, M., Schlichtig, M., &#38; Bodden, E. (2023). Evaluation of Usability
    Criteria Addressed by Static Analysis Tools on a Large Scale. In <i>Software Engineering
    2023</i> (pp. 95–96). Gesellschaft für Informatik e.V.
  bibtex: '@inbook{Nachtigall_Schlichtig_Bodden_2023, place={Bonn}, title={Evaluation
    of Usability Criteria Addressed by Static Analysis Tools on a Large Scale}, booktitle={Software
    Engineering 2023}, publisher={Gesellschaft für Informatik e.V.}, author={Nachtigall,
    Marcus and Schlichtig, Michael and Bodden, Eric}, year={2023}, pages={95–96} }'
  chicago: 'Nachtigall, Marcus, Michael Schlichtig, and Eric Bodden. “Evaluation of
    Usability Criteria Addressed by Static Analysis Tools on a Large Scale.” In <i>Software
    Engineering 2023</i>, 95–96. Bonn: Gesellschaft für Informatik e.V., 2023.'
  ieee: 'M. Nachtigall, M. Schlichtig, and E. Bodden, “Evaluation of Usability Criteria
    Addressed by Static Analysis Tools on a Large Scale,” in <i>Software Engineering
    2023</i>, Bonn: Gesellschaft für Informatik e.V., 2023, pp. 95–96.'
  mla: Nachtigall, Marcus, et al. “Evaluation of Usability Criteria Addressed by Static
    Analysis Tools on a Large Scale.” <i>Software Engineering 2023</i>, Gesellschaft
    für Informatik e.V., 2023, pp. 95–96.
  short: 'M. Nachtigall, M. Schlichtig, E. Bodden, in: Software Engineering 2023,
    Gesellschaft für Informatik e.V., Bonn, 2023, pp. 95–96.'
date_created: 2024-03-20T09:26:29Z
date_updated: 2024-03-20T09:27:41Z
department:
- _id: '76'
keyword:
- Automated static analysis
- Software usability
language:
- iso: eng
main_file_link:
- url: https://dl.gi.de/items/5afe477f-2f6a-4b3d-b391-f024baf0b7a5
page: 95–96
place: Bonn
publication: Software Engineering 2023
publication_identifier:
  isbn:
  - 978-3-88579-726-5
publisher: Gesellschaft für Informatik e.V.
status: public
title: Evaluation of Usability Criteria Addressed by Static Analysis Tools on a Large
  Scale
type: book_chapter
user_id: '32312'
year: '2023'
...
---
_id: '32410'
abstract:
- lang: eng
  text: "Static analysis tools support developers in detecting potential coding issues,
    such as bugs or vulnerabilities. Research on static analysis emphasizes its technical
    challenges but also mentions severe usability shortcomings. These shortcomings
    hinder the adoption of static analysis tools, and in some cases, user dissatisfaction
    even leads to tool abandonment.\r\nTo comprehensively assess the current state
    of the art, this paper presents the first systematic usability evaluation in a
    wide range of static analysis tools. We derived a set of 36 relevant criteria
    from the scientific literature and gathered a collection of 46 static analysis
    tools complying with our inclusion and exclusion criteria - a representative set
    of mainly non-proprietary tools. Then, we evaluated how well these tools fulfill
    the aforementioned criteria.\r\nThe evaluation shows that more than half of the
    considered tools offer poor warning messages, while about three-quarters of the
    tools provide hardly any fix support. Furthermore, the integration of user knowledge
    is strongly neglected, which could be used for improved handling of false positives
    and tuning the results for the corresponding developer. Finally, issues regarding
    workflow integration and specialized user interfaces are proved further.\r\nThese
    findings should prove useful in guiding and focusing further research and development
    in the area of user experience for static code analyses."
author:
- first_name: Marcus
  full_name: Nachtigall, Marcus
  id: '41213'
  last_name: Nachtigall
- first_name: Michael
  full_name: Schlichtig, Michael
  id: '32312'
  last_name: Schlichtig
  orcid: 0000-0001-6600-6171
- first_name: Eric
  full_name: Bodden, Eric
  id: '59256'
  last_name: Bodden
  orcid: 0000-0003-3470-3647
citation:
  ama: 'Nachtigall M, Schlichtig M, Bodden E. A Large-Scale Study of Usability Criteria
    Addressed by Static Analysis Tools. In: <i>Proceedings of the 31st ACM SIGSOFT
    International Symposium on Software Testing and Analysis</i>. ACM; 2022:532-543.
    doi:<a href="https://doi.org/10.1145/3533767">10.1145/3533767</a>'
  apa: Nachtigall, M., Schlichtig, M., &#38; Bodden, E. (2022). A Large-Scale Study
    of Usability Criteria Addressed by Static Analysis Tools. <i>Proceedings of the
    31st ACM SIGSOFT International Symposium on Software Testing and Analysis</i>,
    532–543. <a href="https://doi.org/10.1145/3533767">https://doi.org/10.1145/3533767</a>
  bibtex: '@inproceedings{Nachtigall_Schlichtig_Bodden_2022, title={A Large-Scale
    Study of Usability Criteria Addressed by Static Analysis Tools}, DOI={<a href="https://doi.org/10.1145/3533767">10.1145/3533767</a>},
    booktitle={Proceedings of the 31st ACM SIGSOFT International Symposium on Software
    Testing and Analysis}, publisher={ACM}, author={Nachtigall, Marcus and Schlichtig,
    Michael and Bodden, Eric}, year={2022}, pages={532–543} }'
  chicago: Nachtigall, Marcus, Michael Schlichtig, and Eric Bodden. “A Large-Scale
    Study of Usability Criteria Addressed by Static Analysis Tools.” In <i>Proceedings
    of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis</i>,
    532–43. ACM, 2022. <a href="https://doi.org/10.1145/3533767">https://doi.org/10.1145/3533767</a>.
  ieee: 'M. Nachtigall, M. Schlichtig, and E. Bodden, “A Large-Scale Study of Usability
    Criteria Addressed by Static Analysis Tools,” in <i>Proceedings of the 31st ACM
    SIGSOFT International Symposium on Software Testing and Analysis</i>, 2022, pp.
    532–543, doi: <a href="https://doi.org/10.1145/3533767">10.1145/3533767</a>.'
  mla: Nachtigall, Marcus, et al. “A Large-Scale Study of Usability Criteria Addressed
    by Static Analysis Tools.” <i>Proceedings of the 31st ACM SIGSOFT International
    Symposium on Software Testing and Analysis</i>, ACM, 2022, pp. 532–43, doi:<a
    href="https://doi.org/10.1145/3533767">10.1145/3533767</a>.
  short: 'M. Nachtigall, M. Schlichtig, E. Bodden, in: Proceedings of the 31st ACM
    SIGSOFT International Symposium on Software Testing and Analysis, ACM, 2022, pp.
    532–543.'
date_created: 2022-07-25T08:02:36Z
date_updated: 2022-07-26T11:42:23Z
department:
- _id: '76'
doi: 10.1145/3533767
keyword:
- Automated static analysis
- Software usability
language:
- iso: eng
page: 532 - 543
publication: Proceedings of the 31st ACM SIGSOFT International Symposium on Software
  Testing and Analysis
publication_identifier:
  isbn:
  - '9781450393799'
publication_status: published
publisher: ACM
quality_controlled: '1'
related_material:
  link:
  - relation: confirmation
    url: https://dl.acm.org/doi/10.1145/3533767.3534374
status: public
title: A Large-Scale Study of Usability Criteria Addressed by Static Analysis Tools
type: conference
user_id: '32312'
year: '2022'
...