---
_id: '49439'
abstract:
- lang: eng
  text: <jats:title>Abstract</jats:title><jats:p>The use of static analysis security
    testing (SAST) tools has been increasing in recent years. However, previous studies
    have shown that, when shipped to end users such as development or security teams,
    the findings of these tools are often unsatisfying. Users report high numbers
    of false positives or long analysis times, making the tools unusable in the daily
    workflow. To address this, SAST tool creators provide a wide range of configuration
    options, such as customization of rules through domain-specific languages or specification
    of the application-specific analysis scope. In this paper, we study the configuration
    space of selected existing SAST tools when used within the integrated development
    environment (IDE). We focus on the configuration options that impact three dimensions,
    for which a trade-off is unavoidable, i.e., precision, recall, and analysis runtime.
    We perform a between-subjects user study with 40 users from multiple development
    and security teams - to our knowledge, the largest population for this kind of
    user study in the software engineering community. The results show that users
    who configure SAST tools are more effective in resolving security vulnerabilities
    detected by the tools than those using the default configuration. Based on post-study
    interviews, we identify common strategies that users have while configuring the
    SAST tools to provide further insights for tool creators. Finally, an evaluation
    of the configuration options of two commercial SAST tools, <jats:sc>Fortify</jats:sc>
    and <jats:sc>CheckMarx</jats:sc>, reveals that a quarter of the users do not understand
    the configuration options provided. The configuration options that are found most
    useful relate to the analysis scope.</jats:p>
article_number: '118'
author:
- first_name: Goran
  full_name: Piskachev, Goran
  id: '41936'
  last_name: Piskachev
  orcid: 0000-0003-4424-5838
- first_name: Matthias
  full_name: Becker, Matthias
  id: '4870'
  last_name: Becker
  orcid: https://orcid.org/0000-0003-2465-9347
- first_name: Eric
  full_name: Bodden, Eric
  id: '59256'
  last_name: Bodden
  orcid: 0000-0003-3470-3647
citation:
  ama: Piskachev G, Becker M, Bodden E. Can the configuration of static analyses make
    resolving security vulnerabilities more effective? - A user study. <i>Empirical
    Software Engineering</i>. 2023;28(5). doi:<a href="https://doi.org/10.1007/s10664-023-10354-3">10.1007/s10664-023-10354-3</a>
  apa: Piskachev, G., Becker, M., &#38; Bodden, E. (2023). Can the configuration of
    static analyses make resolving security vulnerabilities more effective? - A user
    study. <i>Empirical Software Engineering</i>, <i>28</i>(5), Article 118. <a href="https://doi.org/10.1007/s10664-023-10354-3">https://doi.org/10.1007/s10664-023-10354-3</a>
  bibtex: '@article{Piskachev_Becker_Bodden_2023, title={Can the configuration of
    static analyses make resolving security vulnerabilities more effective? - A user
    study}, volume={28}, DOI={<a href="https://doi.org/10.1007/s10664-023-10354-3">10.1007/s10664-023-10354-3</a>},
    number={5118}, journal={Empirical Software Engineering}, publisher={Springer Science
    and Business Media LLC}, author={Piskachev, Goran and Becker, Matthias and Bodden,
    Eric}, year={2023} }'
  chicago: Piskachev, Goran, Matthias Becker, and Eric Bodden. “Can the Configuration
    of Static Analyses Make Resolving Security Vulnerabilities More Effective? - A
    User Study.” <i>Empirical Software Engineering</i> 28, no. 5 (2023). <a href="https://doi.org/10.1007/s10664-023-10354-3">https://doi.org/10.1007/s10664-023-10354-3</a>.
  ieee: 'G. Piskachev, M. Becker, and E. Bodden, “Can the configuration of static
    analyses make resolving security vulnerabilities more effective? - A user study,”
    <i>Empirical Software Engineering</i>, vol. 28, no. 5, Art. no. 118, 2023, doi:
    <a href="https://doi.org/10.1007/s10664-023-10354-3">10.1007/s10664-023-10354-3</a>.'
  mla: Piskachev, Goran, et al. “Can the Configuration of Static Analyses Make Resolving
    Security Vulnerabilities More Effective? - A User Study.” <i>Empirical Software
    Engineering</i>, vol. 28, no. 5, 118, Springer Science and Business Media LLC,
    2023, doi:<a href="https://doi.org/10.1007/s10664-023-10354-3">10.1007/s10664-023-10354-3</a>.
  short: G. Piskachev, M. Becker, E. Bodden, Empirical Software Engineering 28 (2023).
date_created: 2023-12-04T11:14:34Z
date_updated: 2023-12-04T11:29:49Z
department:
- _id: '76'
- _id: '662'
doi: 10.1007/s10664-023-10354-3
intvolume: '        28'
issue: '5'
keyword:
- Software
language:
- iso: eng
publication: Empirical Software Engineering
publication_identifier:
  issn:
  - 1382-3256
  - 1573-7616
publication_status: published
publisher: Springer Science and Business Media LLC
status: public
title: Can the configuration of static analyses make resolving security vulnerabilities
  more effective? - A user study
type: journal_article
user_id: '15249'
volume: 28
year: '2023'
...
---
_id: '27045'
abstract:
- lang: eng
  text: 'Due to the lack of established real-world benchmark suites for static taint
    analyses of Android applications, evaluations of these analyses are often restricted
    and hard to compare. Even in evaluations that do use real-world apps, details
    about the ground truth in those apps are rarely documented, which makes it difficult
    to compare and reproduce the results. To push Android taint analysis research
    forward, this paper thus recommends criteria for constructing real-world benchmark
    suites for this specific domain, and presents TaintBench, the first real-world
    malware benchmark suite with documented taint flows. TaintBench benchmark apps
    include taint flows with complex structures, and addresses static challenges that
    are commonly agreed on by the community. Together with the TaintBench suite, we
    introduce the TaintBench framework, whose goal is to simplify real-world benchmarking
    of Android taint analyses. First, a usability test shows that the framework improves
    experts’ performance and perceived usability when documenting and inspecting taint
    flows. Second, experiments using TaintBench reveal new insights for the taint
    analysis tools Amandroid and FlowDroid: (i) They are less effective on real-world
    malware apps than on synthetic benchmark apps. (ii) Predefined lists of sources
    and sinks heavily impact the tools’ accuracy. (iii) Surprisingly, up-to-date versions
    of both tools are less accurate than their predecessors.'
author:
- first_name: Linghui
  full_name: Luo, Linghui
  last_name: Luo
- first_name: Felix
  full_name: Pauck, Felix
  id: '22398'
  last_name: Pauck
- first_name: Goran
  full_name: Piskachev, Goran
  id: '41936'
  last_name: Piskachev
  orcid: 0000-0003-4424-5838
- first_name: Manuel
  full_name: Benz, Manuel
  last_name: Benz
- first_name: Ivan
  full_name: Pashchenko, Ivan
  last_name: Pashchenko
- first_name: Martin
  full_name: Mory, Martin
  id: '65667'
  last_name: Mory
  orcid: 0000-0001-5609-0031
- first_name: Eric
  full_name: Bodden, Eric
  id: '59256'
  last_name: Bodden
  orcid: 0000-0003-3470-3647
- first_name: Ben
  full_name: Hermann, Ben
  id: '66173'
  last_name: Hermann
  orcid: 0000-0001-9848-2017
- first_name: Fabio
  full_name: Massacci, Fabio
  last_name: Massacci
citation:
  ama: 'Luo L, Pauck F, Piskachev G, et al. TaintBench: Automatic real-world malware
    benchmarking of Android taint analyses. <i>Empirical Software Engineering</i>.
    Published online 2021. doi:<a href="https://doi.org/10.1007/s10664-021-10013-5">10.1007/s10664-021-10013-5</a>'
  apa: 'Luo, L., Pauck, F., Piskachev, G., Benz, M., Pashchenko, I., Mory, M., Bodden,
    E., Hermann, B., &#38; Massacci, F. (2021). TaintBench: Automatic real-world malware
    benchmarking of Android taint analyses. <i>Empirical Software Engineering</i>.
    <a href="https://doi.org/10.1007/s10664-021-10013-5">https://doi.org/10.1007/s10664-021-10013-5</a>'
  bibtex: '@article{Luo_Pauck_Piskachev_Benz_Pashchenko_Mory_Bodden_Hermann_Massacci_2021,
    title={TaintBench: Automatic real-world malware benchmarking of Android taint
    analyses}, DOI={<a href="https://doi.org/10.1007/s10664-021-10013-5">10.1007/s10664-021-10013-5</a>},
    journal={Empirical Software Engineering}, author={Luo, Linghui and Pauck, Felix
    and Piskachev, Goran and Benz, Manuel and Pashchenko, Ivan and Mory, Martin and
    Bodden, Eric and Hermann, Ben and Massacci, Fabio}, year={2021} }'
  chicago: 'Luo, Linghui, Felix Pauck, Goran Piskachev, Manuel Benz, Ivan Pashchenko,
    Martin Mory, Eric Bodden, Ben Hermann, and Fabio Massacci. “TaintBench: Automatic
    Real-World Malware Benchmarking of Android Taint Analyses.” <i>Empirical Software
    Engineering</i>, 2021. <a href="https://doi.org/10.1007/s10664-021-10013-5">https://doi.org/10.1007/s10664-021-10013-5</a>.'
  ieee: 'L. Luo <i>et al.</i>, “TaintBench: Automatic real-world malware benchmarking
    of Android taint analyses,” <i>Empirical Software Engineering</i>, 2021, doi:
    <a href="https://doi.org/10.1007/s10664-021-10013-5">10.1007/s10664-021-10013-5</a>.'
  mla: 'Luo, Linghui, et al. “TaintBench: Automatic Real-World Malware Benchmarking
    of Android Taint Analyses.” <i>Empirical Software Engineering</i>, 2021, doi:<a
    href="https://doi.org/10.1007/s10664-021-10013-5">10.1007/s10664-021-10013-5</a>.'
  short: L. Luo, F. Pauck, G. Piskachev, M. Benz, I. Pashchenko, M. Mory, E. Bodden,
    B. Hermann, F. Massacci, Empirical Software Engineering (2021).
date_created: 2021-11-02T05:13:49Z
date_updated: 2022-01-06T06:57:32Z
ddc:
- '000'
department:
- _id: '77'
- _id: '76'
doi: 10.1007/s10664-021-10013-5
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://link.springer.com/content/pdf/10.1007/s10664-021-10013-5.pdf
oa: '1'
project:
- _id: '1'
  name: SFB 901
- _id: '3'
  name: SFB 901 - Project Area B
- _id: '12'
  name: SFB 901 - Subproject B4
publication: Empirical Software Engineering
publication_identifier:
  issn:
  - 1382-3256
  - 1573-7616
publication_status: published
status: public
title: 'TaintBench: Automatic real-world malware benchmarking of Android taint analyses'
type: journal_article
user_id: '15249'
year: '2021'
...