---
_id: '66837'
abstract:
- lang: eng
  text: 'Manufacturers operate under dynamic regulatory conditions, leading to frequently
    changing requirements that must be managed with high effort. AI-based requirements
    extraction using Large Language Models (LLMs) has the potential to support engineers
    in this task by analyzing large documents, for example. Existing evaluation approaches
    in AI-supported requirements engineering mainly assess individual requirements
    and their consistency with source documents, while quality criteria of requirements
    sets are largely neglected. This paper presents a five-step approach to evaluate
    the performance of current LLMs in generating requirements sets: a systematic
    literature review to identify quality criteria of requirements sets (1), exemplary
    generation of requirements sets from the Machinery Directive 2006/42/EC using
    LLMs (2), examination (3), evaluation (4) and comparison (5) of the generated
    requirements sets. The results support engineers in selecting LLMs for creating
    requirement sets and in evaluating future LLMs.'
author:
- first_name: Iris
  full_name: Gräßler, Iris
  id: '47565'
  last_name: Gräßler
  orcid: 0000-0001-5765-971X
- first_name: Jan Niklas
  full_name: Pfeifer, Jan Niklas
  id: '62841'
  last_name: Pfeifer
citation:
  ama: 'Gräßler I, Pfeifer JN. Performance evaluation of large language models in
    extracting requirements sets from technical documents. In: Graessler I, ed. <i>1st
    International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn
    University</i>. Universitätsbibliothek; 2026. doi:<a href="https://doi.org/10.17619/UNIPB/1-2646">10.17619/UNIPB/1-2646</a>'
  apa: 'Gräßler, I., &#38; Pfeifer, J. N. (2026). Performance evaluation of large
    language models in extracting requirements sets from technical documents. In I.
    Graessler (Ed.), <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf
    Institute, Paderborn University</i>. Universitätsbibliothek. <a href="https://doi.org/10.17619/UNIPB/1-2646">https://doi.org/10.17619/UNIPB/1-2646</a>'
  bibtex: '@inproceedings{Gräßler_Pfeifer_2026, title={Performance evaluation of large
    language models in extracting requirements sets from technical documents}, DOI={<a
    href="https://doi.org/10.17619/UNIPB/1-2646">10.17619/UNIPB/1-2646</a>}, booktitle={1st
    International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn
    University}, publisher={Universitätsbibliothek}, author={Gräßler, Iris and Pfeifer,
    Jan Niklas}, editor={Graessler, Iris}, year={2026} }'
  chicago: 'Gräßler, Iris, and Jan Niklas Pfeifer. “Performance Evaluation of Large
    Language Models in Extracting Requirements Sets from Technical Documents.” In
    <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute,
    Paderborn University</i>, edited by Iris Graessler. Universitätsbibliothek, 2026.
    <a href="https://doi.org/10.17619/UNIPB/1-2646">https://doi.org/10.17619/UNIPB/1-2646</a>.'
  ieee: 'I. Gräßler and J. N. Pfeifer, “Performance evaluation of large language models
    in extracting requirements sets from technical documents,” in <i>1st International
    Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>,
    Paderborn, 2026, doi: <a href="https://doi.org/10.17619/UNIPB/1-2646">10.17619/UNIPB/1-2646</a>.'
  mla: 'Gräßler, Iris, and Jan Niklas Pfeifer. “Performance Evaluation of Large Language
    Models in Extracting Requirements Sets from Technical Documents.” <i>1st International
    Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>,
    edited by Iris Graessler, Universitätsbibliothek, 2026, doi:<a href="https://doi.org/10.17619/UNIPB/1-2646">10.17619/UNIPB/1-2646</a>.'
  short: 'I. Gräßler, J.N. Pfeifer, in: I. Graessler (Ed.), 1st International Symposium:
    March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University, Universitätsbibliothek,
    2026.'
conference:
  end_date: 26.03.2026
  location: Paderborn
  name: International Symposium on Hybrid Intelligence in Product and Production Engineering
    1. 2026 Paderborn
  start_date: 24.03.2026
date_created: 2026-08-24T14:31:30Z
date_updated: 2026-08-24T14:41:41Z
doi: 10.17619/UNIPB/1-2646
editor:
- first_name: Iris
  full_name: Graessler, Iris
  last_name: Graessler
has_accepted_license: '1'
keyword:
- Requirements Engineering
- Artificial Intelligence
- Large Language Models
- Generative AI
language:
- iso: eng
publication: '1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute,
  Paderborn University'
publisher: Universitätsbibliothek
quality_controlled: '1'
status: public
title: Performance evaluation of large language models in extracting requirements
  sets from technical documents
type: conference
user_id: '62841'
year: '2026'
...