{"language":[{"iso":"eng"}],"_id":"66837","publisher":"Universitätsbibliothek","doi":"10.17619/UNIPB/1-2646","user_id":"62841","editor":[{"last_name":"Graessler","first_name":"Iris","full_name":"Graessler, Iris"}],"year":"2026","title":"Performance evaluation of large language models in extracting requirements sets from technical documents","status":"public","conference":{"start_date":"24.03.2026","name":"International Symposium on Hybrid Intelligence in Product and Production Engineering 1. 2026 Paderborn","location":"Paderborn","end_date":"26.03.2026"},"author":[{"id":"47565","full_name":"Gräßler, Iris","orcid":"0000-0001-5765-971X","first_name":"Iris","last_name":"Gräßler"},{"id":"62841","full_name":"Pfeifer, Jan Niklas","first_name":"Jan Niklas","last_name":"Pfeifer"}],"date_updated":"2026-08-24T14:41:41Z","has_accepted_license":"1","date_created":"2026-08-24T14:31:30Z","type":"conference","keyword":["Requirements Engineering","Artificial Intelligence","Large Language Models","Generative AI"],"publication":"1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University","citation":{"mla":"Gräßler, Iris, and Jan Niklas Pfeifer. “Performance Evaluation of Large Language Models in Extracting Requirements Sets from Technical Documents.” <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>, edited by Iris Graessler, Universitätsbibliothek, 2026, doi:<a href=\"https://doi.org/10.17619/UNIPB/1-2646\">10.17619/UNIPB/1-2646</a>.","bibtex":"@inproceedings{Gräßler_Pfeifer_2026, title={Performance evaluation of large language models in extracting requirements sets from technical documents}, DOI={<a href=\"https://doi.org/10.17619/UNIPB/1-2646\">10.17619/UNIPB/1-2646</a>}, booktitle={1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University}, publisher={Universitätsbibliothek}, author={Gräßler, Iris and Pfeifer, Jan Niklas}, editor={Graessler, Iris}, year={2026} }","ama":"Gräßler I, Pfeifer JN. Performance evaluation of large language models in extracting requirements sets from technical documents. In: Graessler I, ed. <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>. Universitätsbibliothek; 2026. doi:<a href=\"https://doi.org/10.17619/UNIPB/1-2646\">10.17619/UNIPB/1-2646</a>","ieee":"I. Gräßler and J. N. Pfeifer, “Performance evaluation of large language models in extracting requirements sets from technical documents,” in <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>, Paderborn, 2026, doi: <a href=\"https://doi.org/10.17619/UNIPB/1-2646\">10.17619/UNIPB/1-2646</a>.","apa":"Gräßler, I., &#38; Pfeifer, J. N. (2026). Performance evaluation of large language models in extracting requirements sets from technical documents. In I. Graessler (Ed.), <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>. Universitätsbibliothek. <a href=\"https://doi.org/10.17619/UNIPB/1-2646\">https://doi.org/10.17619/UNIPB/1-2646</a>","short":"I. Gräßler, J.N. Pfeifer, in: I. Graessler (Ed.), 1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University, Universitätsbibliothek, 2026.","chicago":"Gräßler, Iris, and Jan Niklas Pfeifer. “Performance Evaluation of Large Language Models in Extracting Requirements Sets from Technical Documents.” In <i>1st International Symposium: March 24 – 26, 2026, Heinz Nixdorf Institute, Paderborn University</i>, edited by Iris Graessler. Universitätsbibliothek, 2026. <a href=\"https://doi.org/10.17619/UNIPB/1-2646\">https://doi.org/10.17619/UNIPB/1-2646</a>."},"abstract":[{"text":"Manufacturers operate under dynamic regulatory conditions, leading to frequently changing requirements that must be managed with high effort. AI-based requirements extraction using Large Language Models (LLMs) has the potential to support engineers in this task by analyzing large documents, for example. Existing evaluation approaches in AI-supported requirements engineering mainly assess individual requirements and their consistency with source documents, while quality criteria of requirements sets are largely neglected. This paper presents a five-step approach to evaluate the performance of current LLMs in generating requirements sets: a systematic literature review to identify quality criteria of requirements sets (1), exemplary generation of requirements sets from the Machinery Directive 2006/42/EC using LLMs (2), examination (3), evaluation (4) and comparison (5) of the generated requirements sets. The results support engineers in selecting LLMs for creating requirement sets and in evaluating future LLMs.","lang":"eng"}],"quality_controlled":"1"}