---
_id: '56579'
abstract:
- lang: eng
text: Question answering engines have become one of the most popular type of applications
driven by Semantic Web technologies. Consequently, the provision of means to quantify
the performance of current question answering approaches on current datasets has
become ever more important. However, a large percentage of the queries found in
popular question answering benchmarks cannot be executed on current versions of
their reference dataset. There is a consequently a clear need to curate question
answering benchmarks periodically. However, the manual alteration of question
answering benchmarks is often error-prone. We alleviate this problem by presenting
QUANT, a novel framework for the creation and curation of question answering benchmarks.
QUANT sup-ports the curation of benchmarks by generating smart edit suggestions
for question-query pair and for the corresponding metadata. In addition, our framework
supports the creation of new benchmark entries by pro-viding predefined quality
checks for queries. We evaluate QUANT on 653questions obtained from QALD-1 to
QALD-8 with 10 users. Our results show that our framework generates reliable suggestions
and can reduce the curation effort for QA benchmarks by up to 91%.
author:
- first_name: Ria Hari
full_name: Gusmita, Ria Hari
id: '71039'
last_name: Gusmita
- first_name: Rricha
full_name: Jalota, Rricha
last_name: Jalota
- first_name: Daniel
full_name: Vollmers, Daniel
last_name: Vollmers
- first_name: Jan
full_name: Reineke, Jan
last_name: Reineke
- first_name: Axel-Cyrille
full_name: Ngonga Ngomo, Axel-Cyrille
id: '65716'
last_name: Ngonga Ngomo
- first_name: Ricardo
full_name: Usbeck, Ricardo
last_name: Usbeck
citation:
ama: 'Gusmita RH, Jalota R, Vollmers D, Reineke J, Ngonga Ngomo A-C, Usbeck R. QUANT
- Question Answering Benchmark Curator. In: Acosta M, Cudr{\’e}-Mauroux P, Maleshkova
M, Pellegrini T, Sack H, Sure-Vetter Y, eds. Semantic Systems. The Power of
AI and Knowledge Graphs. Springer International Publishing; 2019:343--358.
doi:10.1007/978-3-030-33220-4_25'
apa: Gusmita, R. H., Jalota, R., Vollmers, D., Reineke, J., Ngonga Ngomo, A.-C.,
& Usbeck, R. (2019). QUANT - Question Answering Benchmark Curator. In M. Acosta,
P. Cudr{\’e}-Mauroux, M. Maleshkova, T. Pellegrini, H. Sack, & Y. Sure-Vetter
(Eds.), Semantic Systems. The Power of AI and Knowledge Graphs (pp. 343--358).
Springer International Publishing. https://doi.org/10.1007/978-3-030-33220-4_25
bibtex: '@inbook{Gusmita_Jalota_Vollmers_Reineke_Ngonga Ngomo_Usbeck_2019, place={Cham},
title={QUANT - Question Answering Benchmark Curator}, DOI={10.1007/978-3-030-33220-4_25},
booktitle={Semantic Systems. The Power of AI and Knowledge Graphs}, publisher={Springer
International Publishing}, author={Gusmita, Ria Hari and Jalota, Rricha and Vollmers,
Daniel and Reineke, Jan and Ngonga Ngomo, Axel-Cyrille and Usbeck, Ricardo}, editor={Acosta,
Maribel and Cudr{\’e}-Mauroux, Philippe and Maleshkova, Maria and Pellegrini,
Tassilo and Sack, Harald and Sure-Vetter, York}, year={2019}, pages={343--358}
}'
chicago: 'Gusmita, Ria Hari, Rricha Jalota, Daniel Vollmers, Jan Reineke, Axel-Cyrille
Ngonga Ngomo, and Ricardo Usbeck. “QUANT - Question Answering Benchmark Curator.”
In Semantic Systems. The Power of AI and Knowledge Graphs, edited by Maribel
Acosta, Philippe Cudr{\’e}-Mauroux, Maria Maleshkova, Tassilo Pellegrini, Harald
Sack, and York Sure-Vetter, 343--358. Cham: Springer International Publishing,
2019. https://doi.org/10.1007/978-3-030-33220-4_25.'
ieee: 'R. H. Gusmita, R. Jalota, D. Vollmers, J. Reineke, A.-C. Ngonga Ngomo, and
R. Usbeck, “QUANT - Question Answering Benchmark Curator,” in Semantic Systems.
The Power of AI and Knowledge Graphs, M. Acosta, P. Cudr{\’e}-Mauroux, M.
Maleshkova, T. Pellegrini, H. Sack, and Y. Sure-Vetter, Eds. Cham: Springer International
Publishing, 2019, pp. 343--358.'
mla: Gusmita, Ria Hari, et al. “QUANT - Question Answering Benchmark Curator.” Semantic
Systems. The Power of AI and Knowledge Graphs, edited by Maribel Acosta et
al., Springer International Publishing, 2019, pp. 343--358, doi:10.1007/978-3-030-33220-4_25.
short: 'R.H. Gusmita, R. Jalota, D. Vollmers, J. Reineke, A.-C. Ngonga Ngomo, R.
Usbeck, in: M. Acosta, P. Cudr{\’e}-Mauroux, M. Maleshkova, T. Pellegrini, H.
Sack, Y. Sure-Vetter (Eds.), Semantic Systems. The Power of AI and Knowledge Graphs,
Springer International Publishing, Cham, 2019, pp. 343--358.'
conference:
end_date: 2019-09-12
location: Karlsruhe, Germany
name: SEMANTiCS 2019
start_date: 2019-09-09
date_created: 2024-10-10T14:25:00Z
date_updated: 2024-11-19T15:34:24Z
ddc:
- '004'
doi: 10.1007/978-3-030-33220-4_25
editor:
- first_name: Maribel
full_name: Acosta, Maribel
last_name: Acosta
- first_name: Philippe
full_name: Cudr{\'e}-Mauroux, Philippe
last_name: Cudr{\'e}-Mauroux
- first_name: Maria
full_name: Maleshkova, Maria
last_name: Maleshkova
- first_name: Tassilo
full_name: Pellegrini, Tassilo
last_name: Pellegrini
- first_name: Harald
full_name: Sack, Harald
last_name: Sack
- first_name: York
full_name: Sure-Vetter, York
last_name: Sure-Vetter
file:
- access_level: closed
content_type: application/pdf
creator: gusmita
date_created: 2024-11-19T15:28:45Z
date_updated: 2024-11-19T15:28:45Z
file_id: '57244'
file_name: Gusmita_et_al-2019-Semantic_Systems._The_Power_of_AI_and_Knowledge_Graphs,_15th_International....pdf
file_size: 1179910
relation: main_file
success: 1
file_date_updated: 2024-11-19T15:28:45Z
has_accepted_license: '1'
keyword:
- Benchmark
- Question answering
- Knowledge base
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://rdcu.be/d0KJc
oa: '1'
page: 343--358
place: Cham
publication: Semantic Systems. The Power of AI and Knowledge Graphs
publication_identifier:
eisbn:
- 978-3-030-33220-4
isbn:
- 978-3-030-33219-8
issn:
- 0302-9743
- 1611-3349
publication_status: published
publisher: Springer International Publishing
status: public
title: QUANT - Question Answering Benchmark Curator
type: book_chapter
user_id: '71039'
year: '2019'
...