---
_id: '47522'
abstract:
- lang: eng
text: Artificial benchmark functions are commonly used in optimization research
because of their ability to rapidly evaluate potential solutions, making them
a preferred substitute for real-world problems. However, these benchmark functions
have faced criticism for their limited resemblance to real-world problems. In
response, recent research has focused on automatically generating new benchmark
functions for areas where established test suites are inadequate. These approaches
have limitations, such as the difficulty of generating new benchmark functions
that exhibit exploratory landscape analysis (ELA) features beyond those of existing
benchmarks.The objective of this work is to develop a method for generating benchmark
functions for single-objective continuous optimization with user-specified structural
properties. Specifically, we aim to demonstrate a proof of concept for a method
that uses an ELA feature vector to specify these properties in advance. To achieve
this, we begin by generating a random sample of decision space variables and objective
values. We then adjust the objective values using CMA-ES until the corresponding
features of our new problem match the predefined ELA features within a specified
threshold. By iteratively transforming the landscape in this way, we ensure that
the resulting function exhibits the desired properties. To create the final function,
we use the resulting point cloud as training data for a simple neural network
that produces a function exhibiting the target ELA features. We demonstrate the
effectiveness of this approach by replicating the existing functions of the well-known
BBOB suite and creating new functions with ELA feature values that are not present
in BBOB.
author:
- first_name: Raphael Patrick
full_name: Prager, Raphael Patrick
last_name: Prager
- first_name: Konstantin
full_name: Dietrich, Konstantin
last_name: Dietrich
- first_name: Lennart
full_name: Schneider, Lennart
last_name: Schneider
- first_name: Lennart
full_name: Schäpermeier, Lennart
last_name: Schäpermeier
- first_name: Bernd
full_name: Bischl, Bernd
last_name: Bischl
- first_name: Pascal
full_name: Kerschke, Pascal
last_name: Kerschke
- first_name: Heike
full_name: Trautmann, Heike
id: '100740'
last_name: Trautmann
orcid: 0000-0002-9788-8282
- first_name: Olaf
full_name: Mersmann, Olaf
last_name: Mersmann
citation:
ama: 'Prager RP, Dietrich K, Schneider L, et al. Neural Networks as Black-Box Benchmark
Functions Optimized for Exploratory Landscape Features. In: Proceedings of
the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms. FOGA
’23. Association for Computing Machinery; 2023:129–139. doi:10.1145/3594805.3607136'
apa: Prager, R. P., Dietrich, K., Schneider, L., Schäpermeier, L., Bischl, B., Kerschke,
P., Trautmann, H., & Mersmann, O. (2023). Neural Networks as Black-Box Benchmark
Functions Optimized for Exploratory Landscape Features. Proceedings of the
17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms, 129–139.
https://doi.org/10.1145/3594805.3607136
bibtex: '@inproceedings{Prager_Dietrich_Schneider_Schäpermeier_Bischl_Kerschke_Trautmann_Mersmann_2023,
place={New York, NY, USA}, series={FOGA ’23}, title={Neural Networks as Black-Box
Benchmark Functions Optimized for Exploratory Landscape Features}, DOI={10.1145/3594805.3607136},
booktitle={Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic
Algorithms}, publisher={Association for Computing Machinery}, author={Prager,
Raphael Patrick and Dietrich, Konstantin and Schneider, Lennart and Schäpermeier,
Lennart and Bischl, Bernd and Kerschke, Pascal and Trautmann, Heike and Mersmann,
Olaf}, year={2023}, pages={129–139}, collection={FOGA ’23} }'
chicago: 'Prager, Raphael Patrick, Konstantin Dietrich, Lennart Schneider, Lennart
Schäpermeier, Bernd Bischl, Pascal Kerschke, Heike Trautmann, and Olaf Mersmann.
“Neural Networks as Black-Box Benchmark Functions Optimized for Exploratory Landscape
Features.” In Proceedings of the 17th ACM/SIGEVO Conference on Foundations
of Genetic Algorithms, 129–139. FOGA ’23. New York, NY, USA: Association for
Computing Machinery, 2023. https://doi.org/10.1145/3594805.3607136.'
ieee: 'R. P. Prager et al., “Neural Networks as Black-Box Benchmark Functions
Optimized for Exploratory Landscape Features,” in Proceedings of the 17th ACM/SIGEVO
Conference on Foundations of Genetic Algorithms, 2023, pp. 129–139, doi: 10.1145/3594805.3607136.'
mla: Prager, Raphael Patrick, et al. “Neural Networks as Black-Box Benchmark Functions
Optimized for Exploratory Landscape Features.” Proceedings of the 17th ACM/SIGEVO
Conference on Foundations of Genetic Algorithms, Association for Computing
Machinery, 2023, pp. 129–139, doi:10.1145/3594805.3607136.
short: 'R.P. Prager, K. Dietrich, L. Schneider, L. Schäpermeier, B. Bischl, P. Kerschke,
H. Trautmann, O. Mersmann, in: Proceedings of the 17th ACM/SIGEVO Conference on
Foundations of Genetic Algorithms, Association for Computing Machinery, New York,
NY, USA, 2023, pp. 129–139.'
date_created: 2023-09-27T15:43:17Z
date_updated: 2023-10-16T12:33:02Z
department:
- _id: '34'
- _id: '819'
doi: 10.1145/3594805.3607136
keyword:
- Benchmarking
- Instance Generator
- Black-Box Continuous Optimization
- Exploratory Landscape Analysis
- Neural Networks
language:
- iso: eng
page: 129–139
place: New York, NY, USA
publication: Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic
Algorithms
publication_identifier:
isbn:
- '9798400702020'
publisher: Association for Computing Machinery
series_title: FOGA ’23
status: public
title: Neural Networks as Black-Box Benchmark Functions Optimized for Exploratory
Landscape Features
type: conference
user_id: '15504'
year: '2023'
...
---
_id: '48882'
abstract:
- lang: eng
text: In multimodal multi-objective optimization (MMMOO), the focus is not solely
on convergence in objective space, but rather also on explicitly ensuring diversity
in decision space. We illustrate why commonly used diversity measures are not
entirely appropriate for this task and propose a sophisticated basin-based evaluation
(BBE) method. Also, BBE variants are developed, capturing the anytime behavior
of algorithms. The set of BBE measures is tested by means of an algorithm configuration
study. We show that these new measures also transfer properties of the well-established
hypervolume (HV) indicator to the domain of MMMOO, thus also accounting for objective
space convergence. Moreover, we advance MMMOO research by providing insights into
the multimodal performance of the considered algorithms. Specifically, algorithms
exploiting local structures are shown to outperform classical evolutionary multi-objective
optimizers regarding the BBE variants and respective trade-off with HV.
author:
- first_name: Jonathan
full_name: Heins, Jonathan
last_name: Heins
- first_name: Jeroen
full_name: Rook, Jeroen
last_name: Rook
- first_name: Lennart
full_name: Schäpermeier, Lennart
last_name: Schäpermeier
- first_name: Pascal
full_name: Kerschke, Pascal
last_name: Kerschke
- first_name: Jakob
full_name: Bossek, Jakob
id: '102979'
last_name: Bossek
orcid: 0000-0002-4121-4668
- first_name: Heike
full_name: Trautmann, Heike
last_name: Trautmann
citation:
ama: 'Heins J, Rook J, Schäpermeier L, Kerschke P, Bossek J, Trautmann H. BBE: Basin-Based
Evaluation of Multimodal Multi-objective Optimization Problems. In: Rudolph G,
Kononova AV, Aguirre H, Kerschke P, Ochoa G, Tusar T, eds. Parallel Problem
Solving from Nature (PPSN XVII). Lecture Notes in Computer Science. Springer
International Publishing; 2022:192–206. doi:10.1007/978-3-031-14714-2_14'
apa: 'Heins, J., Rook, J., Schäpermeier, L., Kerschke, P., Bossek, J., & Trautmann,
H. (2022). BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization
Problems. In G. Rudolph, A. V. Kononova, H. Aguirre, P. Kerschke, G. Ochoa, &
T. Tusar (Eds.), Parallel Problem Solving from Nature (PPSN XVII) (pp.
192–206). Springer International Publishing. https://doi.org/10.1007/978-3-031-14714-2_14'
bibtex: '@inproceedings{Heins_Rook_Schäpermeier_Kerschke_Bossek_Trautmann_2022,
place={Cham}, series={Lecture Notes in Computer Science}, title={BBE: Basin-Based
Evaluation of Multimodal Multi-objective Optimization Problems}, DOI={10.1007/978-3-031-14714-2_14},
booktitle={Parallel Problem Solving from Nature (PPSN XVII)}, publisher={Springer
International Publishing}, author={Heins, Jonathan and Rook, Jeroen and Schäpermeier,
Lennart and Kerschke, Pascal and Bossek, Jakob and Trautmann, Heike}, editor={Rudolph,
Günter and Kononova, Anna V. and Aguirre, Hernán and Kerschke, Pascal and Ochoa,
Gabriela and Tusar, Tea}, year={2022}, pages={192–206}, collection={Lecture Notes
in Computer Science} }'
chicago: 'Heins, Jonathan, Jeroen Rook, Lennart Schäpermeier, Pascal Kerschke, Jakob
Bossek, and Heike Trautmann. “BBE: Basin-Based Evaluation of Multimodal Multi-Objective
Optimization Problems.” In Parallel Problem Solving from Nature (PPSN XVII),
edited by Günter Rudolph, Anna V. Kononova, Hernán Aguirre, Pascal Kerschke, Gabriela
Ochoa, and Tea Tusar, 192–206. Lecture Notes in Computer Science. Cham: Springer
International Publishing, 2022. https://doi.org/10.1007/978-3-031-14714-2_14.'
ieee: 'J. Heins, J. Rook, L. Schäpermeier, P. Kerschke, J. Bossek, and H. Trautmann,
“BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems,”
in Parallel Problem Solving from Nature (PPSN XVII), 2022, pp. 192–206,
doi: 10.1007/978-3-031-14714-2_14.'
mla: 'Heins, Jonathan, et al. “BBE: Basin-Based Evaluation of Multimodal Multi-Objective
Optimization Problems.” Parallel Problem Solving from Nature (PPSN XVII),
edited by Günter Rudolph et al., Springer International Publishing, 2022, pp.
192–206, doi:10.1007/978-3-031-14714-2_14.'
short: 'J. Heins, J. Rook, L. Schäpermeier, P. Kerschke, J. Bossek, H. Trautmann,
in: G. Rudolph, A.V. Kononova, H. Aguirre, P. Kerschke, G. Ochoa, T. Tusar (Eds.),
Parallel Problem Solving from Nature (PPSN XVII), Springer International Publishing,
Cham, 2022, pp. 192–206.'
date_created: 2023-11-14T15:58:58Z
date_updated: 2023-12-13T10:47:50Z
department:
- _id: '819'
doi: 10.1007/978-3-031-14714-2_14
editor:
- first_name: Günter
full_name: Rudolph, Günter
last_name: Rudolph
- first_name: Anna V.
full_name: Kononova, Anna V.
last_name: Kononova
- first_name: Hernán
full_name: Aguirre, Hernán
last_name: Aguirre
- first_name: Pascal
full_name: Kerschke, Pascal
last_name: Kerschke
- first_name: Gabriela
full_name: Ochoa, Gabriela
last_name: Ochoa
- first_name: Tea
full_name: Tusar, Tea
last_name: Tusar
extern: '1'
keyword:
- Anytime behavior
- Benchmarking
- Continuous optimization
- Multi-objective optimization
- Multimodality
- Performance metric
language:
- iso: eng
page: 192–206
place: Cham
publication: Parallel Problem Solving from Nature (PPSN XVII)
publication_identifier:
isbn:
- 978-3-031-14714-2
publisher: Springer International Publishing
series_title: Lecture Notes in Computer Science
status: public
title: 'BBE: Basin-Based Evaluation of Multimodal Multi-objective Optimization Problems'
type: conference
user_id: '102979'
year: '2022'
...
---
_id: '46318'
abstract:
- lang: eng
text: 'Multi-objective (MO) optimization, i.e., the simultaneous optimization of
multiple conflicting objectives, is gaining more and more attention in various
research areas, such as evolutionary computation, machine learning (e.g., (hyper-)parameter
optimization), or logistics (e.g., vehicle routing). Many works in this domain
mention the structural problem property of multimodality as a challenge from two
classical perspectives: (1) finding all globally optimal solution sets, and (2)
avoiding to get trapped in local optima. Interestingly, these streams seem to
transfer many traditional concepts of single-objective (SO) optimization into
claims, assumptions, or even terminology regarding the MO domain, but mostly neglect
the understanding of the structural properties as well as the algorithmic search
behavior on a problem’s landscape. However, some recent works counteract this
trend, by investigating the fundamentals and characteristics of MO problems using
new visualization techniques and gaining surprising insights. Using these visual
insights, this work proposes a step towards a unified terminology to capture multimodality
and locality in a broader way than it is usually done. This enables us to investigate
current research activities in multimodal continuous MO optimization and to highlight
new implications and promising research directions for the design of benchmark
suites, the discovery of MO landscape features, the development of new MO (or
even SO) optimization algorithms, and performance indicators. For all these topics,
we provide a review of ideas and methods but also an outlook on future challenges,
research potential and perspectives that result from recent developments.'
author:
- first_name: Christian
full_name: Grimme, Christian
last_name: Grimme
- first_name: Pascal
full_name: Kerschke, Pascal
last_name: Kerschke
- first_name: Pelin
full_name: Aspar, Pelin
last_name: Aspar
- first_name: Heike
full_name: Trautmann, Heike
id: '100740'
last_name: Trautmann
orcid: 0000-0002-9788-8282
- first_name: Mike
full_name: Preuss, Mike
last_name: Preuss
- first_name: André H.
full_name: Deutz, André H.
last_name: Deutz
- first_name: Hao
full_name: Wang, Hao
last_name: Wang
- first_name: Michael
full_name: Emmerich, Michael
last_name: Emmerich
citation:
ama: 'Grimme C, Kerschke P, Aspar P, et al. Peeking beyond peaks: Challenges and
research potentials of continuous multimodal multi-objective optimization. Computers
& Operations Research. 2021;136:105489. doi:https://doi.org/10.1016/j.cor.2021.105489'
apa: 'Grimme, C., Kerschke, P., Aspar, P., Trautmann, H., Preuss, M., Deutz, A.
H., Wang, H., & Emmerich, M. (2021). Peeking beyond peaks: Challenges and
research potentials of continuous multimodal multi-objective optimization. Computers
& Operations Research, 136, 105489. https://doi.org/10.1016/j.cor.2021.105489'
bibtex: '@article{Grimme_Kerschke_Aspar_Trautmann_Preuss_Deutz_Wang_Emmerich_2021,
title={Peeking beyond peaks: Challenges and research potentials of continuous
multimodal multi-objective optimization}, volume={136}, DOI={https://doi.org/10.1016/j.cor.2021.105489},
journal={Computers & Operations Research}, author={Grimme, Christian and Kerschke,
Pascal and Aspar, Pelin and Trautmann, Heike and Preuss, Mike and Deutz, André
H. and Wang, Hao and Emmerich, Michael}, year={2021}, pages={105489} }'
chicago: 'Grimme, Christian, Pascal Kerschke, Pelin Aspar, Heike Trautmann, Mike
Preuss, André H. Deutz, Hao Wang, and Michael Emmerich. “Peeking beyond Peaks:
Challenges and Research Potentials of Continuous Multimodal Multi-Objective Optimization.”
Computers & Operations Research 136 (2021): 105489. https://doi.org/10.1016/j.cor.2021.105489.'
ieee: 'C. Grimme et al., “Peeking beyond peaks: Challenges and research potentials
of continuous multimodal multi-objective optimization,” Computers & Operations
Research, vol. 136, p. 105489, 2021, doi: https://doi.org/10.1016/j.cor.2021.105489.'
mla: 'Grimme, Christian, et al. “Peeking beyond Peaks: Challenges and Research Potentials
of Continuous Multimodal Multi-Objective Optimization.” Computers & Operations
Research, vol. 136, 2021, p. 105489, doi:https://doi.org/10.1016/j.cor.2021.105489.'
short: C. Grimme, P. Kerschke, P. Aspar, H. Trautmann, M. Preuss, A.H. Deutz, H.
Wang, M. Emmerich, Computers & Operations Research 136 (2021) 105489.
date_created: 2023-08-04T07:28:34Z
date_updated: 2023-10-16T12:58:42Z
department:
- _id: '34'
- _id: '819'
doi: https://doi.org/10.1016/j.cor.2021.105489
intvolume: ' 136'
keyword:
- Multimodal optimization
- Multi-objective continuous optimization
- Landscape analysis
- Visualization
- Benchmarking
- Theory
- Algorithms
language:
- iso: eng
page: '105489'
publication: Computers & Operations Research
publication_identifier:
issn:
- 0305-0548
status: public
title: 'Peeking beyond peaks: Challenges and research potentials of continuous multimodal
multi-objective optimization'
type: journal_article
user_id: '15504'
volume: 136
year: '2021'
...
---
_id: '25212'
abstract:
- lang: eng
text: "Finding a good query plan is key to the optimization of query runtime. This
holds in particular for cost-based federation\r\nengines, which make use of cardinality
estimations to achieve this goal. A number of studies compare SPARQL federation
engines across different performance metrics, including query runtime, result
set completeness and correctness, number of sources selected and number of requests
sent. Albeit informative, these metrics are generic and unable to quantify and
evaluate the accuracy of the cardinality estimators of cost-based federation engines.
To thoroughly evaluate cost-based federation engines, the effect of estimated
cardinality errors on the overall query runtime performance must be measured.
In this paper, we address this challenge by presenting novel evaluation metrics
targeted at a fine-grained benchmarking of cost-based federated SPARQL query engines.
We evaluate five cost-based federated SPARQL query engines using existing as well
as novel evaluation metrics by using LargeRDFBench queries. Our results provide
a detailed analysis of the experimental outcomes that reveal novel insights, useful
for the development of future cost-based federated SPARQL query processing engines."
article_type: original
author:
- first_name: Umair
full_name: Qudus, Umair
id: '83392'
last_name: Qudus
orcid: 0000-0001-6714-8729
- first_name: Muhammad
full_name: Saleem, Muhammad
last_name: Saleem
- first_name: Axel-Cyrille
full_name: Ngonga Ngomo, Axel-Cyrille
id: '65716'
last_name: Ngonga Ngomo
- first_name: Young-Koo
full_name: Lee, Young-Koo
last_name: Lee
citation:
ama: Qudus U, Saleem M, Ngonga Ngomo A-C, Lee Y-K. An Empirical Evaluation of Cost-based
Federated SPARQL Query Processing Engines. Semantic Web. 12(6):843-868.
doi:10.3233/SW-200420
apa: Qudus, U., Saleem, M., Ngonga Ngomo, A.-C., & Lee, Y.-K. (n.d.). An Empirical
Evaluation of Cost-based Federated SPARQL Query Processing Engines. Semantic
Web, 12(6), 843–868. https://doi.org/10.3233/SW-200420
bibtex: '@article{Qudus_Saleem_Ngonga Ngomo_Lee, title={An Empirical Evaluation
of Cost-based Federated SPARQL Query Processing Engines}, volume={12}, DOI={10.3233/SW-200420}, number={6}, journal={Semantic
Web}, publisher={ISO Press}, author={Qudus, Umair and Saleem, Muhammad and Ngonga
Ngomo, Axel-Cyrille and Lee, Young-Koo}, pages={843–868} }'
chicago: 'Qudus, Umair, Muhammad Saleem, Axel-Cyrille Ngonga Ngomo, and Young-Koo
Lee. “An Empirical Evaluation of Cost-Based Federated SPARQL Query Processing
Engines.” Semantic Web 12, no. 6 (n.d.): 843–68. https://doi.org/10.3233/SW-200420.'
ieee: 'U. Qudus, M. Saleem, A.-C. Ngonga Ngomo, and Y.-K. Lee, “An Empirical Evaluation
of Cost-based Federated SPARQL Query Processing Engines,” Semantic Web,
vol. 12, no. 6, pp. 843–868, doi: 10.3233/SW-200420.'
mla: Qudus, Umair, et al. “An Empirical Evaluation of Cost-Based Federated SPARQL
Query Processing Engines.” Semantic Web, vol. 12, no. 6, ISO Press, pp.
843–68, doi:10.3233/SW-200420.
short: U. Qudus, M. Saleem, A.-C. Ngonga Ngomo, Y.-K. Lee, Semantic Web 12 (n.d.)
843–868.
date_created: 2021-10-01T06:52:52Z
date_updated: 2025-09-11T09:50:14Z
ddc:
- '000'
department:
- _id: '574'
doi: 10.3233/SW-200420
file:
- access_level: closed
content_type: application/pdf
creator: uqudus
date_created: 2024-01-13T11:35:53Z
date_updated: 2024-01-13T11:35:53Z
file_id: '50483'
file_name: swj2604.pdf
file_size: 978478
relation: main_file
success: 1
file_date_updated: 2024-01-13T11:35:53Z
has_accepted_license: '1'
intvolume: ' 12'
issue: '6'
keyword:
- SPARQL
- benchmarking
- cost-based
- cost-free
- federated
- querying
language:
- iso: eng
page: 843-868
project:
- _id: '410'
name: 'KnowGraphs: KnowGraphs: Knowledge Graphs at Scale'
publication: Semantic Web
publication_identifier:
issn:
- 2210-4968
publication_status: accepted
publisher: ISO Press
status: public
title: An Empirical Evaluation of Cost-based Federated SPARQL Query Processing Engines
type: journal_article
user_id: '83392'
volume: 12
year: '2021'
...
---
_id: '21632'
abstract:
- lang: eng
text: FPGAs have found increasing adoption in data center applications since a new
generation of high-level tools have become available which noticeably reduce development
time for FPGA accelerators and still provide high-quality results. There is, however,
no high-level benchmark suite available, which specifically enables a comparison
of FPGA architectures, programming tools, and libraries for HPC applications.
To fill this gap, we have developed an OpenCL-based open-source implementation
of the HPCC benchmark suite for Xilinx and Intel FPGAs. This benchmark can serve
to analyze the current capabilities of FPGA devices, cards, and development tool
flows, track progress over time, and point out specific difficulties for FPGA
acceleration in the HPC domain. Additionally, the benchmark documents proven performance
optimization patterns. We will continue optimizing and porting the benchmark for
new generations of FPGAs and design tools and encourage active participation to
create a valuable tool for the community. To fill this gap, we have developed
an OpenCL-based open-source implementation of the HPCC benchmark suite for Xilinx
and Intel FPGAs. This benchmark can serve to analyze the current capabilities
of FPGA devices, cards, and development tool flows, track progress over time,
and point out specific difficulties for FPGA acceleration in the HPC domain. Additionally,
the benchmark documents proven performance optimization patterns. We will continue
optimizing and porting the benchmark for new generations of FPGAs and design tools
and encourage active participation to create a valuable tool for the community.
author:
- first_name: Marius
full_name: Meyer, Marius
id: '40778'
last_name: Meyer
- first_name: Tobias
full_name: Kenter, Tobias
id: '3145'
last_name: Kenter
- first_name: Christian
full_name: Plessl, Christian
id: '16153'
last_name: Plessl
orcid: 0000-0001-5728-9982
citation:
ama: 'Meyer M, Kenter T, Plessl C. Evaluating FPGA Accelerator Performance with
a Parameterized OpenCL Adaptation of Selected Benchmarks of the HPCChallenge Benchmark
Suite. In: 2020 IEEE/ACM International Workshop on Heterogeneous High-Performance
Reconfigurable Computing (H2RC). ; 2020. doi:10.1109/h2rc51942.2020.00007'
apa: Meyer, M., Kenter, T., & Plessl, C. (2020). Evaluating FPGA Accelerator
Performance with a Parameterized OpenCL Adaptation of Selected Benchmarks of the
HPCChallenge Benchmark Suite. 2020 IEEE/ACM International Workshop on Heterogeneous
High-Performance Reconfigurable Computing (H2RC). https://doi.org/10.1109/h2rc51942.2020.00007
bibtex: '@inproceedings{Meyer_Kenter_Plessl_2020, title={Evaluating FPGA Accelerator
Performance with a Parameterized OpenCL Adaptation of Selected Benchmarks of the
HPCChallenge Benchmark Suite}, DOI={10.1109/h2rc51942.2020.00007},
booktitle={2020 IEEE/ACM International Workshop on Heterogeneous High-performance
Reconfigurable Computing (H2RC)}, author={Meyer, Marius and Kenter, Tobias and
Plessl, Christian}, year={2020} }'
chicago: Meyer, Marius, Tobias Kenter, and Christian Plessl. “Evaluating FPGA Accelerator
Performance with a Parameterized OpenCL Adaptation of Selected Benchmarks of the
HPCChallenge Benchmark Suite.” In 2020 IEEE/ACM International Workshop on Heterogeneous
High-Performance Reconfigurable Computing (H2RC), 2020. https://doi.org/10.1109/h2rc51942.2020.00007.
ieee: 'M. Meyer, T. Kenter, and C. Plessl, “Evaluating FPGA Accelerator Performance
with a Parameterized OpenCL Adaptation of Selected Benchmarks of the HPCChallenge
Benchmark Suite,” 2020, doi: 10.1109/h2rc51942.2020.00007.'
mla: Meyer, Marius, et al. “Evaluating FPGA Accelerator Performance with a Parameterized
OpenCL Adaptation of Selected Benchmarks of the HPCChallenge Benchmark Suite.”
2020 IEEE/ACM International Workshop on Heterogeneous High-Performance Reconfigurable
Computing (H2RC), 2020, doi:10.1109/h2rc51942.2020.00007.
short: 'M. Meyer, T. Kenter, C. Plessl, in: 2020 IEEE/ACM International Workshop
on Heterogeneous High-Performance Reconfigurable Computing (H2RC), 2020.'
date_created: 2021-04-16T10:17:22Z
date_updated: 2023-09-26T11:42:53Z
department:
- _id: '27'
- _id: '518'
doi: 10.1109/h2rc51942.2020.00007
keyword:
- FPGA
- OpenCL
- High Level Synthesis
- HPC benchmarking
language:
- iso: eng
main_file_link:
- url: https://ieeexplore.ieee.org/document/9306963
project:
- _id: '52'
name: Computing Resources Provided by the Paderborn Center for Parallel Computing
publication: 2020 IEEE/ACM International Workshop on Heterogeneous High-performance
Reconfigurable Computing (H2RC)
publication_identifier:
isbn:
- '9781665415927'
publication_status: published
quality_controlled: '1'
related_material:
link:
- description: Official repository of the benchmark suite on GitHub
relation: supplementary_material
url: https://github.com/pc2/HPCC_FPGA
status: public
title: Evaluating FPGA Accelerator Performance with a Parameterized OpenCL Adaptation
of Selected Benchmarks of the HPCChallenge Benchmark Suite
type: conference
user_id: '15278'
year: '2020'
...
---
_id: '48842'
abstract:
- lang: eng
text: 'Evolutionary algorithms have successfully been applied to evolve problem
instances that exhibit a significant difference in performance for a given algorithm
or a pair of algorithms inter alia for the Traveling Salesperson Problem (TSP).
Creating a large variety of instances is crucial for successful applications in
the blooming field of algorithm selection. In this paper, we introduce new and
creative mutation operators for evolving instances of the TSP. We show that adopting
those operators in an evolutionary algorithm allows for the generation of benchmark
sets with highly desirable properties: (1) novelty by clear visual distinction
to established benchmark sets in the field, (2) visual and quantitative diversity
in the space of TSP problem characteristics, and (3) significant performance differences
with respect to the restart versions of heuristic state-of-the-art TSP solvers
EAX and LKH. The important aspect of diversity is addressed and achieved solely
by the proposed mutation operators and not enforced by explicit diversity preservation.'
author:
- first_name: Jakob
full_name: Bossek, Jakob
id: '102979'
last_name: Bossek
orcid: 0000-0002-4121-4668
- first_name: Pascal
full_name: Kerschke, Pascal
last_name: Kerschke
- first_name: Aneta
full_name: Neumann, Aneta
last_name: Neumann
- first_name: Markus
full_name: Wagner, Markus
last_name: Wagner
- first_name: Frank
full_name: Neumann, Frank
last_name: Neumann
- first_name: Heike
full_name: Trautmann, Heike
last_name: Trautmann
citation:
ama: 'Bossek J, Kerschke P, Neumann A, Wagner M, Neumann F, Trautmann H. Evolving
Diverse TSP Instances by Means of Novel and Creative Mutation Operators. In: Proceedings
of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms. FOGA
’19. Association for Computing Machinery; 2019:58–71. doi:10.1145/3299904.3340307'
apa: Bossek, J., Kerschke, P., Neumann, A., Wagner, M., Neumann, F., & Trautmann,
H. (2019). Evolving Diverse TSP Instances by Means of Novel and Creative Mutation
Operators. Proceedings of the 15th ACM/SIGEVO Conference on Foundations of
Genetic Algorithms, 58–71. https://doi.org/10.1145/3299904.3340307
bibtex: '@inproceedings{Bossek_Kerschke_Neumann_Wagner_Neumann_Trautmann_2019, place={New
York, NY, USA}, series={FOGA ’19}, title={Evolving Diverse TSP Instances by Means
of Novel and Creative Mutation Operators}, DOI={10.1145/3299904.3340307},
booktitle={Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic
Algorithms}, publisher={Association for Computing Machinery}, author={Bossek,
Jakob and Kerschke, Pascal and Neumann, Aneta and Wagner, Markus and Neumann,
Frank and Trautmann, Heike}, year={2019}, pages={58–71}, collection={FOGA ’19}
}'
chicago: 'Bossek, Jakob, Pascal Kerschke, Aneta Neumann, Markus Wagner, Frank Neumann,
and Heike Trautmann. “Evolving Diverse TSP Instances by Means of Novel and Creative
Mutation Operators.” In Proceedings of the 15th ACM/SIGEVO Conference on Foundations
of Genetic Algorithms, 58–71. FOGA ’19. New York, NY, USA: Association for
Computing Machinery, 2019. https://doi.org/10.1145/3299904.3340307.'
ieee: 'J. Bossek, P. Kerschke, A. Neumann, M. Wagner, F. Neumann, and H. Trautmann,
“Evolving Diverse TSP Instances by Means of Novel and Creative Mutation Operators,”
in Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic
Algorithms, 2019, pp. 58–71, doi: 10.1145/3299904.3340307.'
mla: Bossek, Jakob, et al. “Evolving Diverse TSP Instances by Means of Novel and
Creative Mutation Operators.” Proceedings of the 15th ACM/SIGEVO Conference
on Foundations of Genetic Algorithms, Association for Computing Machinery,
2019, pp. 58–71, doi:10.1145/3299904.3340307.
short: 'J. Bossek, P. Kerschke, A. Neumann, M. Wagner, F. Neumann, H. Trautmann,
in: Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms,
Association for Computing Machinery, New York, NY, USA, 2019, pp. 58–71.'
date_created: 2023-11-14T15:58:52Z
date_updated: 2023-12-13T10:42:57Z
department:
- _id: '819'
doi: 10.1145/3299904.3340307
extern: '1'
keyword:
- benchmarking
- instance features
- optimization
- problem generation
- traveling salesperson problem
language:
- iso: eng
page: 58–71
place: New York, NY, USA
publication: Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic
Algorithms
publication_identifier:
isbn:
- 978-1-4503-6254-2
publication_status: published
publisher: Association for Computing Machinery
series_title: FOGA ’19
status: public
title: Evolving Diverse TSP Instances by Means of Novel and Creative Mutation Operators
type: conference
user_id: '102979'
year: '2019'
...
---
_id: '46396'
abstract:
- lang: eng
text: The steady supply of new optimization methods makes the algorithm selection
problem (ASP) an increasingly pressing and challenging task, specially for real-world
black-box optimization problems. The introduced approach considers the ASP as
a cost-sensitive classification task which is based on Exploratory Landscape Analysis.
Low-level features gathered by systematic sampling of the function on the feasible
set are used to predict a well-performing algorithm out of a given portfolio.
Example-specific label costs are defined by the expected runtime of each candidate
algorithm. We use one-sided support vector regression to solve this learning problem.
The approach is illustrated by means of the optimization problems and algorithms
of the BBOB’09/10 workshop.
author:
- first_name: Bernd
full_name: Bischl, Bernd
last_name: Bischl
- first_name: Olaf
full_name: Mersmann, Olaf
last_name: Mersmann
- first_name: Heike
full_name: Trautmann, Heike
id: '100740'
last_name: Trautmann
orcid: 0000-0002-9788-8282
- first_name: Mike
full_name: Preuß, Mike
last_name: Preuß
citation:
ama: 'Bischl B, Mersmann O, Trautmann H, Preuß M. Algorithm Selection Based on Exploratory
Landscape Analysis and Cost-Sensitive Learning. In: Proceedings of the 14th
Annual Conference on Genetic and Evolutionary Computation. GECCO ’12. Association
for Computing Machinery; 2012:313–320. doi:10.1145/2330163.2330209'
apa: Bischl, B., Mersmann, O., Trautmann, H., & Preuß, M. (2012). Algorithm
Selection Based on Exploratory Landscape Analysis and Cost-Sensitive Learning.
Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation,
313–320. https://doi.org/10.1145/2330163.2330209
bibtex: '@inproceedings{Bischl_Mersmann_Trautmann_Preuß_2012, place={New York, NY,
USA}, series={GECCO ’12}, title={Algorithm Selection Based on Exploratory Landscape
Analysis and Cost-Sensitive Learning}, DOI={10.1145/2330163.2330209},
booktitle={Proceedings of the 14th Annual Conference on Genetic and Evolutionary
Computation}, publisher={Association for Computing Machinery}, author={Bischl,
Bernd and Mersmann, Olaf and Trautmann, Heike and Preuß, Mike}, year={2012}, pages={313–320},
collection={GECCO ’12} }'
chicago: 'Bischl, Bernd, Olaf Mersmann, Heike Trautmann, and Mike Preuß. “Algorithm
Selection Based on Exploratory Landscape Analysis and Cost-Sensitive Learning.”
In Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation,
313–320. GECCO ’12. New York, NY, USA: Association for Computing Machinery, 2012.
https://doi.org/10.1145/2330163.2330209.'
ieee: 'B. Bischl, O. Mersmann, H. Trautmann, and M. Preuß, “Algorithm Selection
Based on Exploratory Landscape Analysis and Cost-Sensitive Learning,” in Proceedings
of the 14th Annual Conference on Genetic and Evolutionary Computation, 2012,
pp. 313–320, doi: 10.1145/2330163.2330209.'
mla: Bischl, Bernd, et al. “Algorithm Selection Based on Exploratory Landscape Analysis
and Cost-Sensitive Learning.” Proceedings of the 14th Annual Conference on
Genetic and Evolutionary Computation, Association for Computing Machinery,
2012, pp. 313–320, doi:10.1145/2330163.2330209.
short: 'B. Bischl, O. Mersmann, H. Trautmann, M. Preuß, in: Proceedings of the 14th
Annual Conference on Genetic and Evolutionary Computation, Association for Computing
Machinery, New York, NY, USA, 2012, pp. 313–320.'
date_created: 2023-08-04T15:51:56Z
date_updated: 2023-10-16T13:48:48Z
department:
- _id: '34'
- _id: '819'
doi: 10.1145/2330163.2330209
keyword:
- machine learning
- exploratory landscape analysis
- fitness landscape
- benchmarking
- evolutionary optimization
- bbob test set
- algorithm selection
language:
- iso: eng
page: 313–320
place: New York, NY, USA
publication: Proceedings of the 14th Annual Conference on Genetic and Evolutionary
Computation
publication_identifier:
isbn:
- '9781450311779'
publisher: Association for Computing Machinery
series_title: GECCO ’12
status: public
title: Algorithm Selection Based on Exploratory Landscape Analysis and Cost-Sensitive
Learning
type: conference
user_id: '15504'
year: '2012'
...
---
_id: '46401'
abstract:
- lang: eng
text: Exploratory Landscape Analysis subsumes a number of techniques employed to
obtain knowledge about the properties of an unknown optimization problem, especially
insofar as these properties are important for the performance of optimization
algorithms. Where in a first attempt, one could rely on high-level features designed
by experts, we approach the problem from a different angle here, namely by using
relatively cheap low-level computer generated features. Interestingly, very few
features are needed to separate the BBOB problem groups and also for relating
a problem to high-level, expert designed features, paving the way for automatic
algorithm selection.
author:
- first_name: Olaf
full_name: Mersmann, Olaf
last_name: Mersmann
- first_name: Bernd
full_name: Bischl, Bernd
last_name: Bischl
- first_name: Heike
full_name: Trautmann, Heike
id: '100740'
last_name: Trautmann
orcid: 0000-0002-9788-8282
- first_name: Mike
full_name: Preuss, Mike
last_name: Preuss
- first_name: Claus
full_name: Weihs, Claus
last_name: Weihs
- first_name: Günter
full_name: Rudolph, Günter
last_name: Rudolph
citation:
ama: 'Mersmann O, Bischl B, Trautmann H, Preuss M, Weihs C, Rudolph G. Exploratory
Landscape Analysis. In: Proceedings of the 13th Annual Conference on Genetic
and Evolutionary Computation. GECCO ’11. Association for Computing Machinery;
2011:829–836. doi:10.1145/2001576.2001690'
apa: Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., & Rudolph,
G. (2011). Exploratory Landscape Analysis. Proceedings of the 13th Annual Conference
on Genetic and Evolutionary Computation, 829–836. https://doi.org/10.1145/2001576.2001690
bibtex: '@inproceedings{Mersmann_Bischl_Trautmann_Preuss_Weihs_Rudolph_2011, place={New
York, NY, USA}, series={GECCO ’11}, title={Exploratory Landscape Analysis}, DOI={10.1145/2001576.2001690}, booktitle={Proceedings
of the 13th Annual Conference on Genetic and Evolutionary Computation}, publisher={Association
for Computing Machinery}, author={Mersmann, Olaf and Bischl, Bernd and Trautmann,
Heike and Preuss, Mike and Weihs, Claus and Rudolph, Günter}, year={2011}, pages={829–836},
collection={GECCO ’11} }'
chicago: 'Mersmann, Olaf, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs,
and Günter Rudolph. “Exploratory Landscape Analysis.” In Proceedings of the
13th Annual Conference on Genetic and Evolutionary Computation, 829–836. GECCO
’11. New York, NY, USA: Association for Computing Machinery, 2011. https://doi.org/10.1145/2001576.2001690.'
ieee: 'O. Mersmann, B. Bischl, H. Trautmann, M. Preuss, C. Weihs, and G. Rudolph,
“Exploratory Landscape Analysis,” in Proceedings of the 13th Annual Conference
on Genetic and Evolutionary Computation, 2011, pp. 829–836, doi: 10.1145/2001576.2001690.'
mla: Mersmann, Olaf, et al. “Exploratory Landscape Analysis.” Proceedings of
the 13th Annual Conference on Genetic and Evolutionary Computation, Association
for Computing Machinery, 2011, pp. 829–836, doi:10.1145/2001576.2001690.
short: 'O. Mersmann, B. Bischl, H. Trautmann, M. Preuss, C. Weihs, G. Rudolph, in:
Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation,
Association for Computing Machinery, New York, NY, USA, 2011, pp. 829–836.'
date_created: 2023-08-04T15:58:22Z
date_updated: 2023-10-16T13:54:34Z
department:
- _id: '34'
- _id: '819'
doi: 10.1145/2001576.2001690
keyword:
- exploratory landscape analysis
- evolutionary optimization
- fitness landscape
- benchmarking
- BBOB test set
language:
- iso: eng
page: 829–836
place: New York, NY, USA
publication: Proceedings of the 13th Annual Conference on Genetic and Evolutionary
Computation
publication_identifier:
isbn:
- '9781450305570'
publisher: Association for Computing Machinery
series_title: GECCO ’11
status: public
title: Exploratory Landscape Analysis
type: conference
user_id: '15504'
year: '2011'
...
---
_id: '46405'
abstract:
- lang: eng
text: 'We present methods to answer two basic questions that arise when benchmarking
optimization algorithms. The first one is: which algorithm is the ’best’ one?
and the second one: which algorithm should I use for my real world problem? Both
are connected and neither is easy to answer. We present methods which can be used
to analyse the raw data of a benchmark experiment and derive some insight regarding
the answers to these questions. We employ the presented methods to analyse the
BBOB’09 benchmark results and present some initial findings.'
author:
- first_name: Olaf
full_name: Mersmann, Olaf
last_name: Mersmann
- first_name: Mike
full_name: Preuss, Mike
last_name: Preuss
- first_name: Heike
full_name: Trautmann, Heike
id: '100740'
last_name: Trautmann
orcid: 0000-0002-9788-8282
citation:
ama: 'Mersmann O, Preuss M, Trautmann H. Benchmarking Evolutionary Algorithms: Towards
Exploratory Landscape Analysis. In: Proceedings of the 11th International Conference
on Parallel Problem Solving from Nature: Part I. PPSN’10. Springer-Verlag;
2010:73–82.'
apa: 'Mersmann, O., Preuss, M., & Trautmann, H. (2010). Benchmarking Evolutionary
Algorithms: Towards Exploratory Landscape Analysis. Proceedings of the 11th
International Conference on Parallel Problem Solving from Nature: Part I,
73–82.'
bibtex: '@inproceedings{Mersmann_Preuss_Trautmann_2010, place={Berlin, Heidelberg},
series={PPSN’10}, title={Benchmarking Evolutionary Algorithms: Towards Exploratory
Landscape Analysis}, booktitle={Proceedings of the 11th International Conference
on Parallel Problem Solving from Nature: Part I}, publisher={Springer-Verlag},
author={Mersmann, Olaf and Preuss, Mike and Trautmann, Heike}, year={2010}, pages={73–82},
collection={PPSN’10} }'
chicago: 'Mersmann, Olaf, Mike Preuss, and Heike Trautmann. “Benchmarking Evolutionary
Algorithms: Towards Exploratory Landscape Analysis.” In Proceedings of the
11th International Conference on Parallel Problem Solving from Nature: Part I,
73–82. PPSN’10. Berlin, Heidelberg: Springer-Verlag, 2010.'
ieee: 'O. Mersmann, M. Preuss, and H. Trautmann, “Benchmarking Evolutionary Algorithms:
Towards Exploratory Landscape Analysis,” in Proceedings of the 11th International
Conference on Parallel Problem Solving from Nature: Part I, 2010, pp. 73–82.'
mla: 'Mersmann, Olaf, et al. “Benchmarking Evolutionary Algorithms: Towards Exploratory
Landscape Analysis.” Proceedings of the 11th International Conference on Parallel
Problem Solving from Nature: Part I, Springer-Verlag, 2010, pp. 73–82.'
short: 'O. Mersmann, M. Preuss, H. Trautmann, in: Proceedings of the 11th International
Conference on Parallel Problem Solving from Nature: Part I, Springer-Verlag, Berlin,
Heidelberg, 2010, pp. 73–82.'
date_created: 2023-08-04T16:02:28Z
date_updated: 2023-10-16T13:55:43Z
department:
- _id: '34'
- _id: '819'
keyword:
- benchmarking
- multidimensional scaling
- consensus ranking
- evolutionary optimization
- BBOB test set
language:
- iso: eng
page: 73–82
place: Berlin, Heidelberg
publication: 'Proceedings of the 11th International Conference on Parallel Problem
Solving from Nature: Part I'
publication_identifier:
isbn:
- '3642158439'
publisher: Springer-Verlag
series_title: PPSN’10
status: public
title: 'Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis'
type: conference
user_id: '15504'
year: '2010'
...