---
_id: '46649'
abstract:
- lang: eng
text: "Different conflicting optimization criteria arise naturally in various Deep\r\nLearning
scenarios. These can address different main tasks (i.e., in the\r\nsetting of
Multi-Task Learning), but also main and secondary tasks such as loss\r\nminimization
versus sparsity. The usual approach is a simple weighting of the\r\ncriteria,
which formally only works in the convex setting. In this paper, we\r\npresent
a Multi-Objective Optimization algorithm using a modified Weighted\r\nChebyshev
scalarization for training Deep Neural Networks (DNNs) with respect\r\nto several
tasks. By employing this scalarization technique, the algorithm can\r\nidentify
all optimal solutions of the original problem while reducing its\r\ncomplexity
to a sequence of single-objective problems. The simplified problems\r\nare then
solved using an Augmented Lagrangian method, enabling the use of\r\npopular optimization
techniques such as Adam and Stochastic Gradient Descent,\r\nwhile efficaciously
handling constraints. Our work aims to address the\r\n(economical and also ecological)
sustainability issue of DNN models, with a\r\nparticular focus on Deep Multi-Task
models, which are typically designed with a\r\nvery large number of weights to
perform equally well on multiple tasks. Through\r\nexperiments conducted on two
Machine Learning datasets, we demonstrate the\r\npossibility of adaptively sparsifying
the model during training without\r\nsignificantly impacting its performance,
if we are willing to apply\r\ntask-specific adaptations to the network weights.
Code is available at\r\nhttps://github.com/salomonhotegni/MDMTN."
author:
- first_name: Sedjro Salomon
full_name: Hotegni, Sedjro Salomon
id: '97995'
last_name: Hotegni
- first_name: Manuel Bastian
full_name: Berkemeier, Manuel Bastian
id: '51701'
last_name: Berkemeier
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
citation:
ama: 'Hotegni SS, Berkemeier MB, Peitz S. Multi-Objective Optimization for Sparse
Deep Multi-Task Learning. In: 2024 International Joint Conference on Neural
Networks (IJCNN). IEEE; 2024:9. doi:10.1109/IJCNN60899.2024.10650994'
apa: Hotegni, S. S., Berkemeier, M. B., & Peitz, S. (2024). Multi-Objective
Optimization for Sparse Deep Multi-Task Learning. 2024 International Joint
Conference on Neural Networks (IJCNN), 9. https://doi.org/10.1109/IJCNN60899.2024.10650994
bibtex: '@inproceedings{Hotegni_Berkemeier_Peitz_2024, place={Yokohama, Japan},
title={Multi-Objective Optimization for Sparse Deep Multi-Task Learning}, DOI={10.1109/IJCNN60899.2024.10650994},
booktitle={2024 International Joint Conference on Neural Networks (IJCNN)}, publisher={IEEE},
author={Hotegni, Sedjro Salomon and Berkemeier, Manuel Bastian and Peitz, Sebastian},
year={2024}, pages={9} }'
chicago: 'Hotegni, Sedjro Salomon, Manuel Bastian Berkemeier, and Sebastian Peitz.
“Multi-Objective Optimization for Sparse Deep Multi-Task Learning.” In 2024
International Joint Conference on Neural Networks (IJCNN), 9. Yokohama, Japan:
IEEE, 2024. https://doi.org/10.1109/IJCNN60899.2024.10650994.'
ieee: 'S. S. Hotegni, M. B. Berkemeier, and S. Peitz, “Multi-Objective Optimization
for Sparse Deep Multi-Task Learning,” in 2024 International Joint Conference
on Neural Networks (IJCNN), Yokohama, Japan, 2024, p. 9, doi: 10.1109/IJCNN60899.2024.10650994.'
mla: Hotegni, Sedjro Salomon, et al. “Multi-Objective Optimization for Sparse Deep
Multi-Task Learning.” 2024 International Joint Conference on Neural Networks
(IJCNN), IEEE, 2024, p. 9, doi:10.1109/IJCNN60899.2024.10650994.
short: 'S.S. Hotegni, M.B. Berkemeier, S. Peitz, in: 2024 International Joint Conference
on Neural Networks (IJCNN), IEEE, Yokohama, Japan, 2024, p. 9.'
conference:
end_date: 2024-07-05
location: Yokohama, Japan
name: 2024 International Joint Conference on Neural Networks (IJCNN)
start_date: 2024-06-30
date_created: 2023-08-24T07:44:36Z
date_updated: 2024-09-27T10:24:22Z
department:
- _id: '655'
doi: 10.1109/IJCNN60899.2024.10650994
has_accepted_license: '1'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://ieeexplore.ieee.org/document/10650994
oa: '1'
page: '9'
place: Yokohama, Japan
publication: 2024 International Joint Conference on Neural Networks (IJCNN)
publication_identifier:
eisbn:
- 979-8-3503-5931-2
eissn:
- ' 2161-4407'
publication_status: published
publisher: IEEE
status: public
title: Multi-Objective Optimization for Sparse Deep Multi-Task Learning
type: conference
user_id: '97995'
year: '2024'
...
---
_id: '33150'
abstract:
- lang: eng
text: In this article, we build on previous work to present an optimization algorithm
for nonlinearly constrained multi-objective optimization problems. The algorithm
combines a surrogate-assisted derivative-free trust-region approach with the filter
method known from single-objective optimization. Instead of the true objective
and constraint functions, so-called fully linear models are employed and we show
how to deal with the gradient inexactness in the composite step setting, adapted
from single-objective optimization as well. Under standard assumptions, we prove
convergence of a subset of iterates to a quasi-stationary point and if constraint
qualifications hold, then the limit point is also a KKT-point of the multi-objective
problem.
author:
- first_name: Manuel Bastian
full_name: Berkemeier, Manuel Bastian
id: '51701'
last_name: Berkemeier
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
citation:
ama: Berkemeier MB, Peitz S. Multi-Objective Trust-Region Filter Method for Nonlinear
Constraints using Inexact Gradients. arXiv:220812094. Published online
2022.
apa: Berkemeier, M. B., & Peitz, S. (2022). Multi-Objective Trust-Region Filter
Method for Nonlinear Constraints using Inexact Gradients. In arXiv:2208.12094.
bibtex: '@article{Berkemeier_Peitz_2022, title={Multi-Objective Trust-Region Filter
Method for Nonlinear Constraints using Inexact Gradients}, journal={arXiv:2208.12094},
author={Berkemeier, Manuel Bastian and Peitz, Sebastian}, year={2022} }'
chicago: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Multi-Objective Trust-Region
Filter Method for Nonlinear Constraints Using Inexact Gradients.” ArXiv:2208.12094,
2022.
ieee: M. B. Berkemeier and S. Peitz, “Multi-Objective Trust-Region Filter Method
for Nonlinear Constraints using Inexact Gradients,” arXiv:2208.12094. 2022.
mla: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Multi-Objective Trust-Region
Filter Method for Nonlinear Constraints Using Inexact Gradients.” ArXiv:2208.12094,
2022.
short: M.B. Berkemeier, S. Peitz, ArXiv:2208.12094 (2022).
date_created: 2022-08-26T06:08:06Z
date_updated: 2022-08-26T06:12:10Z
department:
- _id: '101'
- _id: '655'
external_id:
arxiv:
- '2208.12094'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/pdf/2208.12094
oa: '1'
publication: arXiv:2208.12094
status: public
title: Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using
Inexact Gradients
type: preprint
user_id: '47427'
year: '2022'
...
---
_id: '21337'
abstract:
- lang: eng
text: "We present a flexible trust region descend algorithm for unconstrained and\r\nconvexly
constrained multiobjective optimization problems. It is targeted at\r\nheterogeneous
and expensive problems, i.e., problems that have at least one\r\nobjective function
that is computationally expensive. The method is\r\nderivative-free in the sense
that neither need derivative information be\r\navailable for the expensive objectives
nor are gradients approximated using\r\nrepeated function evaluations as is the
case in finite-difference methods.\r\nInstead, a multiobjective trust region approach
is used that works similarly to\r\nits well-known scalar pendants. Local surrogate
models constructed from\r\nevaluation data of the true objective functions are
employed to compute\r\npossible descent directions. In contrast to existing multiobjective
trust\r\nregion algorithms, these surrogates are not polynomial but carefully\r\nconstructed
radial basis function networks. This has the important advantage\r\nthat the number
of data points scales linearly with the parameter space\r\ndimension. The local
models qualify as fully linear and the corresponding\r\ngeneral scalar framework
is adapted for problems with multiple objectives.\r\nConvergence to Pareto critical
points is proven and numerical examples\r\nillustrate our findings."
article_number: '31'
author:
- first_name: Manuel Bastian
full_name: Berkemeier, Manuel Bastian
id: '51701'
last_name: Berkemeier
- first_name: Sebastian
full_name: Peitz, Sebastian
id: '47427'
last_name: Peitz
orcid: 0000-0002-3389-793X
citation:
ama: Berkemeier MB, Peitz S. Derivative-Free Multiobjective Trust Region Descent
Method Using Radial Basis Function Surrogate Models. Mathematical and Computational
Applications. 2021;26(2). doi:10.3390/mca26020031
apa: Berkemeier, M. B., & Peitz, S. (2021). Derivative-Free Multiobjective Trust
Region Descent Method Using Radial Basis Function Surrogate Models. Mathematical
and Computational Applications, 26(2). https://doi.org/10.3390/mca26020031
bibtex: '@article{Berkemeier_Peitz_2021, title={Derivative-Free Multiobjective Trust
Region Descent Method Using Radial Basis Function Surrogate Models}, volume={26},
DOI={10.3390/mca26020031}, number={231},
journal={Mathematical and Computational Applications}, author={Berkemeier, Manuel
Bastian and Peitz, Sebastian}, year={2021} }'
chicago: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Derivative-Free Multiobjective
Trust Region Descent Method Using Radial Basis Function Surrogate Models.” Mathematical
and Computational Applications 26, no. 2 (2021). https://doi.org/10.3390/mca26020031.
ieee: M. B. Berkemeier and S. Peitz, “Derivative-Free Multiobjective Trust Region
Descent Method Using Radial Basis Function Surrogate Models,” Mathematical
and Computational Applications, vol. 26, no. 2, 2021.
mla: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Derivative-Free Multiobjective
Trust Region Descent Method Using Radial Basis Function Surrogate Models.” Mathematical
and Computational Applications, vol. 26, no. 2, 31, 2021, doi:10.3390/mca26020031.
short: M.B. Berkemeier, S. Peitz, Mathematical and Computational Applications 26
(2021).
date_created: 2021-03-01T10:46:48Z
date_updated: 2022-01-06T06:54:55Z
department:
- _id: '101'
- _id: '655'
doi: 10.3390/mca26020031
intvolume: ' 26'
issue: '2'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://www.mdpi.com/2297-8747/26/2/31/pdf
oa: '1'
publication: Mathematical and Computational Applications
publication_identifier:
eissn:
- 2297-8747
publication_status: published
status: public
title: Derivative-Free Multiobjective Trust Region Descent Method Using Radial Basis
Function Surrogate Models
type: journal_article
user_id: '47427'
volume: 26
year: '2021'
...