--- _id: '46649' abstract: - lang: eng text: "Different conflicting optimization criteria arise naturally in various Deep\r\nLearning scenarios. These can address different main tasks (i.e., in the\r\nsetting of Multi-Task Learning), but also main and secondary tasks such as loss\r\nminimization versus sparsity. The usual approach is a simple weighting of the\r\ncriteria, which formally only works in the convex setting. In this paper, we\r\npresent a Multi-Objective Optimization algorithm using a modified Weighted\r\nChebyshev scalarization for training Deep Neural Networks (DNNs) with respect\r\nto several tasks. By employing this scalarization technique, the algorithm can\r\nidentify all optimal solutions of the original problem while reducing its\r\ncomplexity to a sequence of single-objective problems. The simplified problems\r\nare then solved using an Augmented Lagrangian method, enabling the use of\r\npopular optimization techniques such as Adam and Stochastic Gradient Descent,\r\nwhile efficaciously handling constraints. Our work aims to address the\r\n(economical and also ecological) sustainability issue of DNN models, with a\r\nparticular focus on Deep Multi-Task models, which are typically designed with a\r\nvery large number of weights to perform equally well on multiple tasks. Through\r\nexperiments conducted on two Machine Learning datasets, we demonstrate the\r\npossibility of adaptively sparsifying the model during training without\r\nsignificantly impacting its performance, if we are willing to apply\r\ntask-specific adaptations to the network weights. Code is available at\r\nhttps://github.com/salomonhotegni/MDMTN." author: - first_name: Sedjro Salomon full_name: Hotegni, Sedjro Salomon id: '97995' last_name: Hotegni - first_name: Sebastian full_name: Peitz, Sebastian id: '47427' last_name: Peitz orcid: 0000-0002-3389-793X - first_name: Manuel Bastian full_name: Berkemeier, Manuel Bastian id: '51701' last_name: Berkemeier citation: ama: Hotegni SS, Peitz S, Berkemeier MB. Multi-Objective Optimization for Sparse Deep Neural Network Training. arXiv:230812243. Published online 2023. apa: Hotegni, S. S., Peitz, S., & Berkemeier, M. B. (2023). Multi-Objective Optimization for Sparse Deep Neural Network Training. In arXiv:2308.12243. bibtex: '@article{Hotegni_Peitz_Berkemeier_2023, title={Multi-Objective Optimization for Sparse Deep Neural Network Training}, journal={arXiv:2308.12243}, author={Hotegni, Sedjro Salomon and Peitz, Sebastian and Berkemeier, Manuel Bastian}, year={2023} }' chicago: Hotegni, Sedjro Salomon, Sebastian Peitz, and Manuel Bastian Berkemeier. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” ArXiv:2308.12243, 2023. ieee: S. S. Hotegni, S. Peitz, and M. B. Berkemeier, “Multi-Objective Optimization for Sparse Deep Neural Network Training,” arXiv:2308.12243. 2023. mla: Hotegni, Sedjro Salomon, et al. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” ArXiv:2308.12243, 2023. short: S.S. Hotegni, S. Peitz, M.B. Berkemeier, ArXiv:2308.12243 (2023). date_created: 2023-08-24T07:44:36Z date_updated: 2023-08-24T08:22:17Z department: - _id: '655' external_id: arxiv: - '2308.12243' has_accepted_license: '1' language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/2308.12243 oa: '1' page: '13' publication: arXiv:2308.12243 status: public title: Multi-Objective Optimization for Sparse Deep Neural Network Training type: preprint user_id: '97995' year: '2023' ... --- _id: '33150' abstract: - lang: eng text: In this article, we build on previous work to present an optimization algorithm for nonlinearly constrained multi-objective optimization problems. The algorithm combines a surrogate-assisted derivative-free trust-region approach with the filter method known from single-objective optimization. Instead of the true objective and constraint functions, so-called fully linear models are employed and we show how to deal with the gradient inexactness in the composite step setting, adapted from single-objective optimization as well. Under standard assumptions, we prove convergence of a subset of iterates to a quasi-stationary point and if constraint qualifications hold, then the limit point is also a KKT-point of the multi-objective problem. author: - first_name: Manuel Bastian full_name: Berkemeier, Manuel Bastian id: '51701' last_name: Berkemeier - first_name: Sebastian full_name: Peitz, Sebastian id: '47427' last_name: Peitz orcid: 0000-0002-3389-793X citation: ama: Berkemeier MB, Peitz S. Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using Inexact Gradients. arXiv:220812094. Published online 2022. apa: Berkemeier, M. B., & Peitz, S. (2022). Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using Inexact Gradients. In arXiv:2208.12094. bibtex: '@article{Berkemeier_Peitz_2022, title={Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using Inexact Gradients}, journal={arXiv:2208.12094}, author={Berkemeier, Manuel Bastian and Peitz, Sebastian}, year={2022} }' chicago: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Multi-Objective Trust-Region Filter Method for Nonlinear Constraints Using Inexact Gradients.” ArXiv:2208.12094, 2022. ieee: M. B. Berkemeier and S. Peitz, “Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using Inexact Gradients,” arXiv:2208.12094. 2022. mla: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Multi-Objective Trust-Region Filter Method for Nonlinear Constraints Using Inexact Gradients.” ArXiv:2208.12094, 2022. short: M.B. Berkemeier, S. Peitz, ArXiv:2208.12094 (2022). date_created: 2022-08-26T06:08:06Z date_updated: 2022-08-26T06:12:10Z department: - _id: '101' - _id: '655' external_id: arxiv: - '2208.12094' language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/pdf/2208.12094 oa: '1' publication: arXiv:2208.12094 status: public title: Multi-Objective Trust-Region Filter Method for Nonlinear Constraints using Inexact Gradients type: preprint user_id: '47427' year: '2022' ... --- _id: '21337' abstract: - lang: eng text: "We present a flexible trust region descend algorithm for unconstrained and\r\nconvexly constrained multiobjective optimization problems. It is targeted at\r\nheterogeneous and expensive problems, i.e., problems that have at least one\r\nobjective function that is computationally expensive. The method is\r\nderivative-free in the sense that neither need derivative information be\r\navailable for the expensive objectives nor are gradients approximated using\r\nrepeated function evaluations as is the case in finite-difference methods.\r\nInstead, a multiobjective trust region approach is used that works similarly to\r\nits well-known scalar pendants. Local surrogate models constructed from\r\nevaluation data of the true objective functions are employed to compute\r\npossible descent directions. In contrast to existing multiobjective trust\r\nregion algorithms, these surrogates are not polynomial but carefully\r\nconstructed radial basis function networks. This has the important advantage\r\nthat the number of data points scales linearly with the parameter space\r\ndimension. The local models qualify as fully linear and the corresponding\r\ngeneral scalar framework is adapted for problems with multiple objectives.\r\nConvergence to Pareto critical points is proven and numerical examples\r\nillustrate our findings." article_number: '31' author: - first_name: Manuel Bastian full_name: Berkemeier, Manuel Bastian id: '51701' last_name: Berkemeier - first_name: Sebastian full_name: Peitz, Sebastian id: '47427' last_name: Peitz orcid: 0000-0002-3389-793X citation: ama: Berkemeier MB, Peitz S. Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models. Mathematical and Computational Applications. 2021;26(2). doi:10.3390/mca26020031 apa: Berkemeier, M. B., & Peitz, S. (2021). Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models. Mathematical and Computational Applications, 26(2). https://doi.org/10.3390/mca26020031 bibtex: '@article{Berkemeier_Peitz_2021, title={Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models}, volume={26}, DOI={10.3390/mca26020031}, number={231}, journal={Mathematical and Computational Applications}, author={Berkemeier, Manuel Bastian and Peitz, Sebastian}, year={2021} }' chicago: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models.” Mathematical and Computational Applications 26, no. 2 (2021). https://doi.org/10.3390/mca26020031. ieee: M. B. Berkemeier and S. Peitz, “Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models,” Mathematical and Computational Applications, vol. 26, no. 2, 2021. mla: Berkemeier, Manuel Bastian, and Sebastian Peitz. “Derivative-Free Multiobjective Trust Region Descent Method Using Radial  Basis Function Surrogate Models.” Mathematical and Computational Applications, vol. 26, no. 2, 31, 2021, doi:10.3390/mca26020031. short: M.B. Berkemeier, S. Peitz, Mathematical and Computational Applications 26 (2021). date_created: 2021-03-01T10:46:48Z date_updated: 2022-01-06T06:54:55Z department: - _id: '101' - _id: '655' doi: 10.3390/mca26020031 intvolume: ' 26' issue: '2' language: - iso: eng main_file_link: - open_access: '1' url: https://www.mdpi.com/2297-8747/26/2/31/pdf oa: '1' publication: Mathematical and Computational Applications publication_identifier: eissn: - 2297-8747 publication_status: published status: public title: Derivative-Free Multiobjective Trust Region Descent Method Using Radial Basis Function Surrogate Models type: journal_article user_id: '47427' volume: 26 year: '2021' ...