Multi-Objective Optimization for Sparse Deep Neural Network Training
S.S. Hotegni, S. Peitz, M.B. Berkemeier, ArXiv:2308.12243 (2023).
Download (ext.)
Preprint
| English
Author
Department
Abstract
Different conflicting optimization criteria arise naturally in various Deep
Learning scenarios. These can address different main tasks (i.e., in the
setting of Multi-Task Learning), but also main and secondary tasks such as loss
minimization versus sparsity. The usual approach is a simple weighting of the
criteria, which formally only works in the convex setting. In this paper, we
present a Multi-Objective Optimization algorithm using a modified Weighted
Chebyshev scalarization for training Deep Neural Networks (DNNs) with respect
to several tasks. By employing this scalarization technique, the algorithm can
identify all optimal solutions of the original problem while reducing its
complexity to a sequence of single-objective problems. The simplified problems
are then solved using an Augmented Lagrangian method, enabling the use of
popular optimization techniques such as Adam and Stochastic Gradient Descent,
while efficaciously handling constraints. Our work aims to address the
(economical and also ecological) sustainability issue of DNN models, with a
particular focus on Deep Multi-Task models, which are typically designed with a
very large number of weights to perform equally well on multiple tasks. Through
experiments conducted on two Machine Learning datasets, we demonstrate the
possibility of adaptively sparsifying the model during training without
significantly impacting its performance, if we are willing to apply
task-specific adaptations to the network weights. Code is available at
https://github.com/salomonhotegni/MDMTN.
Publishing Year
Journal Title
arXiv:2308.12243
Page
13
LibreCat-ID
Cite this
Hotegni SS, Peitz S, Berkemeier MB. Multi-Objective Optimization for Sparse Deep Neural Network Training. arXiv:230812243. Published online 2023.
Hotegni, S. S., Peitz, S., & Berkemeier, M. B. (2023). Multi-Objective Optimization for Sparse Deep Neural Network Training. In arXiv:2308.12243.
@article{Hotegni_Peitz_Berkemeier_2023, title={Multi-Objective Optimization for Sparse Deep Neural Network Training}, journal={arXiv:2308.12243}, author={Hotegni, Sedjro Salomon and Peitz, Sebastian and Berkemeier, Manuel Bastian}, year={2023} }
Hotegni, Sedjro Salomon, Sebastian Peitz, and Manuel Bastian Berkemeier. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” ArXiv:2308.12243, 2023.
S. S. Hotegni, S. Peitz, and M. B. Berkemeier, “Multi-Objective Optimization for Sparse Deep Neural Network Training,” arXiv:2308.12243. 2023.
Hotegni, Sedjro Salomon, et al. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” ArXiv:2308.12243, 2023.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]
Link(s) to Main File(s)
Access Level
Closed Access