{"title":"Multi-Objective Optimization for Sparse Deep Neural Network Training","abstract":[{"lang":"eng","text":"Different conflicting optimization criteria arise naturally in various Deep\r\nLearning scenarios. These can address different main tasks (i.e., in the\r\nsetting of Multi-Task Learning), but also main and secondary tasks such as loss\r\nminimization versus sparsity. The usual approach is a simple weighting of the\r\ncriteria, which formally only works in the convex setting. In this paper, we\r\npresent a Multi-Objective Optimization algorithm using a modified Weighted\r\nChebyshev scalarization for training Deep Neural Networks (DNNs) with respect\r\nto several tasks. By employing this scalarization technique, the algorithm can\r\nidentify all optimal solutions of the original problem while reducing its\r\ncomplexity to a sequence of single-objective problems. The simplified problems\r\nare then solved using an Augmented Lagrangian method, enabling the use of\r\npopular optimization techniques such as Adam and Stochastic Gradient Descent,\r\nwhile efficaciously handling constraints. Our work aims to address the\r\n(economical and also ecological) sustainability issue of DNN models, with a\r\nparticular focus on Deep Multi-Task models, which are typically designed with a\r\nvery large number of weights to perform equally well on multiple tasks. Through\r\nexperiments conducted on two Machine Learning datasets, we demonstrate the\r\npossibility of adaptively sparsifying the model during training without\r\nsignificantly impacting its performance, if we are willing to apply\r\ntask-specific adaptations to the network weights. Code is available at\r\nhttps://github.com/salomonhotegni/MDMTN."}],"publication":"arXiv:2308.12243","language":[{"iso":"eng"}],"page":"13","date_updated":"2023-08-24T08:22:17Z","type":"preprint","oa":"1","has_accepted_license":"1","citation":{"short":"S.S. Hotegni, S. Peitz, M.B. Berkemeier, ArXiv:2308.12243 (2023).","bibtex":"@article{Hotegni_Peitz_Berkemeier_2023, title={Multi-Objective Optimization for Sparse Deep Neural Network Training}, journal={arXiv:2308.12243}, author={Hotegni, Sedjro Salomon and Peitz, Sebastian and Berkemeier, Manuel Bastian}, year={2023} }","ieee":"S. S. Hotegni, S. Peitz, and M. B. Berkemeier, “Multi-Objective Optimization for Sparse Deep Neural Network Training,” <i>arXiv:2308.12243</i>. 2023.","ama":"Hotegni SS, Peitz S, Berkemeier MB. Multi-Objective Optimization for Sparse Deep Neural Network Training. <i>arXiv:230812243</i>. Published online 2023.","mla":"Hotegni, Sedjro Salomon, et al. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” <i>ArXiv:2308.12243</i>, 2023.","chicago":"Hotegni, Sedjro Salomon, Sebastian Peitz, and Manuel Bastian Berkemeier. “Multi-Objective Optimization for Sparse Deep Neural Network Training.” <i>ArXiv:2308.12243</i>, 2023.","apa":"Hotegni, S. S., Peitz, S., &#38; Berkemeier, M. B. (2023). Multi-Objective Optimization for Sparse Deep Neural Network Training. In <i>arXiv:2308.12243</i>."},"department":[{"_id":"655"}],"external_id":{"arxiv":["2308.12243"]},"date_created":"2023-08-24T07:44:36Z","status":"public","_id":"46649","user_id":"97995","main_file_link":[{"open_access":"1","url":"https://arxiv.org/abs/2308.12243"}],"author":[{"last_name":"Hotegni","full_name":"Hotegni, Sedjro Salomon","id":"97995","first_name":"Sedjro Salomon"},{"full_name":"Peitz, Sebastian","last_name":"Peitz","id":"47427","first_name":"Sebastian","orcid":"0000-0002-3389-793X"},{"last_name":"Berkemeier","full_name":"Berkemeier, Manuel Bastian","first_name":"Manuel Bastian","id":"51701"}],"year":"2023"}