---
_id: '62072'
abstract:
- lang: eng
text: 'The vision of Automated Machine Learning (AutoML) is to produce high performing
ML pipelines that require very little human involvement or domain expertise to
use. Competitions and benchmarks have been critical tools for accelerating progress
in AutoML. However, much of the prior work on AutoML competitions has focused
on well-studied domains in machine learning such as vision and language—these
are domains which have benefited from several years of ML pipeline design by domain
experts, which brings the usage of AutoML into question in the first place. Recently,
AutoML for diverse tasks has emerged as an important research area that aims to
bring AutoML to the domains where it can have the most impact: the long tail of
ML tasks beyond vision and language. We present a retrospective report
of the AutoML Decathlon—an AutoML for diverse tasks competition hosted at NeurIPS
2022. The AutoML Decathlon presented participants with a set of 10 machine learning
tasks that are diverse along several axes: domain, input dimension, output dimension,
output type, objective function, and scale. Participants were tasked with developing
AutoML methods that performed well on a separate set of 10 hidden diverse
test tasks within a certain time budget, so as to discourage overfitting to the
initial set of tasks and to encourage efficiency. In this report, we outline the
details of the competition, discuss the top-5 submissions, analyze the results,
and compare top submissions to additional state-of-the-art baselines designed
specifically for diverse tasks. We conclude that the combination of existing efficient
AutoML techniques with modern advancements in ML such as large-scale transfer
learning, modern architectures, and differentiable Neural Architecture Search
(NAS) is a promising direction for AutoML for diverse tasks.'
author:
- first_name: Nicholas
full_name: Roberts, Nicholas
last_name: Roberts
- first_name: Samuel
full_name: Guo, Samuel
last_name: Guo
- first_name: Cong
full_name: Xu, Cong
last_name: Xu
- first_name: Ameet
full_name: Talwalkar, Ameet
last_name: Talwalkar
- first_name: David
full_name: Lander, David
last_name: Lander
- first_name: Lvfang
full_name: Tao, Lvfang
last_name: Tao
- first_name: Linhang
full_name: Cai, Linhang
last_name: Cai
- first_name: Shuaicheng
full_name: Niu, Shuaicheng
last_name: Niu
- first_name: Jianyu
full_name: Heng, Jianyu
last_name: Heng
- first_name: Hongyang
full_name: Qin, Hongyang
last_name: Qin
- first_name: Minwen
full_name: Deng, Minwen
last_name: Deng
- first_name: Johannes
full_name: Hog, Johannes
last_name: Hog
- first_name: Alexander
full_name: Pfefferle, Alexander
last_name: Pfefferle
- first_name: Sushil
full_name: Ammanaghatta Shivakumar, Sushil
last_name: Ammanaghatta Shivakumar
- first_name: Arjun
full_name: Krishnakumar, Arjun
last_name: Krishnakumar
- first_name: Yubo
full_name: Wang, Yubo
last_name: Wang
- first_name: Rhea
full_name: Sukthanker, Rhea
last_name: Sukthanker
- first_name: Frank
full_name: Hutter, Frank
last_name: Hutter
- first_name: Euxhen
full_name: Hasanaj, Euxhen
last_name: Hasanaj
- first_name: Tien-Dung
full_name: Le, Tien-Dung
last_name: Le
- first_name: Mikhail
full_name: Khodak, Mikhail
last_name: Khodak
- first_name: Yuriy
full_name: Nevmyvaka, Yuriy
last_name: Nevmyvaka
- first_name: Kashif
full_name: Rasul, Kashif
last_name: Rasul
- first_name: Frederic
full_name: Sala, Frederic
last_name: Sala
- first_name: Anderson
full_name: Schneider, Anderson
last_name: Schneider
- first_name: Junhong
full_name: Shen, Junhong
last_name: Shen
- first_name: Evan
full_name: Sparks, Evan
last_name: Sparks
citation:
ama: 'Roberts N, Guo S, Xu C, et al. AutoML Decathlon: Diverse Tasks, Modern Methods,
and Efficiency at Scale. In: Ciccone M, Stolovitzky G, Albrecht J, eds. Proceedings
of the NeurIPS 2022 Competitions Track. Vol 220. Proceedings of Machine Learning
Research. PMLR; 2022:151–170.'
apa: 'Roberts, N., Guo, S., Xu, C., Talwalkar, A., Lander, D., Tao, L., Cai, L.,
Niu, S., Heng, J., Qin, H., Deng, M., Hog, J., Pfefferle, A., Ammanaghatta Shivakumar,
S., Krishnakumar, A., Wang, Y., Sukthanker, R., Hutter, F., Hasanaj, E., … Sparks,
E. (2022). AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at
Scale. In M. Ciccone, G. Stolovitzky, & J. Albrecht (Eds.), Proceedings
of the NeurIPS 2022 Competitions Track (Vol. 220, pp. 151–170). PMLR.'
bibtex: '@inproceedings{Roberts_Guo_Xu_Talwalkar_Lander_Tao_Cai_Niu_Heng_Qin_et
al._2022, series={Proceedings of Machine Learning Research}, title={AutoML Decathlon:
Diverse Tasks, Modern Methods, and Efficiency at Scale}, volume={220}, booktitle={Proceedings
of the NeurIPS 2022 Competitions Track}, publisher={PMLR}, author={Roberts, Nicholas
and Guo, Samuel and Xu, Cong and Talwalkar, Ameet and Lander, David and Tao, Lvfang
and Cai, Linhang and Niu, Shuaicheng and Heng, Jianyu and Qin, Hongyang and et
al.}, editor={Ciccone, Marco and Stolovitzky, Gustavo and Albrecht, Jacob}, year={2022},
pages={151–170}, collection={Proceedings of Machine Learning Research} }'
chicago: 'Roberts, Nicholas, Samuel Guo, Cong Xu, Ameet Talwalkar, David Lander,
Lvfang Tao, Linhang Cai, et al. “AutoML Decathlon: Diverse Tasks, Modern Methods,
and Efficiency at Scale.” In Proceedings of the NeurIPS 2022 Competitions Track,
edited by Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht, 220:151–170.
Proceedings of Machine Learning Research. PMLR, 2022.'
ieee: 'N. Roberts et al., “AutoML Decathlon: Diverse Tasks, Modern Methods,
and Efficiency at Scale,” in Proceedings of the NeurIPS 2022 Competitions Track,
2022, vol. 220, pp. 151–170.'
mla: 'Roberts, Nicholas, et al. “AutoML Decathlon: Diverse Tasks, Modern Methods,
and Efficiency at Scale.” Proceedings of the NeurIPS 2022 Competitions Track,
edited by Marco Ciccone et al., vol. 220, PMLR, 2022, pp. 151–170.'
short: 'N. Roberts, S. Guo, C. Xu, A. Talwalkar, D. Lander, L. Tao, L. Cai, S. Niu,
J. Heng, H. Qin, M. Deng, J. Hog, A. Pfefferle, S. Ammanaghatta Shivakumar, A.
Krishnakumar, Y. Wang, R. Sukthanker, F. Hutter, E. Hasanaj, T.-D. Le, M. Khodak,
Y. Nevmyvaka, K. Rasul, F. Sala, A. Schneider, J. Shen, E. Sparks, in: M. Ciccone,
G. Stolovitzky, J. Albrecht (Eds.), Proceedings of the NeurIPS 2022 Competitions
Track, PMLR, 2022, pp. 151–170.'
date_created: 2025-11-04T12:24:24Z
date_updated: 2025-11-04T12:25:54Z
editor:
- first_name: Marco
full_name: Ciccone, Marco
last_name: Ciccone
- first_name: Gustavo
full_name: Stolovitzky, Gustavo
last_name: Stolovitzky
- first_name: Jacob
full_name: Albrecht, Jacob
last_name: Albrecht
extern: '1'
intvolume: ' 220'
language:
- iso: eng
page: 151–170
publication: Proceedings of the NeurIPS 2022 Competitions Track
publisher: PMLR
series_title: Proceedings of Machine Learning Research
status: public
title: 'AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale'
type: conference
user_id: '124732'
volume: 220
year: '2022'
...