Memorization-Dilation: Modeling Neural Collapse Under Noise

D.A. Nguyen, R. Levie, J. Lienen, G. Kutyniok, E. Hüllermeier, ArXiv:2206.05530 (2022).

Preprint | English
Author
Nguyen, Duc Anh; Levie, Ron; Lienen, JulianLibreCat; Kutyniok, Gitta; Hüllermeier, EykeLibreCat
Abstract
The notion of neural collapse refers to several emergent phenomena that have been empirically observed across various canonical classification problems. During the terminal phase of training a deep neural network, the feature embedding of all examples of the same class tend to collapse to a single representation, and the features of different classes tend to separate as much as possible. Neural collapse is often studied through a simplified model, called the unconstrained feature representation, in which the model is assumed to have "infinite expressivity" and can map each data point to any arbitrary representation. In this work, we propose a more realistic variant of the unconstrained feature representation that takes the limited expressivity of the network into account. Empirical evidence suggests that the memorization of noisy data points leads to a degradation (dilation) of the neural collapse. Using a model of the memorization-dilation (M-D) phenomenon, we show one mechanism by which different losses lead to different performances of the trained network on noisy data. Our proofs reveal why label smoothing, a modification of cross-entropy empirically observed to produce a regularization effect, leads to improved generalization in classification tasks.
Publishing Year
Journal Title
arXiv:2206.05530
LibreCat-ID

Cite this

Nguyen DA, Levie R, Lienen J, Kutyniok G, Hüllermeier E. Memorization-Dilation: Modeling Neural Collapse Under Noise. arXiv:220605530. Published online 2022.
Nguyen, D. A., Levie, R., Lienen, J., Kutyniok, G., & Hüllermeier, E. (2022). Memorization-Dilation: Modeling Neural Collapse Under Noise. In arXiv:2206.05530.
@article{Nguyen_Levie_Lienen_Kutyniok_Hüllermeier_2022, title={Memorization-Dilation: Modeling Neural Collapse Under Noise}, journal={arXiv:2206.05530}, author={Nguyen, Duc Anh and Levie, Ron and Lienen, Julian and Kutyniok, Gitta and Hüllermeier, Eyke}, year={2022} }
Nguyen, Duc Anh, Ron Levie, Julian Lienen, Gitta Kutyniok, and Eyke Hüllermeier. “Memorization-Dilation: Modeling Neural Collapse Under Noise.” ArXiv:2206.05530, 2022.
D. A. Nguyen, R. Levie, J. Lienen, G. Kutyniok, and E. Hüllermeier, “Memorization-Dilation: Modeling Neural Collapse Under Noise,” arXiv:2206.05530. 2022.
Nguyen, Duc Anh, et al. “Memorization-Dilation: Modeling Neural Collapse Under Noise.” ArXiv:2206.05530, 2022.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]

Link(s) to Main File(s)
Access Level
Restricted Closed Access

Export

Marked Publications

Open Data LibreCat

Search this title in

Google Scholar