Stochastic Gradient Descent in high dimensions: Effective dynamics and critical scaling

par Gérard Ben Arous

jeudi 19 mai 2022, 14:00 → 15:00 Europe/Paris

435 (UMPA)

(joint work with Reza Gheissari (UC Berkeley) and Aukosh Jagannath (Waterloo))

SGD is a workhorse for optimization and thus for statistics and machine learning, and it is well understood in low dimensions. But understanding its behavior in very high dimensions is not yet a simple task.  We describe interesting and new regimes for the limiting  dynamics of some summary statistics for SGD in high dimensions.

These regimes may differ from the expected one given by the usual wisdom in finite dimensions, i.e. the population gradient flow. We find that a new corrector term may be needed and that the phase portrait of these dynamics is quite complex and substantially different from what would be predicted using the classical low-dimensional approach, including for simple tasks, like Tensor PCA, or simple XOR classification.