On Geometric Dynamics of Deep Linear Networks

par Florian Kogelbauer (ETH Zürich)

mardi 6 oct. 2026, 11:00 → 12:00 Europe/Paris

We discuss the training dynamics of deep linear networks from a geometric and dynamical-systems perspective. Gradient descent induces a Riemannian gradient flow on the end-to-end network map, revealing a slow-fast structure in the evolution of its singular values. This provides a dynamical explanation for the implicit bias toward low-rank solutions and connects learning dynamics with invariant-manifold and model-reduction ideas. We show how this structure can inform initialization strategies for accelerating training.