From Stability of Neural Networks Optimizers to Armijo Rule Generalization

par Bilel Bensaid (IMB)

mardi 26 mars 2024, 11:15 → 12:15 Europe/Paris

Salle K. Johnson, 1er étage (1R3)

These recent years, a great number of algorithms have been developed to optimize neural networks parameters (p-GD, clipping GD, Momentum, RMSProp, Adam) but they need an accurate tuning to be stable and efficient. To get rid of the long and experimental step of GridSearch, we are looking for adaptive and stable optimizers. By re-interpreting Armijo as a Lyapunov preserving discretization, we generalize the backtracking line-search to any ML optimizers and prove the asymptotic stability of this policy using recent tools coming from selection theory. Then, we establish convergence results for Armjo applied to p-GD, Momentum and RMSProp under the Lojasiewicz assumption. This gives new results for the convergence of constant step size RMSProp without the classical bounded assumptions. Finally, we give a new upper bound for Armijo GD under (L0, L1) Lipshitz assumption, promoting the use of adaptive time step strategies for Recurrent Neural Networks.