BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//CERN//INDICO//EN
BEGIN:VEVENT
SUMMARY:Random Matrices and Dynamics of Optimization in Very High Dimensi
ons
DTSTART:20240325T130000Z
DTEND:20240325T150000Z
DTSTAMP:20240522T114000Z
UID:indico-event-11341@indico.math.cnrs.fr
CONTACT:cecile@ihes.fr
DESCRIPTION:Speakers: Gérard Ben Arous (CIMS\, New York University & IHES
)\n\nMachine learning and Data science algorithms involve in their last st
age the need for optimization of complex random functions in very high dim
ensions. Simple algorithms like Stochastic Gradient Descent (with small ba
tches) are used very effectively. I will concentrate on trying to understa
nd why these simple tools can still work in these complex and very over-pa
rametrized regimes. I will first introduce the whole framework for non-ex
perts\, from the structure of the typical tasks to the natural structures
of neural nets used in standard contexts. l will then cover briefly the cl
assical and usual context of SGD in finite dimensions. I will then survey
recent work with Reza Gheissari (Northwestern)\, Aukosh Jagannath (Waterlo
o) giving a general view for the existence of projected “effective dynam
ics” for “summary statistics” in much smaller dimensions\, which sti
ll rule the performance of very high dimensional systems\, as well . These
effective dynamics (as their so-called “critical regime”) define a dy
namical system in finite dimensions which may be quite complex\, and rules
the performance of the learning algorithm.The next step will be to unders
tand how the system finds these “summary statistics”. This is done in
the next work with the same authors and with Jiaoyang Huang (Wharton\, U-P
enn). This is based on a dynamical spectral transition of Random Matrix Th
eory: along the trajectory of the optimization path\, the Gram matix or th
e Hessian matrix develop outliers which carry these effective dynamics. I
will naturally first come back to the Random Matrix Tools needed here (the
behavior of the edge of the spectrum and the BBP transition) in a much br
oader context. And then illustrate the use of this point of view on a few
central examples of ML: multilayer neural nets for classification (of Gaus
sian mixtures)\, and the XOR examples\, for instance.\n\nhttps://indico.ma
th.cnrs.fr/event/11341/
LOCATION:Amphithéâtre Léon Motchane (IHES)
URL:https://indico.math.cnrs.fr/event/11341/
END:VEVENT
END:VCALENDAR