Artificial Neural Networks and Kernel Methods

par M. Franck Gabriel (Ecole Polytechnique Fédérale de Lausanne)

jeudi 24 mars 2022, 14:00 → 15:00 Europe/Paris

Fokko du Cloux (ICJ, Bâtiment Braconnier)

Artificial Neural Networks (ANN) provide complicated yet powerful parametrization of functions used to optimize functional cost over the space of functions. Until recent years, ANN optimisation lacked a solid theoretical basis, although it has given outstanding practical results.

Using a suitable random initialisation of Artificial Neural Networks (ANN), we show how one can describe, in the functional space, the limit of the evolution of fully-connected ANN when their width tends towards infinity. In particular, we will unveil a deep link between Artificial Neural Networks (ANN) and Kernel Methods: within this limit, an ANN is initially a Gaussian process and follows, during learning, a gradient descent convolved with a kernel called the Neural Tangent Kernel.

Connecting neural networks to the well-established theory of kernel methods allows us to understand the dynamics of neural networks, their generalization capability and can help select appropriate architectural features of the network to be trained.