Orateur
Description
We study the spectral structure of Gram matrices arising in high-dimensional learning problems and its implications for optimization dynamics. First, we prove that for independent, centered, anisotropic vectors, the normalized Gram matrix converges to the identity in operator norm, under mild assumptions on intrinsic dimension and tail behavior. This extends classical isotropic results to a broad non-homogeneous setting.
Second, we analyze random feature models and show that Gram matrices converge to the kernel matrix for a large range of activation functions. Combining this with the anisotropic result, we show that, in many regimes of interest, correlations vanish, leading to an approximate (block-)diagonalization of the kernel.
Finally, we leverage these spectral results to study gradient flow dynamics for least-squares objectives. We prove a decoupling property, showing that the dynamics asymptotically behave as independent one-dimensional processes, with a block structure in the presence of statistical dependencies. This provides a theoretical explanation for "task arithmetic" and the "absence" of interference in overparameterized models.
Our results unify independence-based and geometry-based mechanisms for decoupling, and apply to random features and structured data such as mixtures.