Description
Children quickly develop powerful visual representations that support visual recognition, such as object recognition, with minimal supervision. However, the principles underpinning this development are poorly understood. In particular, it is unclear how natural experience interacts with unsupervised learning mechanisms to shape semantic representations.
This study explores whether the daily experience of adult humans, combined with two prominent theories of unsupervised learning, is sufficient to support the emergence of strong visual representations. We simulate key aspects of human visuomotor experience with 300 hours of videos collected with head-mounted cameras, combined with eye and body movements of the camera wearer. As a computational model of human learning, we train a bio-inspired class of machine learning models that learns temporally consistent visual representations and aligns visual representations with co-occurring body movements.
We show that the resulting representations achieve strong performance on a range of downstream tasks, including object recognition, scene recognition, and out-of-distribution generalization, despite the absence of explicit semantic supervision. Our analysis shows that naturalistic eye and body movements favor the encoding of shape-based features over texture-based cues, mirroring biases observed in early visual development in toddlers. Moreover, we observe that a bio-inspired emphasis on central vision promotes $16\times$ more computationally efficient learning, with minimal impact on the quality of semantic representations.
Taken together, these findings suggest that sensorimotor learning may be a key principle underlying the development of robust and generalizable visual representations. This research is part of the machine learning part of DATA program and is carried out at the Institut Pascal.