Temporal Self-Supervision for Unsupervised Video Representation Learning

Abstract:  

Video data is abundant nowadays and provides a rich source for visual representation learning, yet videos' high dimensionality and the high cost of dense human annotation make supervised learning prohibitive. As a result, self-supervised learning (SSL) has emerged as a potential solution, wherein supervision signals are constructed automatically without human labor.

In this talk, we explore SSL methods for video, focusing on learning tasks that exploit the temporal dimension inherent to videos. We will demonstrate how time-related pretext tasks, i.e., recognizing temporal input transformations, benefit the learning of features crucial to action recognition. Besides such pretext tasks, contrastive learning has emerged as a powerful SSL technique. These methods learn to distinguish instances while building invariance to input transformations. Instead, we propose a contrastive objective that also enables the learning of equivariance to input transformations. Our experiments demonstrate the benefit of learning time-equivariant video representations for downstream action recognition.

Finally, we explore how to effectively leverage temporal self-supervision for multimodal audio-visual video representation learning by combining intra- and cross-modal temporal and contrastive tasks.

Short bio:

Simon Jenni is a Research Scientist at Adobe Research. His research interests are in computer vision and deep learning, focusing on self-supervised learning methods for images and videos. Previously, Simon was a Ph.D. student in the Computer Vision Group at the University of Bern, supervised by Paolo Favaro. He received his BSc degree in Computer Science in 2015 and his MSc degree in Computer Science in 2017, both at the University of Bern.