Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training

Add code
Sep 25, 2023
Jiangliu Wang, Jianbo Jiao, Yibing Song, Stephen James, Zhan Tong, Chongjian Ge, Pieter Abbeel, Yun-hui Liu

Figure 1 for Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training
Figure 2 for Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training
Figure 3 for Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training
Figure 4 for Speed Co-Augmentation for Unsupervised Audio-Visual Pre-training

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

This work aims to improve unsupervised audio-visual pre-training. Inspired by the efficacy of data augmentation in visual contrastive learning, we propose a novel speed co-augmentation method that randomly changes the playback speeds of both audio and video data. Despite its simplicity, the speed co-augmentation method possesses two compelling attributes: (1) it increases the diversity of audio-visual pairs and doubles the size of negative pairs, resulting in a significant enhancement in the learned representations, and (2) it changes the strict correlation between audio-visual pairs but introduces a partial relationship between the augmented pairs, which is modeled by our proposed SoftInfoNCE loss to further boost the performance. Experimental results show that the proposed method significantly improves the learned representations when compared to vanilla audio-visual contrastive learning.

* Published at the CVPR 2023 Sight and Sound workshop  
View paper onarxiv icon

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon