Self-supervised Test-time Adaptation on Video Data

Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, Andreas Dengel

In: WACV. IEEE Winter Conference on Applications of Computer Vision (WACV-2022) January 4-8 IEEE 1/2022.


In typical computer vision problems revolving around video data, pre-trained models are simply evaluated at test time without further adaptation. This general approach inevitably fails to capture potential distribution shifts between training and test data. Adapting a pre-trained model to a new video encountered at test time could be essential to improve performance or to avoid the potentially catastrophic effects of such a shift. However, the lack of available annotations prevents practitioners from using vanilla fine-tuning techniques. This paper explores whether the recent progress in self-supervised learning and test-time domain adaptation (TTA) methods in the image domain can be leveraged to efficiently adapt a model to a previously unseen and unlabelled video. We analyze the effectiveness of several recent self-supervised TTA techniques under the effect of both mild (but arbitrary) and severe domain shifts. From our extensive benchmark on multiple self-supervised dense tracking methods under various domain shifts, we find out that self-supervised TTA methods consistently improve the performance compared to the baselines without adaptation, especially in the presence of severe covariate shift.

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence