ORCID
- Nouman Ashraf: 0000-0002-0164-8031
Abstract
Learning generalizable video representations from unlabeled data remains a fundamental challenge in computer vision, as existing self-supervised learning approaches primarily rely on modeling statistical correlations rather than capturing structured temporal dependencies in video data. While recent methods such as masked video modeling and contrastive learning achieve strong empirical performance, they often fail to explicitly account for how changes in motion influence future states, limiting robustness and generalization under distribution shifts. In this work, we propose TRACE, a self-supervised framework for learning intervention-aware spatiotemporal representations by incorporating temporal interventions and counterfactual learning objectives. TRACE operates in latent space by decomposing video representations into content and motion components, enabling targeted perturbations of motion variables while preserving semantic structure. A temporal intervention module generates alternative trajectories by modifying motion dynamics, and a counterfactual prediction objective enforces consistency between predicted and intervention-induced future representations. This formulation encourages the model to learn representations that are sensitive to intervention-induced changes rather than relying on spurious correlations. Extensive experiments demonstrate that TRACE consistently outperforms state-of-the-art self-supervised video learning methods across multiple benchmarks. In particular, TRACE achieves 58.9% on Kinetics-400 and 27.6% on Something-Something V2 under linear evaluation, outperforming prior methods while also improving cross-dataset transfer performance and reducing robustness degradation under motion perturbations (from −9.1% to −4.8%). Furthermore, TRACE shows improved performance on intervention-based and counterfactual evaluation protocols, indicating stronger modeling of temporal dependencies under dynamic changes. These results suggest that incorporating intervention-based reasoning into self-supervised video learning provides a practical pathway toward more robust, transferable, and semantically meaningful video representations.
Keywords
Counterfactual learning, Intervention-based learning, Self-supervised learning, Spatiotemporal modeling, Temporal modeling, Video representation learning, Video understanding, Vision transformers
DOI Link
Publication Date
2026-01-01
Publication Title
Machine Learning with Applications
Volume
25
Deposit Date
2026-11-05
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Additional Links
Recommended Citation
Chaudhry, H., Kulsoom, F., Mohsin, S., Aslam, S., & Ashraf, N. (2026) 'TRACE: Intervention-aware temporal representation learning for video understanding', Machine Learning with Applications, 25. Available at: 10.1016/j.mlwa.2026.100976