Self-Supervised Learning for Invariant Representations From Multi-Spectral and SAR Images

Authors

Pallavi Jain, Technological University DublinFollow
Bianca Schoen Phelan, Technological University Dublin
Robert J. Ross, Technological University DublinFollow

Author ORCID Identifier

https://orcid.org/0000-0002-1731-8993

Document Type

Article

Rights

Available under a Creative Commons Attribution Non-Commercial Share Alike 4.0 International Licence

Disciplines

Computer Sciences, Geosciences, (multidisciplinary)

Publication Details

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract

Self-Supervised learning (SSL) has become the new state of the art in several domain classification and segmentation tasks. One popular category of SSL are distillation networks such as Bootstrap Your Own Latent (BYOL). This work proposes RS-BYOL, which builds on BYOL in the remote sensing (RS) domain where data are non-trivially different from natural RGB images. Since multi-spectral (MS) and synthetic aperture radar (SAR) sensors provide varied spectral and spatial resolution information, we utilise them as an implicit augmentation to learn invariant feature embeddings. In order to learn RS based invariant features with SSL, we trained RS-BYOL in two ways, i.e. single channel feature learning and three channel feature learning. This work explores the usefulness of single channel feature learning from random 10 MS bands of 10m-20 m resolution and VV-VH of SAR bands compared to the common notion of using three or more bands. In our linear probing evaluation, these single channel features reached a 0.92 F1 score on the EuroSAT classification task and 59.6 mIoU on the IEEE Data Fusion Contest (DFC) segmentation task for certain single bands. We also compare our results with ImageNet weights and show that the RS based SSL model outperforms the supervised ImageNet based model. We further explore the usefulness of multi-modal data compared to single modality data, and it is shown that utilising MS and SAR data allows better invariant representations to be learnt than utilising only MS data.

DOI

https://doi.org/10.1109/JSTARS.2022.3204888

Recommended Citation

P. Jain, B. Schoen-Phelan and R. Ross, "Self-Supervised Learning for Invariant Representations From Multi-Spectral and SAR Images," in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, doi: 10.1109/JSTARS.2022.3204888.

Funder

Science Foundation Ireland

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.