When Does Contrastive Visual Representation Learning Work?

[brunosan]

https://arxiv.org/pdf/2105.05837

Recent self-supervised representation learning techniques have largely closed the gap between supervised and
unsupervised learning on ImageNet classification. While
the particulars of pretraining on ImageNet are now relatively well understood, the field still lacks widely accepted
best practices for replicating this success on other datasets.
As a first step in this direction, we study contrastive selfsupervised learning on four diverse large-scale datasets.
By looking through the lenses of data quantity, data domain, data quality, and task granularity, we provide new
insights into the necessary conditions for successful selfsupervised learning. Our key findings include observations
such as: (i) the benefit of additional pretraining data beyond
500k images is modest, (ii) adding pretraining images from
another domain does not lead to more general representations, (iii) corrupted pretraining images have a disparate
impact on supervised and self-supervised pretraining, and
(iv) contrastive learning lags far behind supervised learning on fine-grained visual classification tasks.

[brunosan]

This paper evaluates the effectiveness of contrastive self-supervised learning (SSL) across diverse datasets, identifying key factors influencing its performance.

Note: Contrastive Self-Supervised Learning (SSL) is learning method (like MAE or DINO). It is based on comparisons using a contrastive loss function. Using an anchors as reference, embeddings of positive (similar) examples are pushed closer together, whereas embeddings of negative (dissimilar) examples are pushed apart. You do need a criteria to set positive/negative examples, but the model only get that "label".

The paper di NOT use satellite data, but used images: ImageNet (1.3M images, diverse object categories), iNat21 (2.7M images, fine-grained species classification), Places365 (1.8M images, scene recognition), and GLC20 (1M images, geographical species distribution and land cover classification).

Take aways from the paper:

• Data Quantity: Little to no performance gain from pretraining with over 500,000 images. Self-supervised pretraining saturates with large datasets, suggesting limitations in scaling.
• Pretraining Domain: In-domain pretraining significantly outperforms cross-domain pretraining. Self-supervised methods are still challenged in generalizing across domains, showing limited benefits from mixed-domain data.
• Data Quality: Corrupted pretraining data (e.g., lower resolution, noise) disproportionately affects SSL, emphasizing the need for high-quality training images.
• Task Granularity: SSL performs well for coarse-grained classification tasks but lags behind supervised methods for fine-grained tasks, potentially due to the generic nature of features learned via contrastive loss.

Implications for Clay:

1. We do not use contrastive learning, but MAE. It is not clear if MAE suffer similar issues but worth keeping in mind that ever larger dataset might not be needed, specially when Earth could be considered a semantically sparse domain (compared to random images or text, and defined as the % of the hyperspace with data).
2. This might be more relevant for the earth2text model, where we use contrastive losses. It might be mostly good news in that we might not need to train a "whole world" of OSM tags.