Self-Supervised Vision

The InfoNCE contrastive loss for a positive pair $(z_i, z_i^+)$ with $N-1$ negative pairs $\{z_j^-\}$ is:

$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(z_i, z_i^+) / \tau)}{\exp(\text{sim}(z_i, z_i^+) / \tau) + \sum_{j=1}^{N-1} \exp(\text{sim}(z_i, z_j^-) / \tau)}$

where $\text{sim}(a, b) = a^\top b / (\|a\| \|b\|)$ is cosine similarity and $\tau > 0$ is a temperature parameter.

InfoNCE is a lower bound on the mutual information between the two views: $I(z_i; z_i^+) \geq \log N - \mathcal{L}_{\text{InfoNCE}}$. Minimizing the loss maximizes a lower bound on mutual information.

InfoNCE is a softmax classifier that tries to identify the positive pair among $N$ candidates. If the model produces embeddings where the positive pair is more similar than all negative pairs, the loss is low. The temperature $\tau$ controls the sharpness: small $\tau$ makes the model focus on hard negatives (pairs that are similar but should be pushed apart); large $\tau$ treats all negatives more equally.

The mutual information interpretation means the model learns representations that preserve information shared between the two views. since the views differ only by augmentation, this shared information is the semantic content of the image.

InfoNCE is the foundational objective for contrastive self-supervised learning. SimCLR, MoCo, and CLIP all use variants of this loss. Understanding it reveals why contrastive methods need: (1) strong augmentations to create informative positive pairs, (2) many negatives for a tight MI bound, and (3) careful temperature tuning.

The MI bound is loose when $N$ is small (few negatives). With $N = 2$, the bound is at most $\log 2 \approx 0.69$ bits regardless of the true MI. This is why contrastive methods benefit from large batch sizes (SimCLR) or memory banks (MoCo). Also, the method can learn shortcuts: if the augmentations are too weak, the model can match views based on low-level statistics (color histograms) rather than semantic content.

DINO trains a student $f_{\theta_s}$ to match a teacher $f_{\theta_t}$ via cross-entropy on softmax outputs. Without safeguards, this objective has trivial solutions (collapse): the teacher outputs a uniform distribution or a constant.

DINO prevents collapse through two mechanisms:

1. Centering: subtract the running mean $c$ of teacher outputs before the softmax: $P_t(x) = \text{softmax}((f_{\theta_t}(x) - c) / \tau_t)$. This prevents collapse to a single dominant dimension.
2. Sharpening: use a low teacher temperature $\tau_t < \tau_s$, making the teacher distribution peaked. This prevents collapse to the uniform distribution.

The teacher is updated via EMA: $\theta_t \leftarrow \lambda\theta_t + (1 - \lambda)\theta_s$ with $\lambda$ close to 1 (e.g., 0.996-0.9995).

Without centering, the teacher could collapse to always outputting the same vector for every image. The student would trivially match this by also outputting a constant. Centering forces the mean output to be zero, so different images must produce different (centered) representations.

Without sharpening, the teacher could output a uniform distribution over all dimensions. The student would trivially match this without learning anything. Low temperature forces the teacher distribution to be peaked, requiring the student to identify which dimensions are most active for each image.

Together, centering and sharpening ensure the teacher provides non-trivial, image-specific targets.

DINO showed that self-supervised ViTs learn remarkable emergent properties: the attention maps of the [CLS] token segment objects in images without any segmentation supervision. This suggests that object-level understanding emerges naturally when the model is trained to produce consistent representations across crops. DINOv2 scaled this approach with curated data and distillation, producing features that serve as universal vision backbones.

The centering mechanism uses an exponential moving average of teacher outputs, which means it adapts slowly. If the distribution of images in a batch is systematically different from previous batches (e.g., due to a non-random data loader), centering can lag behind and fail to prevent collapse temporarily. The multi-crop strategy (student sees local crops, teacher sees global crops) is critical: removing it significantly degrades performance because the student no longer needs to infer global structure from local information.

MAE (Masked Autoencoder, He et al. 2022) masks a large fraction (75%) of image patches and trains the model to reconstruct them:

1. Partition the image into patches $\{p_1, \ldots, p_N\}$
2. Randomly mask 75% of patches: visible set $\mathcal{V}$, masked set $\mathcal{M}$
3. Encode only the visible patches with a ViT encoder: $\{h_i\}_{i \in \mathcal{V}} = f_\text{enc}(\{p_i\}_{i \in \mathcal{V}})$
4. Decode all patches (visible + mask tokens) with a lightweight decoder
5. Reconstruction loss is computed only on masked patches:

$\mathcal{L}_{\text{MAE}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \|p_i - \hat{p}_i\|^2$

where $\hat{p}_i$ is the reconstructed patch (in pixel space or normalized pixel space).

MAE is the visual analogue of masked language modeling (BERT). By hiding 75% of the image, the model must understand spatial layout, object structure, and texture patterns to fill in the missing pieces. The high masking ratio is critical: if only 10% is masked, the model can "cheat" by interpolating from nearby visible patches without deep understanding. At 75%, the visible patches are sparse enough that reconstruction requires genuine visual reasoning.

The asymmetric design (heavy encoder on visible patches only, lightweight decoder on all patches) makes training efficient: the encoder processes only 25% of patches, giving a $4\times$ speedup over processing all patches.

MAE demonstrated that masked modeling, hugely successful for language (BERT, GPT), also works for vision. It produces strong features with efficient training and shows that ViTs can learn useful representations from reconstruction alone. The 75% masking ratio was a surprising finding. It suggested that images have much higher redundancy than text, requiring more aggressive masking to create a challenging pretext task.

MAE features are weaker than contrastive or self-distillation features for linear probing (using frozen features with a linear classifier). The reconstruction objective encourages the model to retain low-level details (textures, colors) that are useful for reconstruction but less useful for semantic classification. Fine-tuning MAE features on downstream tasks works well, but the frozen features underperform DINO/DINOv2. This suggests that reconstruction and discrimination learn different aspects of visual representation.