Image Value Scaling/Preprocessing For Pretrained Models

[kaybe20]

Hi,
I'm trying to implement one of the new pretrained LandSat Models. More specifically the LANDSAT_ETM_SR models that were pretrained on the SSL4EO-L dataset.
I read the paper (https://arxiv.org/abs/2306.09424) but for me It's hard to understand the preprocessing for the image values.
It says:
"The official scale factors suggested by the USGS to map between Level-1 and Level-2 Landsat imagery and the visualization range
recommended by GEE for each sensor are used to map from float32 to uint8"

From my understanding this is then the resulting preprocessing step, right? (GEE example)

var image = image.multiply(0.0000275).add(-0.2)
var image8Bit = image.multiply(256).uint8()

The recommended visualization range for the image after applying the scale factor and the offset (so for the "var image...") is
(min: 0, max: 0.3). Is this also taken into account during preprocessing in GEE?
I would be glad if somebody could help me. Is there maybe a ressource where I can find the necessary scaling/preprocessing steps for the corresponding models, that I am missing?

Another question that maybe rises from my lack of understanding of ViTransformers is:
The image sample size, as stated in paper, should be 264 × 264 px. Why does the ViT require 224x224 images?

[adamjstewart]

From my understanding this is then the resulting preprocessing step, right?

Yes. This is where the scale factors come from.

Is this also taken into account during preprocessing in GEE?

This is the script used to map from float32 to uint8. We download the data first and preprocess later. Of course, you could also do the preprocessing on GEE itself.

Why does the ViT require 224x224 images?

There are many ViT models available. The specific ones we trained are designed for 224x224 px patches. We need patches that are larger than that in order to perform random cropping.

Hope this helps! Let me know if there are any other questions I can answer.

[kaybe20]

Thank you, thats exactly what I was searching for :)

[kaybe20]

Hi, I stumbled upon another detail, that I struggle to understand,
I've been trying to replicate the CDL Image Segmentation Experiment with the SSL4EO-L Benchmark Dataset.
The exact setup being, UNet + ResNet-18 backbone (LANDSAT_OLI_SR_MOCO).
In the paper and the Lightning CLI config, it says that due to the class imbalance in the CDL dataset, only classes with > 1% area are considered, which results in 18 classes instead of the original 134 of the CDL dataset.

I struggle to find the point at which the 134 classes are reduced to 18 classes, is it during evaluation or preprocessing, or what am I missing?
Would it be possible to download the weights for UNet decoder fine tuned on CDL Image Segmentation somewhere?
I would be very glad if you could point me in the right direction :)

[adamjstewart]

You'll want to use a config file like this. The 134 classes are reduced to 18 during preprocessing, and the exact classes used are listed in that file for reproducibility.

@AABNassim @nilsleh do we have the U-Net decoder weights saved somewhere?

[kaybe20]

Thanks, I found it now. The corresponding preprocessing step I was looking for, is in the SSL4EOLBenchmark Dataset class, I missed that :)

[adamjstewart]

Yes, all unknown data module kwargs are passed to the dataset. Should have mentioned that.