Python for Earth Observation (PyEO)

PyEO is designed to provide a set of portable, extensible and modular Python scripts for machine learning in earth observation and GIS, including downloading, preprocessing, creation of baseline composites, classification and validation.

Full documentation is available at https://clcr.github.io/pyeo/build/html/index.html

Example notebooks are available at:

• https://github.com/clcr/pyeo_1/tree/main/notebooks
• https://github.com/clcr/pyeo_training_materials

Requirements

Python library requirements are categorised by Platform (Operating System - OS). For use in the Cloud Processing platform SEPAL - pyeo is already installed in a virtual environment. SEPAL is a cloud computing platform for geospatial data which offers remote Linux Instances that are customised for performing geospatial analysis in R or Python. More information can be found here: https://github.com/openforis/sepal
.

Package management is performed by Conda, for instructions on how to install Conda, please refer to: https://docs.conda.io/en/latest/.
Note: Conda can be installed as part of Anaconda https://www.anaconda.com/

For installation locally on an OS of your choice, see the sections below.

To install pyeo_1, put the following commands into Bash (Linux), Terminal (Mac) or the Anaconda Prompt (Windows)

Ubuntu or MacOS

conda install -c conda-forge git
git clone https://github.com/clcr/pyeo_1.git
cd pyeo_1
conda env create --file environment.yml --name pyeo_env
conda activate pyeo_env
python -m pip install -e .

If you do not want to edit pyeo_1, replace python -m pip install -e . line with

python -m pip install -vv .

Windows

conda install -c conda-forge git
git clone https://github.com/clcr/pyeo_1.git
cd pyeo_1
conda env create --file environment_windows.yml --name pyeo_env
conda activate pyeo_env
python -m pip install -e .

If you do not want to edit pyeo_1, replace python -m pip install -e . line with

python -m pip install -vv .

A Note on .ini file encoding on Windows

If the OS that pyeo is running on is Windows, we have noticed that pyeo_windows.ini may need to be saved with ANSI encoding instead of the usual UTF-8. See this webpage for more details.

Satellite Imagery Providers

From July 2023, Scihub will be deprecated in favour of the Copernicus Data Space Ecosystem (CDSE). In the meantime, if you wish to download from Scihub, you will need a Scihub account: https://scihub.copernicus.eu/

To use the CDSE, you will need a separate account: https://dataspace.copernicus.eu

Once you have created your account, you will need to enter your email address and password into the credentials_dummy.ini file in the folder credentials, like this:

[sent_2]
user=replace_this_with_your_email_address
pass=replace_this_with_your_password

[dataspace]
user=replace_this_with_your_email_address
pass=replace_this_with_your_password

Where user and pass under [sent_2] correspond to your scihub account details, and user and pass under [dataspace] correspond to your dataspace account details.

To process Sentinel-2 L1Cs, you will also need Sen2Cor installed: http://step.esa.int/main/third-party-plugins-2/sen2cor/. This installation process is covered in the PyEO_I_Setup.ipynb notebook, available from the notebooks folder.

Installation Test Steps

You can test your installation with by typing the following in Bash/Terminal/Anaconda Prompt:

python

import pyeo_1.classification

or, by running the same import command above, after having started a jupyter notebook via:

jupyter notebook

Please note, if you are using SEPAL, jupyter notebooks have to be started via a GUI method instead of from Bash, see: https://user-images.githubusercontent.com/149204/132491851-5ac0303f-1064-4e12-9627-f34e3f78d880.png

How PyEO works

PyEO operates across two stages:

1. Across Sentinel-2 (S2) tiles
2. Within individual S2 tiles

1. Takes a Region of Interest (ROI) and calculates which Sentinel-2 (S2) tiles overlap with the ROI
2. Builds a Baseline Composite to compare land cover changes against, by downloading S2 images and calculating the median of these images.
3. Downloads images over the Change Period
4. Classifies the Composite and the Change images using a classifier in ./models/
5. Calculates the change between the from classes and the to classes, for each classified image. This could be changes from forest to bare soil.
6. Creates a Change Report describing the consistency of the class changes, highlighting the changes that PyEO is confident.
7. Vectorises the Change Report and removes any changes outside of the ROI

How to Run PyEO

PyEO can be run interactively in the Jupyter Notebooks provided in the Tutorials, but the pipeline method can be run via the Terminal. This process is automated and is suited to the advanced python user.
Both the terminal and notebook methods rely on an a configuration file (e.g. pyeo_linux.ini, pyeo_windows.ini, pyeo_sepal.ini) to make processing decisions.
The below example references pyeo_sepal.ini, but this can be switched for the Linux or Windows equivalent.

1. First, move to where PyEO is installed:

cd pyeo_1

2. Now, the pipeline method runs like this. Here we are telling the terminal that we want to invoke python to run the script run_acd_national.py within the folder pyeo_1, then we pass the absolute path to the initialisation file for your OS. The script run_acd_national.py requires this path as all the processing parameters are stored in the initialisation file. See below:

python pyeo_1/run_acd_national.py <insert_your_absolute_path_to>/pyeo_sepal.ini

The pipeline uses arguments specified in the .ini file (short for initialisation), to decide what processes to run. Here, we will go through the sections of the ini file and what arguments do what.

[forest_sentinel]
# Acquisition dates for Images in the Change Period, in the form yyyymmdd
start_date=20230101
end_date=20230611

# Acquisition dates for Images for the Baseline Composite, in the form yyyymmdd
composite_start=20220101
composite_end=20221231

# EPSG code, for example - Kenya. This epsg is for areas North of equator and East of 36°E is EPSG:21097
# See https://epsg.io/21097 and https://spatialreference.org/ref/epsg/21097/
epsg=21097

# Cloud cover threshold for imagery to download
cloud_cover=25

# Certainty value above which a pixel is considered a cloud from sen2cor
cloud_certainty_threshold=0

# path to the trained machine learning model for land cover in Kenya
model= ./models/model_36MYE_Unoptimised_20230505_no_haze.pkl

Automated Pipeline Execution

To enable parallel processing of the raster and vector processing pipelines with the do_parallel = True option enabled in pyeo_sepal.ini, make the following file an executable by issuing this command:

cd pyeo_1/apps/automation/
chmod u+x automate_launch.sh

Tutorials

Once installation of pyeo_1 is complete, you can follow the tutorial notebooks, which demonstrate the utility of pyeo_1.

How to Train Your Classifier: https://github.com/clcr/pyeo_1/blob/main/notebooks/PyEO_I_Model_Training.ipynb

Downloading Sentinel-2 Imagery, Creating a Baseline Composite, Performing Automatic Change Detection: https://github.com/clcr/pyeo_1/blob/main/notebooks/PyEO_I_Master_Tutorials.ipynb

How to cite this software

Please use the following references when using pyeo:

Roberts, J.F., Mwangi, R., Mukabi, F., Njui, J., Nzioka, K., Ndambiri, J.K., Bispo, P.C., Espirito-Santo, F.D.B., Gou, Y., Johnson, S.C.M. and Louis, V., 2022. Pyeo: A Python package for near-real-time forest cover change detection from Earth observation using machine learning. Computers & Geosciences, 167, p.105192.

Roberts, J., Balzter, H., Gou, Y., Louis, V., Robb, C., 2020. Pyeo: Automated satellite imagery processing. https://doi.org/10.5281/zenodo.3689674

Pacheco-Pascagaza, A.M., Gou, Y., Louis, V., Roberts, J.F., Rodríguez-Veiga, P., da Conceição Bispo, P., Espírito-Santo, F.D., Robb, C., Upton, C., Galindo, G. and Cabrera, E., 2022. Near real-time change detection system using Sentinel-2 and machine learning: A test for Mexican and Colombian forests. Remote Sensing, 14(3), p.707.