Credential YAML

This YAML file will be passed to the --envpth argument to be able to download MODIS/VIIRS/Sentinel-2 data. This file needs to be located inside the mounted volume store in order for the internal processes to access the credentials.

--envpth /data/<credential>.yml

YAML

# register at https://urs.earthdata.nasa.gov/home
EARTHDATA_USER: username_without_quotes
EARTHDATA_PASS: password_without_quotes

# register at https://scihub.copernicus.eu/dhus/#/self-registration 
SENTINELSAT_USER: username_without_quotes
SENTINELSAT_PASS: password_without_quotes

MODIS/VIIRS Pipeline

Docker

Building Docker Image

To build the image from this directory:

docker build -t <tagname> .

Running Docker Image

To run the Docker image use the following schema:

docker run --rm -v <local store>:/data <tagname> <extra_commands>

Input Formats

ARG	VALUES	TYPE
envpth	Linux path	string
date	YYYY.MM.DD	string
sat	modis / viirs / sentinel	string
typ	watersheds / basins	string
days	1 / 5 / 8	int

Docker Options

OPTION	CAUSE
--rm	Removes container once it finishes running
-v	Mount volume to Docker container

Help

To list out commands available:

The default CMD is "--help" to list out the available commands.

docker run --rm -v <mount_point>:/data <tag_name>

FAQ

Why is there an authentication error when downloading MODIS or VIIRS data?

Answer

A HTML 503 Authentication Error is an expected behaviour from the NASA servers while downloading large amounts of data. Be patient as the download will retry and continue once the service is made available again.

Daily-Pipeline

docker run --rm -v <mount_point>:/data <tag_name> daily-pipeline --envpth /data/<creds.yml> --date <target_date: YYYY.MM.DD>

daily-pipeline kicks off the entire process chain that will include performing the following per satellite (MODIS/VIIRS):

Build up directory structure and supporting files
Download raw granule files
Process raw granules to GTiff formats
Calculate snow coverage per watersheds/basin
Build KML of watersheds/basins
Clean up intermediate files

Build Directory Structure

Builds necessary supporting files and directories in order for the process pipeline to properly manage file I/O.

docker run --rm -v <mount_point>:/data <tag_name> build

High Level Directory Structure:

/data
- /analysis
- /basins
- /intermediate_kml
- /intermediate_tif
- /kml
- /modis-terra
- /output_tif
- /plot
- /watersheds

Download

MODIS requires 5 or 8 days in order to build a composite of valid data. Option to download one day is possible. VIIRS will download a single day as it is a cloud-gap-filled product.

docker run --rm -v <mount_point>:/data <tag_name> download --envpth /data/<creds.yml> --sat <modis/viirs> --date <YYYY.MM.DD> --days <1/5/8>

Output:

Raw granules: modis-terra/<product>/<date>/.
- MODIS: MOD10A1.006
- VIIRS: VNP10A1F.001

Process

MODIS requires 5 or 8 days in order to build a composite of valid data. Default value is 5 days.

docker run --rm -v <mount_point>:/data <tag_name> process --sat <modis/viirs> --date <YYYY.MM.DD> --days <1/5/8>

Output:

Clipped watershed/basin GTiff: <watershed/basin>/<name>/<satellite>/<date>/.
- EPSG:4326 -> needed for KML
- EPSG:3153 -> BC Albers projection GTiff

Caclulate Snow Coverage

Analyze each watershed and basin to calculate the snow coverage based on the NDSI value.

docker run --rm -v <mount_point>:/data <tag_name> run-analysis --typ <watersheds/basins> -sat <modis/viirs> --date <YYYY.MM.DD>

Output:

SQLITE3 database: analysis/analysis.db

Database To CSV

Convert the SQLITE3 database into a CSV

docker run --rm -v <mount_point>:/data <tag_name> dbtocsv

Output:

CSV: analysis/analysis.csv

Build KMLs and Colour Ramped GTiffs

Build the colour-ramp GTiff and KML versions of the watersheds/basins.

docker run --rm -v <mount_point>:/data <tag_name> build-kml --date <YYYY.MM.DD> --typ <watersheds/basins> --sat <modis/viirs>

Output :

colourized GTiffs: <watersheds/basins>/<name>/<satellite>/<date>/.
KML : kml/<date>/

Compose KMLs

Compose built KML files into a heirarchal KML

docker run --rm -v <mount_point>:/data <tag_name> compose-kmls --date <YYYY.MM.DD> --sat <modis/viirs>

Zip KMLs

ZIP KMLs into a ZIP file

docker run --rm -v <mount_point>:/data <tag_name> zip-kmls

KNOWN ISSUE: ZIP file is larger than original KMLs -- deprecated

Plot

Plot all watersheds and basins into PNG plots with mapped colour bar.

docker run --rm -v <mount_point>:/data <tag_name> plot --date <YYYY.MM.DD> --sat <modis/viirs>

Clean

Manually clean up files and directories.

TARGET	CAUSE
all	Non-vital directories in `/data`
intermediate	Intermediate files in `intermediate_\[tif/kml\]`
output	Output files in `output_tif`
downloads	Raw granules in `modis-terra/`
watersheds	All files/dirs in `watersheds/`
basins	All files/dirs in `basins/`

docker run --rm -v <mount_point>:/data <tag_name> clean --target <target>

Sentinel-2 Pipeline

Docker

Building Docker Image

To build the image from this directory:

docker build -t <tagname> .

Running Docker Image

To run the Docker image use the following schema:

docker run --rm -it -v <local store>:/data <tagname> <extra_commands>

It is necessary to run the docker container in interactive mode (by including the -it option when calling docker run) as the Sentinel-2 process requires user interaction.

Process-Sentinel

Call the Sentinel-2 pipeline

docker run --rm -it -v <mount point>:/data <tag name> process-sentinel --creds /data/<creds.yml> --lat <latitude> --lng <longtitude> --date <YYYY.MM.DD>

The pipeline will return a date ordered list of 10

OPTIONAL ARGS	VALUES	DEFAULT
--rgb	true / false	false
--max-allowable-cloud	int	50
--force-download	true / false	false
--day-tolerance	int	50

Outputs are logged to a log file in /data/log/.

Argument Details:

Latitude and Longtitude are WSG84 float numbers (-180 <= lat/lng => +180).
--rgb true : Creates a RGB output GTiff of the selected tile
--max-allowable-cloud <int> : Max percentage of cloud cover that is allowable in the query. Default is 50%.
--force-download true : Deletes existing downloads in target directory
--day-tolerance <int> : Number of days to look back from target date with given cloud expections. Default is 50 days.

Example 0

Using default arguments to demonstrate expected output.

docker run --rm -it -v <mount point>:/data <tag name> process-sentinel --creds /data/sat.yml --lat 49.12 --lng -126.5  --date 2021.03.18

0 : DATE: 2021-03-17 || CLOUD%:  45.588648
1 : DATE: 2021-03-14 || CLOUD%:  17.49127
2 : DATE: 2021-03-07 || CLOUD%:  39.499657
3 : DATE: 2021-02-05 || CLOUD%:  16.117952
4 : DATE: 2021-01-31 || CLOUD%:  23.281466
5 : DATE: 2021-01-28 || CLOUD%:  2.606739
Pick which product to download and process [0-5/n]:

Example 1

Calling with high cloud tolerance and demonstrate upper limit of selection and typical output.

docker run --rm -it -v <mount point>:/data <tag name> process-sentinel --creds /data/sat.yml --lat 49.73 --lng -126.5  --date 2021.03.18 --rgb true --max-allowable-cloud 90

0 : DATE: 2021-03-17 || CLOUD%:  32.746968
1 : DATE: 2021-03-14 || CLOUD%:  87.006531
2 : DATE: 2021-03-09 || CLOUD%:  45.446411
3 : DATE: 2021-03-07 || CLOUD%:  74.669036
4 : DATE: 2021-02-25 || CLOUD%:  84.000973
5 : DATE: 2021-02-22 || CLOUD%:  79.523277
6 : DATE: 2021-02-17 || CLOUD%:  56.401063
7 : DATE: 2021-02-15 || CLOUD%:  74.093911
8 : DATE: 2021-02-10 || CLOUD%:  32.880395
9 : DATE: 2021-02-07 || CLOUD%:  75.293885
Pick which product to download and process [0-9/n]: 0

Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.09G/1.09G [03:30<00:00, 5.19MB/s]MD5 checksumming: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.09G/1.09G [00:04<00:00, 231MB/s]
0

Notice the clouds and water are masked out form the analysis.

Example 2

Calling with low cloud tolerance to demonstrate limited selection.

docker run --rm -it -v <mount point>:/data <tag name> process-sentinel --creds /data/sat.yml --lat 49.12 --lng -126.5  --date 2021.03.18 --max-allowable-cloud 20 --max-allowable-cloud 20

0 : DATE: 2021-03-14 || CLOUD%:  17.49127
1 : DATE: 2021-02-05 || CLOUD%:  16.117952
2 : DATE: 2021-01-28 || CLOUD%:  2.606739
Pick which product to download and process [0-2/n]:

Name		Name	Last commit message	Last commit date
Latest commit History 154 Commits
.github/workflows		.github/workflows
.vscode		.vscode
admin		admin
analysis		analysis
aoi		aoi
cicd		cicd
config		config
docs		docs
download_granules		download_granules
hatfieldcmr		hatfieldcmr
imgs		imgs
process		process
snowpack_archive		snowpack_archive
tests		tests
.dockerignore		.dockerignore
.flake8		.flake8
.gitignore		.gitignore
COMPLIANCE.yaml		COMPLIANCE.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
build.jenkins		build.jenkins
calculate_norm.py		calculate_norm.py
demo.jenkins		demo.jenkins
dockerfile.hatfield		dockerfile.hatfield
env2.yaml		env2.yaml
environment.yaml		environment.yaml
explicit.lock		explicit.lock
explicit_junk.lock		explicit_junk.lock
getDate.py		getDate.py
get_available_data.py		get_available_data.py
process.jenkins		process.jenkins
pytest.ini		pytest.ini
renovate.json		renovate.json
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
run.py		run.py
setup.py		setup.py
template.yml		template.yml
test_data.json		test_data.json

License

bcgov/nr-rfc-processing

Folders and files

Latest commit

History

Repository files navigation

Credential YAML

MODIS/VIIRS Pipeline

Docker

Building Docker Image

Running Docker Image

Input Formats

Docker Options

Help

FAQ

Daily-Pipeline

Build Directory Structure

Download

Process

Caclulate Snow Coverage

Database To CSV

Build KMLs and Colour Ramped GTiffs

Compose KMLs

Zip KMLs

Plot

Clean

Sentinel-2 Pipeline

Docker

Building Docker Image

Running Docker Image

Process-Sentinel

Argument Details:

Example 0

Example 1

Example 2

About

Topics

Resources

License

Stars

Watchers

Forks

Languages