slides.qmd

---
title: "Data Strategies for Future Us"
author: "Andy Barrett,\n National Snow and Ice Data Center"
date: last-modified
date-format: iso
format:
    revealjs:
        colorlinks: true
        theme: night
        slide-number: c/t
# Handle redirect from the old URL
aliases:
  - "data_strategies_slides.html"
---

## What are Data Strategies?

Data Strategies enhance collaboration and reproducible science.


 - Workflows;  
 - Data management best practices;  
 - Documentation;  


Good to start from the beginning of a project, great to start
from where you are now.


## Who is future us?

:::: {.columns}  
::: {.column}  
You.

Your team.

The scientific community.
:::  

::: {.column}
![](images/horst-starwars-teamwork.png)
:::
::::

## A simple data workflow
![http://r4ds.hadley.nz/](images/wickham-data-science-workflow.png)


## When to cloud?

 - What is the data volume?
 - How long will it take to download?
 - Can you store all that data (cost and space)?  
 - Do you have the computing power for processing?  
 - Does your team need a common computing environment?  
 - Do you need to share data at each step or just an end product?  


## Workflow Solutions

### Local

```{dot}
digraph G {
	fontname="Helvetica,Arial,sans-serif"
	node [fontname="Helvetica,Arial,sans-serif"]
	edge [fontname="Helvetica,Arial,sans-serif"]

	communicate [shape=plaintext, label="Communicate"];
        cloud [shape=parallelogram, label="Earthdata Cloud"];
        daac [shape=cylinder, label="DAAC"]

        subgraph cluster_incloud {
                style=filled;
                color=lightgrey;
                cloud;
                label="In Cloud";
         }

        subgraph cluster_0 {
		style=filled;
		color=lightgrey;
		node [shape=plaintext, style=filled,color=white];
		Tidy -> Transform -> Visualize -> Model -> Transform;
		label = "Local Machine";
	}

        daac -> Tidy
        cloud -> Tidy
        Model -> communicate;
        
}
```

## Workflow Solutions

### Hybrid

```{dot}
digraph G {
	fontname="Helvetica,Arial,sans-serif"
	node [fontname="Helvetica,Arial,sans-serif"]
	edge [fontname="Helvetica,Arial,sans-serif"]

	communicate [shape=plaintext, label="Communicate"];
        cloud [shape=parallelogram, label="Earthdata Cloud"];
        daac [shape=cylinder, label="DAAC"]

        subgraph cluster_incloud {
                style=filled;
                color=lightgrey;
		tidy_cloud [shape=plaintext, label="Tidy", style=filled,color=white];
                cloud -> tidy_cloud;
                label="In Cloud";
        }
        
        subgraph cluster_local {
		style=filled;
		color=lightgrey;
		node [shape=plaintext, style=filled,color=white];
                tidy_local [shape=plaintext, label="Tidy", style=filled,color=white];
		tidy_local -> Transform -> Visualize -> Model -> Transform;
		label = "Local Machine";
	}

        daac -> tidy_local;
        tidy_cloud -> Transform
        Model -> communicate;
        
}
```

## Workflow Solutions

### All in cloud

```{dot}
digraph G {
	fontname="Helvetica,Arial,sans-serif"
	node [fontname="Helvetica,Arial,sans-serif"]
	edge [fontname="Helvetica,Arial,sans-serif"]

	communicate [shape=plaintext, label="Communicate"];
        cloud [shape=parallelogram, label="Earthdata Cloud"];
        daac [shape=cylinder, label="DAAC"]

        subgraph cluster_incloud {
                style=filled;
                color=lightgrey;
		tidy_cloud [shape=plaintext, label="Tidy", style=filled,color=white];
		node [shape=plaintext, style=filled,color=white];
		cloud -> tidy_cloud -> Transform -> Visualize -> Model -> Transform;
                label="In Cloud";
        }
        
        daac -> tidy_cloud;
        Model -> communicate;
        
}
```

## Workflow Solutions

### Use Cloud-based Services

 - Cloud services are infrastructure, platforms and software hosted in
the cloud and made available to users via an API, often accessed via
a web interface. 

 - NASA's Harmony (https://harmony.earthdata.nasa.gov/) can subset,
reproject and reformat, and serve data.

 - This might save processing steps.


## What does this look like in the cloud?

```{.python}
import earthaccess

auth = earthaccess.login(strategy='netrc')

Query = earthaccess.granule_query().concept_id(
    'C2153572614-NSIDC_CPRD'
).temporal(
    "2020-03-01", "2020-03-30"
).bounding_box(
    -134.7,58.9,-133.9,59.2)

granules = Query.get(4)

files = earthaccess.open(granules)
ds = xr.open_dataset(files[1], group='/gt1l/land_ice_segments')
ds

# Start to do awesome science
```


## How to future-proof workflows and make them reproducible

**FAIR**

 - **F**indable,
 - **A**ccessible,
 - **I**nteroperable,
 - **R**eusable

Applies to the future you and your team as well.


## Make sure data are **F**indable and **A**ccessible

Does everyone on your team know where the data is?

Can they access it?  

Helpful to document this somewhere.


## Data Management

Keep raw data, raw!

Save intermediate data not just final versions.

Use consistent and descriptive folder and file name patterns.

```
(base) nsidc-442-abarrett:data_strategies_for_a_future_us$ tree Data
Data
├── calibrated
├── cleaned
├── figures
├── final
├── monthly_averages
├── raw
└── results

7 directories, 0 files
```

## Standard file formats make data **I**nteroperable

 - GeoTIFF for imagery or 2D raster data   
 - NetCDF for multi-dimensional data (3D+)  
 - Shapefiles or GeoJSON for vector data  
 - csv for tabular data.

Avoid Excel and other proprietary formats.


## Metadata makes data **I**nteroperable and **R**eusable.  

Metadata standards and conventions ensure that standard tools can
read/interpret the data.

Standards also define the meaning of metadata attributes.

 - What is the Coordinate Reference System?  
   (projection, grid mapping)
 - What are the units?  
 - What is the variable name?  
 - What is the source of the data?  
 - What script produced the data?  


## Document the Analysis

Document each step.

 - Where did you get the data, which files, which version?
 - Write it down.  Anywhere is good but using a script is better.

Can you (or anyone else) easily reproduce your processing pipeline?

With GUI interfaces - e.g. ArcGIS, QGIS, Excel - use screenshots, journal commands.