feat: Onboard NOAA dataset #158

nlarge-google · 2021-08-31T14:05:01Z

Description

Based on #

Note: It's recommended to open an issue first for context and discussion.

Note: Delete items below that aren't applicable to your pull request.

Please merge this PR for me once it is approved.
If this PR adds or edits a dataset or pipeline, it was reviewed and approved by the Google Cloud Public Datasets team beforehand.
If this PR adds or edits a dataset or pipeline, I put all my code inside datasets/noaa and nothing outside of that directory.
This PR is appropriately labeled.

datasets/noaa/_images/run_csv_transform_kub_gsod_stations/csv_transform.py

datasets/noaa/_images/run_csv_transform_kub_lightning_strikes_by_year/csv_transform.py

datasets/noaa/lightning_strikes_by_year/pipeline.yaml

adlersantos

Looks good!

nlarge-google added 5 commits August 23, 2021 17:30

image scripts

92eea41

feat: noaa dataset pipelines

ee09974

fix: cleaned up csv_transform.py

dd01dd7

fix: resolved error in conversion with datatype - truncation

2b2af3b

fix: first successful fully tested version

6b57978

google-cla bot added the cla: yes label Aug 31, 2021

nlarge-google added 3 commits August 31, 2021 14:11

fix: resolved pod task id length

770192e

fix: reduced size of dag id

76ddfb2

fix: removed import pdb

f806278

adlersantos self-requested a review August 31, 2021 14:33

adlersantos requested changes Aug 31, 2021

View reviewed changes

adlersantos added the data onboarding Onboard a dataset or submit a pipeline label Aug 31, 2021

nlarge-google added 2 commits August 31, 2021 19:32

fix: resolved as per change request

187161f

fix: resolved flake 8

1c0941f

nlarge-google requested a review from adlersantos September 1, 2021 14:29

adlersantos approved these changes Sep 1, 2021

View reviewed changes

adlersantos merged commit 8371856 into GoogleCloudPlatform:main Sep 1, 2021