Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Onboard EPA historical air quality dataset #221

Merged
merged 28 commits into from Dec 8, 2021
Merged

feat: Onboard EPA historical air quality dataset #221

merged 28 commits into from Dec 8, 2021

Commits on Oct 27, 2021

  1. feat: Added annual_summaries, tested locally. Errors in AF

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    b717e9d View commit details
    Browse the repository at this point in the history

  2. feat: Added co_daily_summaries. Not ready for production

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    bb846aa View commit details
    Browse the repository at this point in the history

  3. feat: Added co_hourly_summary. Not ready for production.

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    953d1cd View commit details
    Browse the repository at this point in the history

  4. fix: Changed dataset name

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    e3966b4 View commit details
    Browse the repository at this point in the history

  5. fix: Attempt to resolve AF load_to_bq errors

    @nlarge-google
    nlarge-google committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    71a9d26 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2021

  1. fix: Resolves issues with AF failure to execute DAG, also, some datat… 

    …ype fixes.
    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    5efa0b5 View commit details
    Browse the repository at this point in the history

  2. feat: Added HAP Hourly Summary. Fixed schema issues in HAP Daily Summ… 

    …ary.
    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    7a64407 View commit details
    Browse the repository at this point in the history

  3. fix: datatype fixes

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    27d962e View commit details
    Browse the repository at this point in the history

  4. feat: Added no2 daily and hourly.

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    38d4e6a View commit details
    Browse the repository at this point in the history

  5. feat: Added NONOxNOy pipelines daily and hourly

    @nlarge-google
    nlarge-google committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    ac99d56 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2021

  1. feat: Added multiple pipelines and assigned destination table to a va… 

    …riable value for each.
    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    f4c09c8 View commit details
    Browse the repository at this point in the history

  2. fix: Added terraform files for new pipelines.

    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    9625f2d View commit details
    Browse the repository at this point in the history

  3. fix: Regenerated some dags.

    @nlarge-google
    nlarge-google committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    c2fb025 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2021

  1. fix: Resolved variable issues in pipeline.yaml files so that they ope… 

    …rate as intended in AF
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    9535847 View commit details
    Browse the repository at this point in the history

  2. fix: clean-up code

    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    54c3d7f View commit details
    Browse the repository at this point in the history

  3. fix: Reduced CHUNKSIZE in order to prevent memory outage in AF, preve… 

    …nting some pipelines to complete successfully.
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    7d93681 View commit details
    Browse the repository at this point in the history

  4. fix: Resolved incorrect path entry in ozone daily summary pipeline.ya… 

    …ml; Fixed flake8 issue in csv_transform
    @nlarge-google
    nlarge-google committed Nov 1, 2021
    Configuration menu
    Copy the full SHA
    7d0f21e View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2021

  1. fix: Requested changes as per PR code review

    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    3869b2d View commit details
    Browse the repository at this point in the history

  2. fix: Resolved black hook issue

    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    9caa687 View commit details
    Browse the repository at this point in the history

  3. fix: Reduced resources used in both lead daily summary and pressure d… 

    …aily summary
    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    642ab67 View commit details
    Browse the repository at this point in the history

  4. fix: Tiered start time for each DAG by converting start time to chron… 

    … expression
    @nlarge-google
    nlarge-google committed Nov 3, 2021
    Configuration menu
    Copy the full SHA
    4636273 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2021

  1. fix: Resolved invalid folder path in pipeline.yaml

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    6a5b3f1 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    cdd1ca7 View commit details
    Browse the repository at this point in the history

  3. fix: Removed out of date terraform file

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    9d7c753 View commit details
    Browse the repository at this point in the history

  4. fix: attempting to resolve code check issues.

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    fcb1503 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'upstream/main' into EPA

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    9e73eee View commit details
    Browse the repository at this point in the history

  6. fix: Missed one change specified in code review.

    @nlarge-google
    nlarge-google committed Nov 4, 2021
    Configuration menu
    Copy the full SHA
    34d67fa View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2021

  1. fix: Increase memory and CPU for pressure_daily_summary and reduced b… 

    …atch size in order to resolve memory pressure resulting in DAG failure.
    @nlarge-google
    nlarge-google committed Nov 29, 2021
    Configuration menu
    Copy the full SHA
    5bf3062 View commit details
    Browse the repository at this point in the history