Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Onboard New York's 311_Service_Requests, Citibike_Stations, Tree_Census_1995 datasets #167

Merged
merged 24 commits into from Oct 6, 2021
Merged

feat: Onboard New York's 311_Service_Requests, Citibike_Stations, Tree_Census_1995 datasets #167

merged 24 commits into from Oct 6, 2021

Commits on Aug 24, 2021

  1. feat: new_york branch

    @nlarge-google
    nlarge-google committed Aug 24, 2021
    Configuration menu
    Copy the full SHA
    8e551e1 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2021

  1. feat: NY 311_service_Requests, NY Citibike_Stations, NY Tree_Census_1995

    @nlarge-google
    nlarge-google committed Aug 26, 2021
    Configuration menu
    Copy the full SHA
    95a97f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2021

  1. fix: resolved multiple pipelines in same dataset.

    @nlarge-google
    nlarge-google committed Aug 27, 2021
    Configuration menu
    Copy the full SHA
    d340c0d View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2021

  1. fix: NY 311 -> memory pressure causes crashes processing input CSV fi… 

    …le 15.1GB
    @nlarge-google
    nlarge-google committed Aug 29, 2021
    Configuration menu
    Copy the full SHA
    fabdc25 View commit details
    Browse the repository at this point in the history

  2. fix: NY 311 -> memory pressure causes crashes processing input CSV fi… 

    …le 15.1GB -> added pipeline.yaml change
    @nlarge-google
    nlarge-google committed Aug 29, 2021
    Configuration menu
    Copy the full SHA
    aa1e9d2 View commit details
    Browse the repository at this point in the history

  3. fix: NY 311 -> memory pressure causes crashes processing input CSV fi… 

    …le 15.1GB -> unremarked download_file required for production
    @nlarge-google
    nlarge-google committed Aug 29, 2021
    Configuration menu
    Copy the full SHA
    95f0030 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2021

  1. fix: NY 311 -> currently tested successfully locally

    @nlarge-google
    nlarge-google committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    5bce9ef View commit details
    Browse the repository at this point in the history

  2. fix: NY 311 -> unremarked download file

    @nlarge-google
    nlarge-google committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    5c10e88 View commit details
    Browse the repository at this point in the history

  3. fix: NY 311 -> resolved variable type issue

    @nlarge-google
    nlarge-google committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    818cf51 View commit details
    Browse the repository at this point in the history

  4. fix: NY 311 -> modified number of batches for execution

    @nlarge-google
    nlarge-google committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    d067299 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2021

  1. fix: resolved flake8

    @nlarge-google
    nlarge-google committed Sep 7, 2021
    Configuration menu
    Copy the full SHA
    0bd9207 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2021

  1. fix: fixed yamllint issues not raised by pre-commit

    @nlarge-google
    nlarge-google committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    85f875e View commit details
    Browse the repository at this point in the history

  2. fix: resolved flake8 issues not identified by pre-commit

    @nlarge-google
    nlarge-google committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    a6394c0 View commit details
    Browse the repository at this point in the history

  3. fix: resolved flake8 issues not identified by pre-commit #2

    @nlarge-google
    nlarge-google committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    c585e23 View commit details
    Browse the repository at this point in the history

  4. fix: attempt to fix issues pertaining to pod not starting and also ad… 

    …ding start timeout to pipeline yaml
    @nlarge-google
    nlarge-google committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    71c19ba View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2021

  1. fix: clean up and replace batch process in 311.

    @nlarge-google
    nlarge-google committed Sep 16, 2021
    Configuration menu
    Copy the full SHA
    15d29c1 View commit details
    Browse the repository at this point in the history

  2. fix: resolved isort issue

    @nlarge-google
    nlarge-google committed Sep 16, 2021
    Configuration menu
    Copy the full SHA
    88961fb View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2021

  1. fix: Incrementing amount of CPUs allocated to BQ task in order to red… 

    …uce possibility of crash
    @nlarge-google
    nlarge-google committed Sep 21, 2021
    Configuration menu
    Copy the full SHA
    f0fd762 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2021

  1. fix: Runs as expected in Airflow 2

    @nlarge-google
    nlarge-google committed Sep 27, 2021
    Configuration menu
    Copy the full SHA
    2663c8a View commit details
    Browse the repository at this point in the history

  2. fix: Resolved flake8 issues

    @nlarge-google
    nlarge-google committed Sep 27, 2021
    Configuration menu
    Copy the full SHA
    fb727bd View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2021

  1. fix: Refactored code. Testing in Airflow shows the following issues -… 

    … 1) csv_transform runs as intended but shows log read interrupted at one point which is most likely a network issue. 2) load to BQ error ... 400 Error while reading data, error message: CSV table references column position 41, but line starting at position:14686524020 contains only 41 columns.
    @nlarge-google
    nlarge-google committed Sep 28, 2021
    Configuration menu
    Copy the full SHA
    3575ceb View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2021

  1. fix: resolve dag errors and clean up code

    @nlarge-google
    nlarge-google committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    0018e9f View commit details
    Browse the repository at this point in the history

  2. fix: resolve isort issue

    @nlarge-google
    nlarge-google committed Sep 30, 2021
    Configuration menu
    Copy the full SHA
    30568ed View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2021

  1. fix: Resolved issues in previous code review, improved code and all r… 

    …un as intended in AF2
    @nlarge-google
    nlarge-google committed Oct 1, 2021
    Configuration menu
    Copy the full SHA
    40a79ba View commit details
    Browse the repository at this point in the history