Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Onboard New York's 311_Service_Requests, Citibike_Stations, Tree_Census_1995 datasets #167

Merged
merged 24 commits into from Oct 6, 2021
Merged

feat: Onboard New York's 311_Service_Requests, Citibike_Stations, Tree_Census_1995 datasets #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
8e551e1
feat: new_york branch
nlarge-google Aug 24, 2021
95a97f9
feat: NY 311_service_Requests, NY Citibike_Stations, NY Tree_Census_1995
nlarge-google Aug 26, 2021
d340c0d
fix: resolved multiple pipelines in same dataset.
nlarge-google Aug 27, 2021
fabdc25
fix: NY 311 -> memory pressure causes crashes processing input CSV fi…
nlarge-google Aug 29, 2021
aa1e9d2
fix: NY 311 -> memory pressure causes crashes processing input CSV fi…
nlarge-google Aug 29, 2021
95f0030
fix: NY 311 -> memory pressure causes crashes processing input CSV fi…
nlarge-google Aug 29, 2021
5bce9ef
fix: NY 311 -> currently tested successfully locally
nlarge-google Aug 30, 2021
5c10e88
fix: NY 311 -> unremarked download file
nlarge-google Aug 30, 2021
818cf51
fix: NY 311 -> resolved variable type issue
nlarge-google Aug 30, 2021
d067299
fix: NY 311 -> modified number of batches for execution
nlarge-google Aug 30, 2021
0bd9207
fix: resolved flake8
nlarge-google Sep 7, 2021
85f875e
fix: fixed yamllint issues not raised by pre-commit
nlarge-google Sep 8, 2021
a6394c0
fix: resolved flake8 issues not identified by pre-commit
nlarge-google Sep 8, 2021
c585e23
fix: resolved flake8 issues not identified by pre-commit #2
nlarge-google Sep 8, 2021
71c19ba
fix: attempt to fix issues pertaining to pod not starting and also ad…
nlarge-google Sep 8, 2021
15d29c1
fix: clean up and replace batch process in 311.
nlarge-google Sep 16, 2021
88961fb
fix: resolved isort issue
nlarge-google Sep 16, 2021
f0fd762
fix: Incrementing amount of CPUs allocated to BQ task in order to red…
nlarge-google Sep 21, 2021
2663c8a
fix: Runs as expected in Airflow 2
nlarge-google Sep 27, 2021
fb727bd
fix: Resolved flake8 issues
nlarge-google Sep 27, 2021
3575ceb
fix: Refactored code. Testing in Airflow shows the following issues -…
nlarge-google Sep 28, 2021
0018e9f
fix: resolve dag errors and clean up code
nlarge-google Sep 30, 2021
30568ed
fix: resolve isort issue
nlarge-google Sep 30, 2021
40a79ba
fix: Resolved issues in previous code review, improved code and all r…
nlarge-google Oct 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
293 changes: 293 additions & 0 deletions datasets/new_york/311_service_requests/311_service_requests_dag.py
View file
Open in desktop
@@ -0,0 +1,293 @@
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


from airflow import DAG
from airflow.contrib.operators import gcs_to_bq, kubernetes_pod_operator

default_args = {
"owner": "Google",
"depends_on_past": False,
"start_date": "2021-03-01",
}


with DAG(
dag_id="new_york.311_service_requests",
default_args=default_args,
max_active_runs=1,
schedule_interval="@daily",
catchup=False,
default_view="graph",
) as dag:

# Run CSV transform within kubernetes pod
transform_csv = kubernetes_pod_operator.KubernetesPodOperator(
task_id="transform_csv",
name="311_service_requests",
namespace="default",
image_pull_policy="Always",
image="{{ var.json.new_york.container_registry.run_csv_transform_kub_311_service_requests }}",
env_vars={
"SOURCE_URL": "https://data.cityofnewyork.us/api/views/erm2-nwe9/rows.csv",
"SOURCE_FILE": "files/data.csv",
"TARGET_FILE": "files/data_output.csv",
"CHUNKSIZE": "500000",
"TARGET_GCS_BUCKET": "{{ var.value.composer_bucket }}",
"TARGET_GCS_PATH": "data/new_york/311_service_requests/data_output.csv",
},
resources={"limit_memory": "8G", "limit_cpu": "3"},
)

# Task to load CSV data to a BigQuery table
load_to_bq = gcs_to_bq.GoogleCloudStorageToBigQueryOperator(
task_id="load_to_bq",
bucket="{{ var.value.composer_bucket }}",
source_objects=["data/new_york/311_service_requests/data_output.csv"],
source_format="CSV",
destination_project_dataset_table="new_york.311_service_requests",
skip_leading_rows=1,
allow_quoted_newlines=True,
write_disposition="WRITE_TRUNCATE",
schema_fields=[
{
"name": "unique_key",
"type": "INTEGER",
"description": "",
"mode": "NULLABLE",
},
{
"name": "created_date",
"type": "TIMESTAMP",
"description": "",
"mode": "NULLABLE",
},
{
"name": "closed_date",
"type": "TIMESTAMP",
"description": "",
"mode": "NULLABLE",
},
{"name": "agency", "type": "STRING", "description": "", "mode": "NULLABLE"},
{
"name": "agency_name",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "complaint_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "descriptor",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "location_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "incident_zip",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "incident_address",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "street_name",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "cross_street_1",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "cross_street_2",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "intersection_street_1",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "intersection_street_2",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "address_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{"name": "city", "type": "STRING", "description": "", "mode": "NULLABLE"},
{
"name": "landmark",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "facility_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{"name": "status", "type": "STRING", "description": "", "mode": "NULLABLE"},
{
"name": "due_date",
"type": "TIMESTAMP",
"description": "",
"mode": "NULLABLE",
},
{
"name": "resolution_description",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "resolution_action_updated_date",
"type": "TIMESTAMP",
"description": "",
"mode": "NULLABLE",
},
{
"name": "community_board",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "borough",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "x_coordinate",
"type": "INTEGER",
"description": "",
"mode": "NULLABLE",
},
{
"name": "y_coordinate",
"type": "INTEGER",
"description": "",
"mode": "NULLABLE",
},
{
"name": "park_facility_name",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "park_borough",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{"name": "bbl", "type": "INTEGER", "description": "", "mode": "NULLABLE"},
{
"name": "open_data_channel_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "vehicle_type",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "taxi_company_borough",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "taxi_pickup_location",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "bridge_highway_name",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "bridge_highway_direction",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "road_ramp",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "bridge_highway_segment",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
{
"name": "latitude",
"type": "FLOAT",
"description": "",
"mode": "NULLABLE",
},
{
"name": "longitude",
"type": "FLOAT",
"description": "",
"mode": "NULLABLE",
},
{
"name": "location",
"type": "STRING",
"description": "",
"mode": "NULLABLE",
},
],
)

transform_csv >> load_to_bq