New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Onboard BLS - CPSAAT 2020 dataset #105
Conversation
datasets/bls/cpsaat18/pipeline.yaml
Outdated
depends_on_past: False | ||
start_date: '2021-06-23' | ||
max_active_runs: 1 | ||
schedule_interval: "@once" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be yearly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! f309da6
datasets/bls/cpsaat18/pipeline.yaml
Outdated
args: | ||
task_id: "load_csv_to_bq" | ||
bucket: "{{ var.json.bls.source_bucket }}" | ||
source_objects: ["cpsaat18/2020.csv"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
csv is 2020 so if we switch to yearly this doesn't work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done! 3c83177
datasets/bls/dataset.yaml
Outdated
resources: | ||
- type: bigquery_dataset | ||
dataset_id: bls | ||
description: "Overview: This dataset includes economic statistics on inflation, prices, unemployment, and pay & benefits provided by the Bureau of Labor Statistics (BLS).\n\nUpdate frequency: Monthly\n\nDataset source: U.S. Bureau of Labor Statistics\n\nTerms of use: This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - http://www.data.gov/privacy-policy#data_policy - and is provided \"AS IS\" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.\n\nSee the GCP Marketplace listing for more details and sample queries: https://console.cloud.google.com/marketplace/details/bls-public-data/bureau-of-labor-statistics" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now that #103 is in we can get rid of the newlines :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here you go! Looks nicer now on the YAML config: e0b82a4
Description
As requested by some teams to be able to join hiring data with labour statistics.
https://www.bls.gov/cps/cpsaat18.htm
Checklist
datasets/<YOUR-DATASET>
and nothing outside of that directory.