Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add labels to all (BigQuery) resources and processes #2134

Open
9 tasks
adam-phillipps opened this issue Mar 27, 2024 · 0 comments
Open
9 tasks

Add labels to all (BigQuery) resources and processes #2134

adam-phillipps opened this issue Mar 27, 2024 · 0 comments
Labels
feature New feature or request

Comments

@adam-phillipps
Copy link

Please describe the feature you'd like to see
It would be very happy-making if I could add arbitrary labels to all BigQuery resources, jobs, queries, etc. that natively support them, so that I can run more accurate cost analysis.

Describe the solution you'd like
Please add tags/labels/etc. to the Table objects or allow us to pass tags and labels through as kwargs in create and load functions.
We are able to add QueryModifiers to some processes but I’d like to label all queries and the tables they create.

Are there any alternatives to this feature?
I’m not aware of any way around this. I can’t remember how it all works in AWS and I’m not sure at all about Azure, Databricks, etc. but in GCP, you can’t add labels to things like queries and jobs after they’ve been submitted and the only way I know how to filter during cost analysis is based on arbitrary metadata, ie cost reporting in AWS and GCP uses tags and labels respectively.

Additional context
This issue is similar. Many users have this need.
It looks like loading a table in BigQuery can be labeled in a few different places, here are two:

destination_table_ref = bigquery.table.TableReference.from_string(
  self.get_table_qualified_name(target_table), 
  default_project=project_id
)
table = self.hook.get_table(destination_table_ref)
table.labels = labels
self.hook.update_table(table, ["labels”])

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@adam-phillipps adam-phillipps added the feature New feature or request label Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant