Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(BigQuery) Running a query and then downloading results via storage read api is awkward #177

Open
magnalite opened this issue Jun 29, 2023 · 1 comment

Comments

@magnalite
Copy link

magnalite commented Jun 29, 2023

I would like to use a syntax like client.query(project_id, query).use_storage_read(true) or client.query_with_storage(project_id, query) to send a query and then read the results via the storage read api.

Currently we need to use client.job.query(...), wait for the job to finish, grab the destination id (if it is temporary/unknown) from the response via client.job().get(...) and then initiate a client.read_table(...) to pull the data. This is fine but I imagine that this is a common enough use case to warrant providing a convenient top level api.

If you could outline the which api interface you think is best I'd be happy to have a go making a PR.

@magnalite magnalite changed the title Running a query and then downloading results via storage read api is awkward (BigQuery) Running a query and then downloading results via storage read api is awkward Jun 29, 2023
@yoshidan
Copy link
Owner

yoshidan commented Jun 30, 2023

Thanks for your suggestion.

I think client.query_with_storage(project_id, query) would be better since it does not affect the existing intanface.

Ideally, it would be best to abstract the return value Iterator like Go implementation.

pub struct RowIterator {
   inner: // enum of Storage or REST API
}
pub async fn query(project_id, query) -> Result<RowIterator, Error>

https://github.com/googleapis/google-cloud-go/blob/2b99e4f39be20fe21e8bc5c1ec1c0e758222c46e/bigquery/query.go#L405

However, there is a lot to modify, so any suggestions are welcome for PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants