Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job retry solution (client.create_job) is broken #297

Closed
bhtucker opened this issue Oct 5, 2020 · 0 comments · Fixed by #300
Closed

Job retry solution (client.create_job) is broken #297

bhtucker opened this issue Oct 5, 2020 · 0 comments · Fixed by #300
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@bhtucker
Copy link

bhtucker commented Oct 5, 2020

A solution was implemented in #71 but is now broken.

This is not caught by unit tests because the failure is within a mocked interface.

Original issue: #14

Environment details

  • Linux
  • Python 3.6, 3.7
  • Client version 1.25 - 2.0.0

Steps to reproduce

Run the code in the unit tests without mocks; rather than failing for nonexistent/unauthorized project/dataset (as you would expect), it fails with an AttributeError.:

Code example

configuration = {
            "load": {
                "destinationTable": {
                    "projectId": 'anyproject',
                    "datasetId": 'anydataset',
                    "tableId": "source_table",
                },
                "sourceUris": ["gs://test_bucket/src_object*"],
            }
        }

bqclient.create_job(configuration)

Stack trace

Fails with:

~/python/lib/python3.7/site-packages/google/cloud/bigquery/client.py in create_job(self, job_config, retry)
   1619             source_uris = _get_sub_prop(job_config, ["load", "sourceUris"])
   1620             return self.load_table_from_uri(
-> 1621                 source_uris, destination, job_config=load_job_config, retry=retry
   1622             )
   1623         elif "copy" in job_config:

~/python/lib/python3.7/site-packages/google/cloud/bigquery/client.py in load_table_from_uri(self, source_uris, destination, job_id, job_id_prefix, location, project, job_config, retry, timeout)
   1967 
   1968         load_job = job.LoadJob(job_ref, source_uris, destination, self, job_config)
-> 1969         load_job._begin(retry=retry, timeout=timeout)
   1970 
   1971         return load_job

~/python/lib/python3.7/site-packages/google/cloud/bigquery/job.py in _begin(self, client, retry, timeout)
    643             method="POST",
    644             path=path,
--> 645             data=self.to_api_repr(),
    646             timeout=timeout,
    647         )

~/python/lib/python3.7/site-packages/google/cloud/bigquery/job.py in to_api_repr(self)
   1694             )
   1695         _helpers._set_sub_prop(
-> 1696             configuration, ["load", "destinationTable"], self.destination.to_api_repr()
   1697         )
   1698 

AttributeError: 'dict' object has no attribute 'to_api_repr'

As you can see here, the mocked client method load_table_from_uri is where the traceback begins.

The destination must be coerced to a TableRef before calling this method.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Oct 5, 2020
@HemangChothani HemangChothani self-assigned this Oct 6, 2020
@HemangChothani HemangChothani added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Oct 6, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. and removed triage me I really want to be triaged. labels Oct 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants