feat: Implemented BigQueryRetryAlgorithm to retry on the basis of the configured re-triable error messages #1426

prash-mi · 2021-07-03T04:41:29Z

Implemented com.google.cloud.bigquery.BigQueryRetryAlgorithm which usescom.google.cloud.bigquery.BigQueryRetryConfig to retry on the basis of the configured re-triable error messages.

Modified BigQueryImpl.getQueryResults to consume BigQueryRetryHelper.runWithRetries to retry using BigQueryRetryAlgorithm

Fixes #1250 ☕️

By default we retry on the following error:

{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "location" : "q",
    "locationType" : "parameter",
    "message" : "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors at [5:1]",
    "reason" : "invalidQuery"
  } ],
  "message" : "Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors at [5:1]",
  "status" : "INVALID_ARGUMENT"
}

…ation

…va-bigquery into bigquery-bugfix-1250

…ased on the error message

…ption ex) method to handle BigQueryRetryHelperException

…yResults

stephaniewang526 · 2021-07-04T20:57:04Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java

@@ -1362,7 +1366,7 @@ private static QueryResponse getQueryResults(
                    : jobId.getLocation());
    try {
      GetQueryResultsResponse results =
-          runWithRetries(
+          BigQueryRetryHelper.runWithRetries(


Can we add this also to TableResult query(...) methods? That should also allow you to verify idempotency using requestId for jobs.query API.

Sure, I have added BigQueryRetryHelper.runWithRetries for TableResult query(...) and have implemented testcase testFastQueryRateLimitIdempotency to test the idempotency

Okay great but I think we need it for the jobs.insert endpoint too (line 1269). There are 2 query paths here. The one you've updated is the "fast" query path (jobs.query).

Sure, I have implemented BigQueryRetryHelper.runWithRetries on QueryResponse waitForQueryResults method, which is used by TableResult getQueryResults method

…yRPC method

…ency of the BigQueryRetryHelper.runWithRetries for TableResult.query(...)

stephaniewang526 · 2021-07-06T15:44:28Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java

@@ -237,6 +237,10 @@
  }

  private final BigQueryRpc bigQueryRpc;
+  private static final BigQueryRetryConfig DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG =


Should we just name this DEFAULT_RETRY_CONFIG? It currently has ratelimitexceeded and we can potentially add more in the future.

sure, updated it

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryErrorMessages.java

stephaniewang526 · 2021-07-06T15:48:45Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java

@@ -237,6 +237,10 @@
  }

  private final BigQueryRpc bigQueryRpc;
+  private static final BigQueryRetryConfig DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG =
+      BigQueryRetryConfig.newBuilder()
+          .retryOnMessage(BigQueryErrorMessages.RATE_LIMIT_EXCEEDED_MSG)


I find this a little extraneous -- why do we need to specify every message we need to retry on? If we build a config, I imagine we just retry on all the specified error messages in the config.

Sure Stephanie. As discussed, I have tried to create it on the lines of com.google.cloud.ExceptionHandler and this may give us additional flexibility to configure hooks like retryOnStatus or retryOnReason later as necessary. Happy to refactor it as needed.

stephaniewang526 · 2021-07-06T15:49:27Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQueryImpl.java

@@ -1362,7 +1366,7 @@ private static QueryResponse getQueryResults(
                    : jobId.getLocation());
    try {
      GetQueryResultsResponse results =
-          runWithRetries(
+          BigQueryRetryHelper.runWithRetries(


Okay great but I think we need it for the jobs.insert endpoint too (line 1269). There are 2 query paths here. The one you've updated is the "fast" query path (jobs.query).

…itForQueryResults` method, which is used by `TableResult getQueryResults` method

…ponse waitForQueryResults` method, which is used by `TableResult getQueryResults` method" This reverts commit 84a3418.

…ETRY_CONFIG" This reverts commit 22b1706.

…ETRY_CONFIG" This reverts commit 2d21e11.

…ForQueryResults` method, which is used by `TableResult getQueryResults` method

stephaniewang526 · 2021-07-13T17:50:54Z

@prash-mi I tried running the ITs a couple times and it continues to fail here:

java.lang.AssertionError: expected null, but was:<[gs://bigquery-prod-upload-us/prod-scotty-b30183f1-0dcf-4712-83aa-d9d9ac06a899]>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotNull(Assert.java:756)
	at org.junit.Assert.assertNull(Assert.java:738)
	at org.junit.Assert.assertNull(Assert.java:748)
	at com.google.cloud.bigquery.it.ITBigQueryTest.testInsertFromFile(ITBigQueryTest.java:3121)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Could you try running locally to see this happens?

prash-mi and others added 22 commits June 29, 2021 20:44

Updated BigQueryImpl

c48231d

Initial Commit

e7c33e6

Merge branch 'googleapis:master' into bigquery-bugfix-1250

faaee99

Using BigQueryRetryAlgorithm as the retry algorithm

dbcfa7d

Created BigQueryRetryAlgorithm as a subclass of RetryAlgorithm

e902f50

BigQueryErrorMessages property file

ab25cea

Implemented Builder Logic for BigQueryRetryConfig

77a874d

Using BigQueryRetryConfig for getQueryResults

ca153b8

Updated shouldRetry with the logic to retry based on error messages

cb15bc6

Implemented null checks on shouldRetryBasedOnBigQueryRetryConfig

7774662

Removed Status from shouldRetryBasedOnBigQueryRetryConfig implement…

a271259

…ation

Removed unused imports

6d7e4bf

created DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG for getQueryResults

68299bf

Merge branch 'googleapis:master' into bigquery-bugfix-1250

480b13b

Merge branch 'bigquery-bugfix-1250' of https://github.com/prash-mi/ja…

43752b3

…va-bigquery into bigquery-bugfix-1250

Added testGetQueryResultsRetry test for testing getQueryResults Retry

0e99fa3

Overriding createNextAttempt method so that it generates an attempt b…

38f3977

…ased on the error message

Linted BigQueryRetryHelper

40393b3

Linted testGetQueryResultsRetry

2e234b0

Linted BigQueryRetryAlgorithm

28fa870

Linted BigQueryErrorMessages

67f4ce1

Linted BigQueryRetryConfig

a986f37

prash-mi requested review from a team and stephaniewang526 July 3, 2021 04:41

product-auto-label bot added the api: bigquery Issues related to the googleapis/java-bigquery API. label Jul 3, 2021

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jul 3, 2021

prash-mi added 4 commits July 3, 2021 10:16

Fixed Linting

3da3430

Fixed Linting

66fd067

Fixed Linting

a72a2f5

Created translateAndThrow(BigQueryRetryHelper.BigQueryRetryHelperExce…

cc5f730

…ption ex) method to handle BigQueryRetryHelperException

Handling BigQueryRetryHelper.BigQueryRetryHelperException for getQuer…

0648071

…yResults

stephaniewang526 reviewed Jul 4, 2021

View reviewed changes

prash-mi added 2 commits July 5, 2021 12:55

Implementing BigQueryRetryHelper.runWithRetries from TableResult.quer…

286e3f1

…yRPC method

Implementing testFastQueryRateLimitIdempotency Method to test Idempot…

68c06e2

…ency of the BigQueryRetryHelper.runWithRetries for TableResult.query(...)

stephaniewang526 suggested changes Jul 6, 2021

View reviewed changes

prash-mi and others added 9 commits July 12, 2021 11:49

Changed DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG to DEFAULT_RETRY_CONFIG

22b1706

Implemented BigQueryRetryHelper.runWithRetries on `QueryResponse wa…

84a3418

…itForQueryResults` method, which is used by `TableResult getQueryResults` method

Merge branch 'googleapis:master' into bigquery-bugfix-1250

13e8132

Revert "Implemented BigQueryRetryHelper.runWithRetries on `QueryRes…

8f5fc45

…ponse waitForQueryResults` method, which is used by `TableResult getQueryResults` method" This reverts commit 84a3418.

Revert "Changed DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG to DEFAULT_R…

c5d6b70

…ETRY_CONFIG" This reverts commit 22b1706.

Renamed DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG to DEFAULT_RETRY_CONFIG

2d21e11

Revert "Renamed DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG to DEFAULT_R…

592eea8

…ETRY_CONFIG" This reverts commit 2d21e11.

Renamed DEFAULT_RATE_LIMIT_EXCEEDED_RETRY_CONFIG to DEFAULT_RETRY_CONFIG

d448132

Implemented BigQueryRetryHelper.runWithRetries on `QueryResponse wait…

9833823

…ForQueryResults` method, which is used by `TableResult getQueryResults` method

stephaniewang526 approved these changes Jul 13, 2021

View reviewed changes

stephaniewang526 merged commit 44d9795 into googleapis:master Jul 14, 2021

release-please bot mentioned this pull request Jan 4, 2022

chore(main): release java-bigquery 3.0.0 #1768

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Implemented BigQueryRetryAlgorithm to retry on the basis of the configured re-triable error messages #1426

feat: Implemented BigQueryRetryAlgorithm to retry on the basis of the configured re-triable error messages #1426

prash-mi commented Jul 3, 2021 •

edited by stephaniewang526

stephaniewang526 Jul 4, 2021 •

edited

prash-mi Jul 5, 2021

stephaniewang526 Jul 6, 2021 •

edited

prash-mi Jul 13, 2021

stephaniewang526 Jul 6, 2021

prash-mi Jul 12, 2021

stephaniewang526 Jul 6, 2021

prash-mi Jul 12, 2021

stephaniewang526 Jul 6, 2021 •

edited

stephaniewang526 commented Jul 13, 2021

feat: Implemented BigQueryRetryAlgorithm to retry on the basis of the configured re-triable error messages #1426

feat: Implemented BigQueryRetryAlgorithm to retry on the basis of the configured re-triable error messages #1426

Conversation

prash-mi commented Jul 3, 2021 • edited by stephaniewang526

stephaniewang526 Jul 4, 2021 • edited

Choose a reason for hiding this comment

prash-mi Jul 5, 2021

Choose a reason for hiding this comment

stephaniewang526 Jul 6, 2021 • edited

Choose a reason for hiding this comment

prash-mi Jul 13, 2021

Choose a reason for hiding this comment

stephaniewang526 Jul 6, 2021

Choose a reason for hiding this comment

prash-mi Jul 12, 2021

Choose a reason for hiding this comment

stephaniewang526 Jul 6, 2021

Choose a reason for hiding this comment

prash-mi Jul 12, 2021

Choose a reason for hiding this comment

stephaniewang526 Jul 6, 2021 • edited

Choose a reason for hiding this comment

stephaniewang526 commented Jul 13, 2021

prash-mi commented Jul 3, 2021 •

edited by stephaniewang526

stephaniewang526 Jul 4, 2021 •

edited

stephaniewang526 Jul 6, 2021 •

edited

stephaniewang526 Jul 6, 2021 •

edited