Google Big Query Write append_row retries on Aborted exceptions

[bohao-cao]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is your feature request related to a problem? Please describe.
I'm using the beta storage write client to upload data to GBQ using append_rows function(v2.6). In production I'm encountering the below error intermittently:

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.ABORTED
	details = "Closing the stream because server is restarted. This is expected and client is advised to reconnect. Entity: projects/<write stream path>"
	debug_error_string = "{"created":"@1632889718.948125699","description":"Error received from peer ipv4:74.125.70.95:443","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Closing the stream because server is restarted. This is expected and client is advised to reconnect. Entity: <write stream path>","grpc_status":10}"
>

According to the error an retry seems to be the best approach, but currently default retry doesn't retry for this type of exception.
Code reference here.

Describe the solution you'd like
Include retry for Aborted type of exception.

Describe alternatives you've considered
My client code pass in retry that include Aborted exception.

[tseaver]

The default retry policy is configured across client libraries for all languages in the service specification, and is part of code generated from that repo here.

You could use implementation details to change the default policy, e.g.:

from google.api_core import exceptions
from google.api_core import retry

_retry_abort_unavailable = retry.if_exception_type(
    exceptions.Abort, exceptions.ServiceUnavailable,
)

transport = client.transport
wrapper = transport._wrapped_methods[transport.append_row]
wrapper._retry._predicate = _retry_abort_unavailable

Ideally, we would make it possible to do that without using private attributes.

[tswast]

I believe this will require a change to the writer module to use ResumableBidiRpc instead of BidiRpc.

One would also need to make sure the queue of unacknowledged requests is re-sent and any duplicate/conflict responses that we get back are ignored properly.

[kamilglod]

How about the retry instance that we can pass to append_rows() method call and it's passed here to the rpc call?