Constant 503 errors in PubSub StreamingPull.

[MrJoy]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

• Search the issues already opened: https://github.com/googleapis/google-api-ruby-client/issues
• Search Stack Overflow: https://stackoverflow.com/questions/tagged/google-api-client+ruby

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS: Fedora CoreOS
• Ruby version: 2.6.6
• Gem name and version: google-cloud-pubsub 2.1.0

Steps to reproduce

1. The error appears to be happening extremely frequently, and involves simply running the code. GCP console reports that most StreamingPull calls are failing. Despite this, delivery seems to be happening. The client library is swallowing the errors and retrying, but I'm alarmed at seeing nearly 100% failure rates on an app that currently has one user (staging environment, and we're pre-launch).

Code example

  def initialize
    @pubsub = Google::Cloud::Pubsub.new(project: PROJECT_ID)
  end

  def start
    subscription = @pubsub.subscription(SUBSCRIPTION_NAME)
    @subscriber  =
      subscription.listen do |msg|
        data = JSON.parse(msg.data)
        GooglePushNotificationWorker.perform_async(data["emailAddress"], data["historyId"])
        msg.acknowledge!
      rescue StandardError => e
        logger.error(e)
      end

    @subscriber.start
  end

My suspicion that this could be a client-library issue comes from the fact that the Python client library apparently had a very similar issue, apparently due to changes in the GRPC lib: googleapis/python-api-core#25

When I enable logging, I see this in my logs:

D, [2020-09-21T11:58:53.389877 #31843] DEBUG -- : bidi-read-loop: null batch #<struct Struct::BatchResult send_message=nil, send_metadata=nil, send_close=nil, send_status=nil, message=nil, metadata={"date"=>"Mon, 21 Sep 2020 18:58:53 GMT", "alt-svc"=>"h3-Q050=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-27=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-T050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""}, status=nil, cancelled=nil>
D, [2020-09-21T11:58:53.390025 #31843] DEBUG -- : bidi-read-loop: done status #<struct Struct::Status code=14, details="The service was unable to fulfill your request. Please try again. [code=8a75]", metadata={}, debug_error_string="{\"created\":\"@1600714733.389971000\",\"description\":\"Error received from peer ipv4:74.125.20.95:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1062,\"grpc_message\":\"The service was unable to fulfill your request. Please try again. [code=8a75]\",\"grpc_status\":14}">
D, [2020-09-21T11:58:53.390112 #31843] DEBUG -- : Failing with status #<struct Struct::Status code=14, details="The service was unable to fulfill your request. Please try again. [code=8a75]", metadata={}, debug_error_string="{\"created\":\"@1600714733.389971000\",\"description\":\"Error received from peer ipv4:74.125.20.95:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1062,\"grpc_message\":\"The service was unable to fulfill your request. Please try again. [code=8a75]\",\"grpc_status\":14}">
W, [2020-09-21T11:58:53.390240 #31843]  WARN -- : bidi: read-loop failed
W, [2020-09-21T11:58:53.390265 #31843]  WARN -- : 14:The service was unable to fulfill your request. Please try again. [code=8a75]. debug_error_string:{"created":"@1600714733.389971000","description":"Error received from peer ipv4:74.125.20.95:443","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"The service was unable to fulfill your request. Please try again. [code=8a75]","grpc_status":14} (GRPC::Unavailable)
/Users/jfrisby/.rbenv/versions/2.6.6/gemsets/thryve-core/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
/Users/jfrisby/.rbenv/versions/2.6.6/gemsets/thryve-core/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:209:in `block in read_loop'
/Users/jfrisby/.rbenv/versions/2.6.6/gemsets/thryve-core/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `loop'
/Users/jfrisby/.rbenv/versions/2.6.6/gemsets/thryve-core/gems/grpc-1.32.0-universal-darwin/src/ruby/lib/grpc/generic/bidi_call.rb:195:in `read_loop'

[MrJoy]

Correction: It looks like I'm either not getting some messages, or I'm not getting them for a very long time. I can reproduce the issue locally and... I'm just not seeing messages I know to be in the topic.

[MrJoy]

Whoops, wrong repo. Closing to re-open in appropriate place.