[PubSub] subscription.on_error is called indefinitely without backoff

[shouichi]

Environment details

• OS: linux
• Ruby version: 3.2.2
• Gem name and version: google-cloud-pubsub

Steps to reproduce

1. create a project, topic, and subscription
2. start a subscriber
3. delete the project

Code example

subscription.listen { ... }
subscription.on_error { |e| p err }

When the subscription is not found, on_error is called without backoff (almost every ms). It might be a good idea to add not found error to this clause.

google-cloud-ruby/google-cloud-pubsub/lib/google/cloud/pubsub/subscriber/stream.rb

Lines 287 to 289 in 24cf551

	rescue GRPC::Cancelled, GRPC::DeadlineExceeded, GRPC::Internal,
	GRPC::ResourceExhausted, GRPC::Unauthenticated,
	GRPC::Unavailable

Full backtrace

5:Requested project not found or user does not have access to it (project=project-id-here). Make sure to specify the unique project identifier and not the Google Cloud Console display name.. debug_error_string:{UNKNOWN:Error received from peer ipv4:173.194.192.95:443 {created_time:"2023-05-02T07:10:56.589985506+00:00", grpc_status:5, grpc_message:"Requested project not found or user does not have access to it (project=project-id-here). Make sure to specify the unique project identifier and not the Google Cloud Console display name."}}

/usr/local/bundle/gems/grpc-1.53.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status'
  /usr/local/bundle/gems/grpc-1.53.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:211:in `block in read_loop'
  /usr/local/bundle/gems/grpc-1.53.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:197:in `loop'
  /usr/local/bundle/gems/grpc-1.53.0-x86_64-linux/src/ruby/lib/grpc/generic/bidi_call.rb:197:in `read_loop'
  /usr/local/bundle/bin/whatever:in `each'

[NivedhaSenthil]

Thanks for bringing this to our notice, the behaviour here has to be updated to shutdown the subscriber on non recoverable errors.