fix(pubsub): retry GOAWAY errors (#4313)

The frontend server for Pub/Sub might occasionally emit GOAWAY errors which are currently not retried. This is not unique to the Go client, though the categorization as `UNKNOWN` vs `UNVAILABLE` is a golang-GRPC issue. Although UNKNOWN [should not generally be retried](https://google.aip.dev/194), this will unblock users of `Receive` until the grpc library can be changed. See internal cl/377393940 for a similar fix in another Go library.

Fixes #4257.

pubsub/service.go

@@ -62,6 +62,14 @@ func (r *defaultRetryer) Retry(err error) (pause time.Duration, shouldRetry bool
 			return r.bo.Pause(), true
 		}
 		return 0, false
+	case codes.Unknown:
+		// Retry GOAWAY, see https://github.com/googleapis/google-cloud-go/issues/4257.
+		isGoaway := strings.Contains(s.Message(), "error reading from server: EOF") &&
+			strings.Contains(s.Message(), "received prior goaway")
+		if isGoaway {
+			return r.bo.Pause(), true
+		}
+		return 0, false
 	default:
 		return 0, false
 	}