recovery: retry on fetching log spec

[jgraettinger]

Observed from a consumer application on a recent release, where gazette and the consumer were rolling in tandem (the consumer happened to pick a gazette broker that was exiting):

{"err":"beginRecovery: fetching log spec: rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.0.35:51576-\u003e10.3.247.101:8080: read: connection reset by peer","level":"error","msg":"serveStandby failed","shard":"/gazette/consumers/flow/reactor/items/ ... ","time":"2021-09-24T18:58:07Z"}

[jgraettinger]

Digging into this, I'm hesitant to slap a retry on it because we are using FailFast(false) or equivalently WaitForReady(true) on these and other calls, and per WaitForReady comments gRPC should be re-trying these requests on a transient failure if the server didn't process the request (details.

We go a graceful gRPC stop, and all of the brokers exited cleanly, so I'm unsure why these pods received a TCP RST.

Likely will take no immediate action unless this is a repeated problem.