rls: update picker synchronously upon receipt of configuration update

[easwars]

#5469 recommends an audit of existing LB policies to ensure that they update their pickers synchronously upon receipt of a configuration update.

rls LB policy does not update its picker synchronously in one corner case.

• Configuration updates are handled here:

  grpc-go/balancer/rls/balancer.go

  Line 252 in b4f7947

  func (b *rlsBalancer) UpdateClientConnState(ccs balancer.ClientConnState) error {

• The LB policy inhibits picker updates until the new configuration is pushed to all child policies. See:

  grpc-go/balancer/rls/balancer.go

  Line 270 in b4f7947

  b.inhibitPickerUpdates = true

• In a blocking fashion, it does the following
  • pushes updates to child policies
  • waits for picker updates from all the child policies
  • sends a new picker
  • See:

    grpc-go/balancer/rls/balancer.go

    Line 293 in b4f7947

    b.updateCh.Put(resumePickerUpdates{done: done})

• At the end of UpdateClientConnState, the LB policy handles any changes to the data cache size here:

  grpc-go/balancer/rls/balancer.go

  Line 297 in b4f7947

  if resizeCache {

  • The return value from the resize operation is ignored here:

    grpc-go/balancer/rls/balancer.go

    Line 306 in b4f7947

    b.dataCache.resize(newCfg.cacheSizeBytes)

  • The return value should not be ignored. Instead if any entry with a backoff timer was evicted, then it should result in a new picker being sent upwards.

Quoting this from the design:

When removing an entry whose backoff_time_ is in the future, the backoff timer 
will be cancelled.  Note that just like when the backoff timer fires normally, a new
 picker is returned to the channel, to force it to re-process any wait-for-ready RPCs
that may still be queued if we failed them while we were in backoff.