RFC-0078: Zone-Aware Replica Reads

[avsej]

No description provided.

[programmatix]

Any way to get this rendering in the diff?

[avsej]

I also noticed, it renders in the doc, but neither absolute, nor relative (like this) is not getting picked up diff. Maybe only if I will make it full URL starting with http://...

[programmatix]

If we want to move to a GDoc-less workflow, I think this support will be important.

[avsej]

Yes, but with google docs still optional (#50 still is not merged) :)

And in a way google docs would require to perform conversion losing all discussions behind.

[programmatix]

One reason I was avoiding using the existing Get[Any|All]Replicas, is I consider their interfaces rather broken. One concrete example - I have no easy way of disambiguating programmatically between the document not existing, and a network error. There are other issues, including that GetAllReplicas swallows all substream errors, making it not a good building block for higher-level replica operations.

We need to consider that the feature needs to be added to transactions also - in fact this is the most important use-case since it is the one the requesting customer will use. And I'm somewhat reluctant to add a broken interface into transactions.

[programmatix]

On transactions - let's focus on non-transactions for now though. Once we nail that API it should be relatively straightforward to add it to transactions too. I'm only bringing up transactions as an argument for why I'd maybe prefer we avoided using getAnyReplica() for this.

[avsej]

new options are essentially change behavior, maybe we can agree that with this new option it will distinguish

• network error
• document is not found
• case where the selected strategy does not allow us to fulfill request
  • not enough local replicas
  • no replicas configured
  • even master is not local group

[programmatix]

Yes, that seems a viable solution.

[programmatix]

I may be changing my feeling on that.. I'm very reluctant to add a not-great interface into transactions.

It is an option to only include getAnyReplica() in transactions, and only allow LOCAL_ONLY mode (with its better behaviour) - but perhaps not a great one.

[avsej]

I believe after all it makes sense to add getAllReplicas and getAnyReplica into transactions and let them behave like regular operations, although implemented using lookupInAllReplicas and lookupInAnyReplica internally.

[programmatix]

If we treat the desire to write to a specific server group as a request not a command, it gives us a lot of lee-way to handle situations like this. If we don't have a config with server groups, we fallback to whatever the operation does normally.

[avsej]

I updated response to this question, basically the behavior for old servers is not defined as they do not support Zone-Aware-Replica-Reads in the way we describe it (which requires serverGroup markers in the configuration).

I think we should rather document this better to make it clear how to use it, and that whole cluster should have at least 7.6.2 on all nodes, and be balanced correctly.

[programmatix]

I'm not sure this has any meaning in GetAllReplicas?

[DemetrisChr]

I think getting all replicas from the local server group could be a valid use case (though that's more like GetSomeReplicas)

[avsej]

Right, also note that we always read from active, so if you have two replicas configured for the bucket, you might have two responses.

[programmatix]

This could be a neat solution to a lot of the concerns above. If we tell the user where we got the replica from, they could add their own warning/metrics if it doesn't match the server group they expected, rather than us needing to spam.

[programmatix]

The trouble with LOCAL_FIRST is that if the non-local node doesn't return anything, it's usually going to be due to a timeout - network blip or undiscovered node failover. So we only find out about it at the point the 2.5s KV user timeout has elapsed - and it's too late for us to try a fallback.

Instead of LOCAL_FIRST, we could support: The user can do an initial call with LOCAL_ONLY and a short timeout, and if that times out, try any fallback strategy they want.

This avoids all the complexity of the other potential solutions to this problem (like having two timeouts, or allowing the user to specify some sort of strategy thing).

[avsej]

@programmatix I've uploaded update for transactions API. It documents DocumentNotFound exception.

[programmatix]

The inability to view these significantly impacts ability to review

[programmatix]

It's maybe worth considering as we think about any fallback options, that we perhaps want to recommend that the user always has a fallback strategy in place:

try {
   return collection.getAnyReplica(docId, options().timeout("20ms").read_preference(SELECTED_SERVER_GROUP)
} catch DocumentUnretrievableException | DocumentNotFoundException {
  return collection.get(docId)
  // or collection.getAnyReplica() or collection.getAllReplicas()  - as they wish
}

This saves them from a bunch of edge cases - like a new doc not having been replicated, mixed-mode clusters, and possibly some rebalance & failover scenarios. Working with replicas is inherently a bit flaky, and we should advise users how to protect themselves.

Plus they don't necessarily know if they are working with 7.6.2 or not.

So in that context, knowing the user is going to have a fallback strategy in place already - I do wonder if there's any benefit to having fallback modes like this?

[programmatix]

s/local availability zone/selected|preferred server group, I think?

[programmatix]

Slight nit: SELECTED_SERVER_GROUP at least is a strict command rather than a preference. Should this be ReadRequirement?

I don't feel strongly on this though.

[programmatix]

Ah so maybe this has got lost in translation but I think this should be getReplicaFromPreferred[Server]Group() or similar.

(Or more accurately: I wish I had the time and capability to design a true getReplica() that handles all single-replica-read concerns. But I cannot figure out what that should look like. So until either inspiration strikes or I get more cycles to spend on it, the thought is to have a single-purpose method instead that is focussed on only this use-case. That doesn't "salt the earth" from doing a true getReplica() at a later date.)

[programmatix]

I feel a bit uneasy with transactions specs being outside of the hackmd docs where they are all grouped together.

Perhaps I can just copy this into the API doc once finalised? We need a better long-term solution though.

[programmatix]

typo: user

[programmatix]

I feel we should nail this down.

What you've proposed makes sense to me, in the context of what I propose elsewhere about the user always have try-catch handling.