hinted handoff: Draining doesn't always happen

[dawmd]

As part of #15567, we introduced a data structure keeping track of mappings between hint endpoint managers (identified by host IDs) and the hint directories managed by them (and represented by IP addresses). The data structure is used before the whole cluster is upgraded to a version of Scylla supporting host-ID-based hinted handoff. As an intended consequence, one hint directory may correspond to multiple target nodes in the cluster. If any of them leaves the cluster, we should drain the hint directory. This currently doesn't happen causing hint directories to potentially remain on disk and not be sent at the right time.

[piodul]

As an intended consequence, one hint directory may correspond to multiple target nodes in the cluster.

So, what happens with such hints in general? Are they sent to both nodes?

[dawmd]

So, what happens with such hints in general? Are they sent to both nodes?

They're sent to the node corresponding to the host ID from the mapping. For example, if we have two nodes – A and B – where the host ID of A is H and the IP of B is I, and if there is a mapping [H, I], then all hints will be stored in the hint directory of name I, but those hints will be sent to the node corresponding to H – in this case to A (unless it stops being the replica. Then the hints will be sent to all current replicas of course).

Although it doesn't abide by the intended semantics of hinted handoff, it only happens before the whole cluster has been upgraded. Then everything goes back to functioning the way it is supposed to, i.e. each node has its own hint directory and those hints will be sent to that specific node.

[piodul]

Although it doesn't abide by the intended semantics of hinted handoff, it only happens before the whole cluster has been upgraded. Then everything goes back to functioning the way it is supposed to, i.e. each node has its own hint directory and those hints will be sent to that specific node.

Ok, I think that's fine.