Repair backups on ReadRepair

[kwapik]

Before this change, ReadRepair mechanism was only restoring primary partitions and not backup ones.

[sonarcloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[buraksezer]

Unfortunately, your approach is wrong. The readRepair method is only called by the primary owner, and running the read repair logic on the replica/backup partition is useless.

It seems that the actual problem relies on the lookupOnReplicas method. The readRepair method works on the versions slice. lookupOnReplicas method somehow fails to add some replicas to the versions slice.

Please take a look at this snippet. It doesn't create a version if the replica owner returns an error when the primary wants to read a key/value pair. I think the replica owner returns a "key-not-found" or similar error message, and then it fails to repair the replica.

func (dm *DMap) lookupOnReplicas(hkey uint64, key string) []*version {
	// Check backup.
	backups := dm.s.backup.PartitionOwnersByHKey(hkey)
	versions := make([]*version, 0, len(backups))
	for _, replica := range backups {
		host := replica
		cmd := protocol.NewGetEntry(dm.name, key).SetReplica().Command(dm.s.ctx)
		rc := dm.s.client.Get(host.String())
		err := rc.Process(dm.s.ctx, cmd)
		err = protocol.ConvertError(err)
		if err != nil {
			if dm.s.log.V(6).Ok() {
				dm.s.log.V(6).Printf("[DEBUG] Failed to call get on"+
					" a replica owner: %s: %v", host, err)
			}
			continue
		}

[buraksezer]

I'm actively working on this problem.