Application of separate dedupe_only and link_only models

[medwar99]

Hi,

I'm looking at applying separate dedupe and link models, as recommend in #1601 (reply in thread) as a way to perform an initial clustering of two datasets and then a link between them, and I was hoping for a little more insight into how this 3-model pipeline would work in practice.

I want to go down this route as I have two datasets where some specific rules are easier to implement with two respective dedupe_only models, and these separate dedupes have provided nice pairwise predictions and clusters within the respective datasets. The aim here isn't really to remove duplicates in the dataset I suppose, but rather provide clusters of records which then want linking by the subsequent linker.

How can I actually apply the link_only model here? I build a link_only model and pass in the two datasets, but how could that take into account the previously identified clusters and predictions during training? Could I also instantiate the m/u parameters based on the two prior models? Can I use the outputs (pairwise or clusters) from the two previous models in some way?

[RobinL]

If I understand the question correctly, I think the approach you want to take is:

• apply your two different models to create pairwise record comparisons. do not cluster at this stage
• vertically concatenate (UNION ALL) the resultant sets of edges
• register this table in Splink
• tell Splink to cluster

I don't think this methodology is documented at all but here's a script that hopefully illustrates the idea. Sorry - wanted to do some working code here for you but ran out of time so this is more a sketch

# do your two linking models here

# my_nodes is then a list of all nodes from both models
# my_edges is then the UNION ALLed table of predictions
# You might need to dedupe it 


linker = SparkLinker(
    my_nodes,
    settings,
    break_lineage_method="parquet",
    set_up_basic_logging=False,
)
df_predict = linker.register_table_predict(my_edges, overwrite=True)
linker.cluster_pairwise_predictions_at_threshold(
            df_predict, threshold_match_probability=v
        )

[medwar99]

Thanks for the insight, I think that's got me on the right track. The snippet was mostly there, but that third linker needs to be trained for link_only and then the predictions added into that UNION ALL as well in order to get the inter-dataset links into the clustering (i.e. the union is over three pairwise prediction tables).

The output of this pipeline is then my three mapping tables (dedupe table for A, dedupe table for B, link table for A-B), along with identified clustering of all pairwise edges. This is fine for my use case, as I can then use the two types of edge (intra- and inter-dataset) in downstream applications as needed and the overall clustering sits at a level above this.

I've implemented a version of this on the historical_50k set. Happy to tidy this up and share.

[AdamChapnik1]

Hi Mike,

I am working through a very similar linkage process as you, and I have been having many of the same questions (so I am grateful to see that you asked them). I would be very interested in seeing your implementation on the historical_50k dataset, if you would be willing to share? Thanks so much.