Training phase running very long

[sumaniitm]

Hi Sonal,
I have a use case where I want to use pre-existing training data.
I am following your documentation (we use Snowflake as our data store) and have tweaked the configs as per the examples using the relevant format for snowflake

However, the training phase seems to go on forever even on relatively small-sized training dataset.
Looking at the logs it seems there is a proliferation of positive (WARN Trainer: Training on positive pairs - 10912086) and negative (WARN Trainer: Training on negative pairs - 3781051) training samples.
I have also looked at the source code of zingg and not able to understand how is the self join happening

The snowflake table used for providing training data has a total of 7,721 records, with 5 negative z_clusters having 3,013 records and 2 positive z_clusters with 4,708 records.
Can you please help me understand how the proliferation of training data is happening?
My data has only one attribute which needs fuzzy matching and no other attribute is present in the dataset. The training data has this attribute along with the other two mandatory z_cluster and z_ismatch attributes

Below is a breakup of the training data
Z_ISMATCH Z_CLUSTER RECORD_COUNT
0 12 3
0 13 18
0 14 25
0 15 226
0 16 2,741
1 10 4,672
1 11 36

[sumaniitm]

Figured out the root cause, it was happening due to duplicates in the training data, closing this thread