[Bug]: When running fastdup as two steps (and there are bad images) connected component ids do not match atrain_features.dat.csv

[Tompil3r]

What happened?

When running fastdup as two steps - run_ mode=0 and then run_mode=1 (and there are bad images), there is a mismatch between the indices in atrain_features.dat.csv and connected_components.

files = [
    'pets/oxford-iiit-pet/images/Abyssinian_1.jpg',
    'pets/oxford-iiit-pet/images/Abyssinian_34.jpg',
    'pets/oxford-iiit-pet/images/Egyptian_Mau_71.jpg',
    'pets/oxford-iiit-pet/images/Egyptian_Mau_139.jpg'
]

fastdup.run(input_dir=files, work_dir='wd2', run_mode=1, turi_param='store_int=1')
fastdup.run(input_dir=files, work_dir='wd2', run_mode=2, turi_param='store_int=1,ccthreshold=0.97')

atrain_features.dat.csv:

┌───────┬─────────────────────────────────────────────────┐
│ index ┆ filename                                        │
│ ---   ┆ ---                                             │
│ i64   ┆ str                                             │
╞═══════╪═════════════════════════════════════════════════╡
│ 0     ┆ pets/oxford-iiit-pet/images/Abyssinian_1.jpg    │
│ 2     ┆ pets/oxford-iiit-pet/images/Egyptian_Mau_71.jpg │
└───────┴─────────────────────────────────────────────────┘

atrain_features.bad.csv:

┌───────┬──────────────────────────────────────────────────┬─────────────────────┬──────────┐
│ index ┆ filename                                         ┆ error_code          ┆ filesize │
│ ---   ┆ ---                                              ┆ ---                 ┆ ---      │
│ i64   ┆ str                                              ┆ str                 ┆ i64      │
╞═══════╪══════════════════════════════════════════════════╪═════════════════════╪══════════╡
│ 1     ┆ pets/oxford-iiit-pet/images/Abyssinian_34.jpg    ┆ ERROR_CORRUPT_IMAGE ┆ 33705    │
│ 3     ┆ pets/oxford-iiit-pet/images/Egyptian_Mau_139.jpg ┆ ERROR_CORRUPT_IMAGE ┆ 55860    │
└───────┴──────────────────────────────────────────────────┴─────────────────────┴──────────┘

connected_components.csv (index 1 is bad):

┌──────┬──────────────┬───────┬───────────────┬──────────────┬──────────────┐
│ __id ┆ component_id ┆ count ┆ mean_distance ┆ min_distance ┆ max_distance │
│ ---  ┆ ---          ┆ ---   ┆ ---           ┆ ---          ┆ ---          │
│ i64  ┆ i64          ┆ i64   ┆ f64           ┆ f64          ┆ f64          │
╞══════╪══════════════╪═══════╪═══════════════╪══════════════╪══════════════╡
│ 0    ┆ 0            ┆ 0     ┆ 0.0           ┆ 0.0          ┆ 0.0          │
│ 1    ┆ 1            ┆ 0     ┆ 0.0           ┆ 0.0          ┆ 0.0          │
└──────┴──────────────┴───────┴───────────────┴──────────────┴──────────────┘

when running in one step the connected component indices are as expected:

fastdup.run(input_dir=files, work_dir='wd1', turi_param='store_int=1,ccthreshold=0.97')

connected_components.csv:

┌──────┬──────────────┬───────┬───────────────┬──────────────┬──────────────┐
│ __id ┆ component_id ┆ count ┆ mean_distance ┆ min_distance ┆ max_distance │
│ ---  ┆ ---          ┆ ---   ┆ ---           ┆ ---          ┆ ---          │
│ i64  ┆ i64          ┆ i64   ┆ f64           ┆ f64          ┆ f64          │
╞══════╪══════════════╪═══════╪═══════════════╪══════════════╪══════════════╡
│ 0    ┆ 0            ┆ 0     ┆ 0.0           ┆ 0.0          ┆ 0.0          │
│ 2    ┆ 2            ┆ 0     ┆ 0.0           ┆ 0.0          ┆ 0.0          │
└──────┴──────────────┴───────┴───────────────┴──────────────┴──────────────┘

What did you expect to see?

No response

What version of fastdup were you runnning on?

1.122

What version of Python were you running on?

Python 3.9

Operating System

Ubuntu 20

Reproduction steps

No response

Relevant log output

No response

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response