[Feature] Python support for fuzzy dedup implementation

[Bytes-Explorer]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/universal/fdedup

Feature

Python code for fuzzy dedup logic for code and NLP.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[blublinsky]

I am sorry, but I have no idea how to do this in pure Python. It will be a completely different implementation. The current code is leveraging Ray capabilities, that do not exist in pure Python. Additionally, fuzzy dedup uses two two passes, which is not really applicable for pure Python.