Duplicate file indexing.

[Wileyfox-F1]

Sorry for skipping the issues section to discuss this. Actually I want to report two bugs (may not be a bug). The first one is related to the latest version and the latter one is linked to the previous version. I tried to clone a (private) channel with 2Lakh+ media files. Actually I started the process from an intermediate id (about 19****), it started indexing for duplicate files and it frozen after a while. I guess it may be due to new messages in the source channel while cloning (not sure). The second bug is actually a query kinda thing to you, I found duplicate files while cloning an another channel (with almost zero probability of duplication). Did you use any kind of checksum stuff for indexing the duplicate files? Just concerned because those backup is pretty much important to me. Btw, it's a great project (similar ones are only paid services by third party devs) and thanks to you for your priceless efforts for this bot!

[m4mallu]

Bot Process:

1. Updating the configuration
2. Start clone
3. Indexing Target chat to a permanent CSV file stored locally, to use in the future (Process will take more time in case of massive chats. Process can be skipped and continue clone).
4. Cloning process (compare previously created list to the cloning media and updated with newcomers and saves in the same CSV)
5. Process finished and updated the CSV file with the final Index of the Target chat.

Case 1: Indexing got struck:
This can be happed due to so many reasons.

1. Heroku dyno restart
2. Copyright contents in the target
3. change of media in the target

Case 2: Duplicate files passed

1. In general, the bot is creating a list with unique_file_id of each file cloned or indexed in a process. so there won't be any chance to pass the same file.
2. Two files having the same properties (size, name, thumbnail, etc..) except in unique-file_id are considered as two identical files, as they have a different unique_file_id. In this case, you will get these two files while you clone a chat. This is the only situation you can get a 'visible duplicate file'.

The bot process is defined in a simple global scenario that can be adopted and updated in the future by anyone. Errors in a chat mentioned can cause the smooth working of the bot, especially in indexing. So wish to get a fine tune when someone has a new logic and pushed.

Thanks for the wishes. !

[Wileyfox-F1]

I will try to skip the first indexing after updating to the latest version then. Thanks for the reply :)