Utilize AniDb Scrapping to DB matching

[salomj]

Description

Summary

Utilize an AniDB Scrapper library to automatically pull AniDB matching, rather than have to enter everything manually.

Details

• Added some support for TMDb/tv
• Stopped using "private" methods outside of convert.py
• Switched X_to_anidb and anidb_to_X to consistently be 1 to Many.

Related Changes

• Adds Automated AniDb Aggregation to anime_ids Anime-IDs#1
• Utilize AniDb online cache #2037

Type of Change

• New feature (non-breaking change which adds functionality)

Checklist

• My code was submitted to the nightly branch of the repository.

[salomj]

Coding part of these changes are done, in process of testing.

[salomj]

show_tmdb isn't a great name, but it gets the point across.

Note: This will only happen if the show doesn't match in both IMDb and TVDb, but has a TMDb entry which is uncommon.

[meisnate12]

the linkage to that github json should just be put into https://github.com/Kometa-Team/Anime-IDs

[meisnate12]

Also do have more information on where this data is coming from? Did you just find this repo or are you involved with it?

https://github.com/notseteve/AnimeAggregations

[meisnate12]

Kometa also uses TVDb IDs as the main source for show ids not TMDb anytime i get TMDb Show IDs theyre just converted to TVDb IDs

[salomj]

the linkage to that github json should just be put into https://github.com/Kometa-Team/Anime-IDs
This doesn't work for many many anime, for two different reasons.

1. This json file does not include TMDb/tv nor a way to distinguish between TMDb/movie and TMDb/tv. So I am unsure how you'd like me to modify this file. Or are you saying you'd like me to make changes to regularly import the JSON to that library?

Also do have more information on where this data is coming from? Did you just find this repo or are you involved with it?
https://github.com/notseteve/AnimeAggregations

1. Data is straight from AniDB, it gets aggregated every 15 days.

Kometa also uses TVDb IDs as the main source for show ids not TMDb anytime I get TMDb Show IDs theyre just converted to TVDb IDs

1. Yes, and that's a problem a lot of AniDb doesn't have TVDb for one reason or another, but nearly all of them have TMDb.

[salomj]

I have ran this against my local database, and I went from a little over 1200 unmatched items to 83.

[Arial-Z]

Not sure why switching from the most trusted source for animes mappings (anime-lists) used by all animes agent (HAMA, anidb.net, MAL agent ...) to this repo is interesting.
A simple search on the list (https://raw.githubusercontent.com/notseteve/AnimeAggregations/main/aggregate/AnimeToExternal.json) return 1056 tmdb tv entry, animes-lists got over 6600 tvdb id entry (including seasons and special mappings), and if looking only for season 1 (what you're doing here i think), it's got 3800 entry.
I found 476 tmdb movie entry, animes-lists got 1600 entry for movies with an imdbid.
Also what happen to seasons and episodes offsets, kometa currently don't use them, but they are needed to work on the seasons (that all other agent use and maybe kometa one day)

[salomj]

Not sure why switching from the most trusted source for animes mappings (anime-lists) used by all animes agent (HAMA, anidb.net, MAL agent ...) to this repo is interesting. A simple search on the list (https://raw.githubusercontent.com/notseteve/AnimeAggregations/main/aggregate/AnimeToExternal.json) return 1056 tmdb tv entry, animes-lists got over 6600 tvdb id entry (including seasons and special mappings), and if looking only for season 1 (what you're doing here i think), it's got 3800 entry. I found 476 tmdb movie entry, animes-lists got 1600 entry for movies with an imdbid. Also what happen to seasons and episodes offsets, kometa currently don't use them, but they are needed to work on the seasons (that all other agent use and maybe kometa one day)

• Values are taken straight from AniDB (Values come from: https://anidb.net/anime/17906/resource and are compiled into a single file for easier matching.)
• Change also allows matching for tmdb/tv, which the hand-crafted list does not handle tmdb/tv
• If you check the code, the hand-craft list takes priority over the automated list. So it is not a list "switch", it is a list augmentation.
• The only thing that uses episode and season offset seems to be TVDb, and AniDb does not directly support TVDb.
• Priority for shows is still TVDb, IMDb, and now TMDb/tv

[Arial-Z]

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

[salomj]

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

That should be easy enough.

[salomj]

As Sohjiro said i think the best would be to add them to kometa animes-id repo, with key like tmdb_tv and tmdb_movie so they could be used as fallback if tvdb or imdb fail.

Kometa-Team/Anime-IDs#1

[salomj]

@meisnate12 Why was this closed?

[meisnate12]

its been added to Kometa in a different way