Feature request: New tag system

[revam]

Abstract

The current tag system is too reliant on AniDB, and is not very user-customizable. This request is for an upgraded system to replace the current system.

The new system should be at minimum be:

• decoupled from any outside sources by default - The system should not be reliant on any outside sources, but it should still be possible to use outside sources to map tags to series, if the users so chooses.

• user-customizable - The user should have more freedom in how tags are added.

And optionally:

• provide plugin definitions - An abstraction of the new tag, an tag provider for pulling in tags from outside sources, and (optionally) a tag manager for managing the tags at a high-level (i.e. without using the repositories directly, but using an abstraction on top to hide them).

To-do

Some pointers to what needs to be done.

• Choose implementation to use (A, B, C, D, or something else entirety)

  • Add the basic models to Shoko.Models

  • Create new repositories to contain the models in the database in ShokoServer

• Modify the existing Anime_HTTP command to only add tags if a preference is set, and follow the mapping from Tag_Reference to the local tags.

  • Decide if it should either
    1. create new tags if no reference if found,
    2. try to search for a similar tag by name/path (depending on the implementation used), or
    3. skip the tag without mapping it.

• Add new endpoints to manage tags

  • Basic endpoints to create, read, update, and delete tags (The exact endpoints needed depends on the chosen implementation)

  • Basic endpoints to create, read and delete references. (May or may not be needed)

• Create database migration script/command/function to convert an existing database over to the new system, converting the records from AniDB_Tag and CustomTag to the new format in AnimeTag.

• Deprecate the current AniDB-based tag system and custom tag system. (Should be done last)

Suggested implementations

Shared details

Shared across all the implementations will be the reference table holding all references to and from outside source, and the tag/anime relation table.

If a tag provider want to integrate with Shoko, it should add its mappings toAnimeTag_Reference on first initialization, and later map it's tags to series using the mappings stored in the table. The table should be user-customizable, so a user can override mappings that a provider has set.

public class AnimeTag_Reference
{
    // Record identifier
    public int ID { get; set; }
    // The tag identifier
    public int TagID { get; set; }
    // Then name of the provider
    public string ProviderName { get; set; }
    // Can be anything that can uniquely identify the tag at the source, using the correct provider plugin
    public string ReferenceID { get; set; }
}

public class AnimeSeries_AnimeTag
{
    public int ID { get; set; }
    public int TagID { get; set; }
    public int AnimeID { get; set; }
    public bool IsLocalSpoiler { get; set; }
}

Example S1

Example containing four different references to the same local tag, using three different providers.

[
  {
    "ID": 1,
    "TagID": 1,
    "ProviderName": "AniDB",
    "ReferenceID": "404"
  },
  {
    "ID": 2,
    "TagID": 1,
    "ProviderName": "AniDB",
    "ReferenceID": "501"
  },
  {
    "ID": 3,
    "TagID": 1,
    "ProviderName": "Plugin1",
    "ReferenceID": "763d6b7c-0435-409e-8332-6422cd6a7c35"
  },
  {
    "ID": 4,
    "TagID": 1,
    "ProviderName": "AniList",
    "ReferenceID": "dystopia"
  }
]

Example S2

Example containing three different relations between a series and a tag, where the second relation also indicates that the tag will spoil the plot for the series, and is therefore marked as a local spoiler.

[
  {
    "ID": 1,
    "TagID": 1,
    "AnimeID": 1,
    "IsLocalSpoiler": false
  },
  {
    "ID": 2,
    "TagID": 2,
    "AnimeID": 1,
    "IsLocalSpoiler": true
  },
  {
    "ID": 3,
    "TagID": 3,
    "AnimeID": 1,
    "IsLocalSpoiler": false
  }
]

Implemention A - Non-relational (simple)

The tag system should implemented with simple tags, each with an unique id, an unique name, an optional description, a hidden flag, and a global spoiler flag.

public class AnimeTag
{
    public int ID { get; set; }
    public string Name { get; set; }
    public string? Description { get; set; }
    public bool IsGlobalSpoiler { get; set; }
    public bool IsHidden { get; set; }
}

Example A1

The dystopia tag can mean anything in the series, be it the theme, the setting, or both.

[
  {
    "ID": 1,
    "Name": "dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  }
]

Implementation B - Non-relational (scoped)

The tag system should implemented with scoped tags, each with an unique id, an unique name, an optional description, a hidden flag, and a global spoiler flag.

A slight twist to implementation A, where tags can be scoped as a relative path.

public class AnimeTag
{
    public int ID { get; set; }
    public string Name { get; set; }
    public string? Description { get; set; }
    public bool IsGlobalSpoiler { get; set; }
    public bool IsHidden { get; set; }
}

Example A1

The dystopia tag can mean anything in the series, be it the theme, the setting, or both. But the theme/dystopia tag will only have dystopia as a theme, and setting/dystopia tag will only have dystopia as the setting. All three can be independently used, either together or separately.

[
  {
    "ID": 1,
    "Name": "dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  },
  {
    "ID": 2,
    "Name": "theme/dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  },
  {
    "ID": 3,
    "Name": "setting/dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  }
]

Implementation C - Relation-tree (One-to-Many)

The tag system should implemented in a relation-tree like structure. Each tag should have an unique id, a non-unique parent id, a name, a full name, an optional description, a hidden flag, and a global spoiler flag. The parent id is stored directly on the object, and all new tags added without specifying a parent tag can added under a user-preferred tag (or no tag at all, which would add them under the root), e.g. creating a tag named uncategorized and setting it as the preferred tag would add all new tags without a parent set as direct descendants (children) of this tag.

The tag name is non-unique, but the fully scoped tag name is unique and dynamically created based on the name and the tag's ancestors' names. The scoped tag name is unique across all tags. So multiple tags with the same name is not allowed under the same parent.

public class AnimeTag
{
    public int ID { get; set; }
    public int ParentID { get; set; }
    public string Name { get; set; }
    public string Path { get; set; }
    public string? Description { get; set; }
    public bool IsGlobalSpoiler { get; set; }
    public bool IsHidden { get; set; }
}

Example C1

If the theme is dystopia (theme/dystopia), then you known that the setting is not dystopia (setting/dystopia), unless both tags are set -- since they can be set independently.

[
  {
    "ID": 1,
    "ParentID": 0,
    "Name": "theme",
    "Path": "theme",
    "Description": null,
    "IsGlobalSpoilder": false,
    "IsHidden": true
  },
  {
    "ID": 2,
    "ParentID": 0,
    "Name": "setting",
    "Path": "setting",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": true
  },
  {
    "ID": 3,
    "ParentID": 1,
    "Name": "dystopia",
    "Path": "theme/dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  },
  {
    "ID": 4,
    "ParentID": 2,
    "Name": "dystopia",
    "Path": "setting/dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": false
  }
]

Implementation D - Many-to-Many

Help needed!

public class AnimeTag
{
    public int ID { get; set; }
    public string Name { get; set; }
    public string? Description { get; set; }
    public bool IsGlobalSpoiler { get; set; }
    public bool IsHidden { get; set; }
}

public class AnimeTag_Relation
{
    public int ID { get; set; }
    public int TagID { get; set; }
    public int ParentTagID { get; set; }
}

Example D1

The tags:

[
  {
    "ID": 1,
    "Name": "theme",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": true
  },
  {
    "ID": 2,
    "Name": "setting",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": true
  },
  {
    "ID": 3,
    "Name": "dystopia",
    "Description": null,
    "IsGlobalSpoiler": false,
    "IsHidden": true
  }
]

The relations:

[
  {
    "ID": 1,
    "TagID": 3,
    "ParentTagID": 1
  },
  {
    "ID": 1,
    "TagID": 3,
    "ParentTagID": 2
  }
]