Spotify: Improve song search/matching

[mgoltzsche]

The Spotify plugin doesn't match/find most songs.

Problem

Running this command in verbose (-vv) mode:

$ beet -vv spotify

Led to this problem:

user configuration: /data/beets/config.yaml
data directory: /data/beets
plugin paths: 
inline: adding item field first_artist
lastgenre: Loading canonicalization tree /etc/beets/genre-tree.yaml
lastgenre: Loading canonicalization tree /etc/beets/genre-tree.yaml
Sending event: pluginload
library database: /data/beets/library.db
library directory: /data/music
Sending event: library_opened
Parsed query: AndQuery([TrueQuery()])
Parsed sort: NullSort()
spotify: Processing 27 tracks...
spotify: Searching Spotify for 'Senegal Fast Food artist:Amadou & Mariam album:'
spotify: Found 0 result(s) from Spotify for 'Senegal Fast Food artist:Amadou & Mariam album:'
spotify: Searching Spotify for 'La Realite artist:Amadou & Mariam album:'
spotify: Found 0 result(s) from Spotify for 'La Realite artist:Amadou & Mariam album:'
spotify: Searching Spotify for 'Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile'
spotify: Found 0 result(s) from Spotify for 'Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile'
spotify: Searching Spotify for 'Deep Jungle Walk artist:Astrix album:'
spotify: Found 0 result(s) from Spotify for 'Deep Jungle Walk artist:Astrix album:'
spotify: Searching Spotify for 'Ephemeral artist:Bassnectar album:'
spotify: Found 0 result(s) from Spotify for 'Ephemeral artist:Bassnectar album:'
spotify: Searching Spotify for 'Street Blues artist:Bellaire album:'
spotify: Found 0 result(s) from Spotify for 'Street Blues artist:Bellaire album:'
spotify: Searching Spotify for 'Redemption artist:Boris Brejcha album:'
spotify: Found 0 result(s) from Spotify for 'Redemption artist:Boris Brejcha album:'
spotify: Searching Spotify for 'Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:'
spotify: Found 0 result(s) from Spotify for 'Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:'
spotify: Searching Spotify for 'Walkin' artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'Walkin' artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'Autumn Leaves artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'Autumn Leaves artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'So What artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'So What artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for ''Round Midnight artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for ''Round Midnight artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'All of You artist:Miles Davis album:Autumn Leaves'
spotify: Found 0 result(s) from Spotify for 'All of You artist:Miles Davis album:Autumn Leaves'
spotify: Searching Spotify for 'Ready or Not (Champion Bootleg) artist:Fugees album:'
spotify: Found 0 result(s) from Spotify for 'Ready or Not (Champion Bootleg) artist:Fugees album:'
spotify: Searching Spotify for 'Amanece artist:EttoreTechnoChannel album:'
spotify: Found 0 result(s) from Spotify for 'Amanece artist:EttoreTechnoChannel album:'
spotify: Searching Spotify for 'Free Tibet (Vini Vici remix) artist:Hilight Tribe album:'
spotify: Found 0 result(s) from Spotify for 'Free Tibet (Vini Vici remix) artist:Hilight Tribe album:'
spotify: Searching Spotify for 'Never Mind artist:Infected Mushroom album:Army of Mushrooms'
spotify: Found 2 result(s) from Spotify for 'Never Mind artist:Infected Mushroom album:Army of Mushrooms'
spotify: Most popular track chosen, count: 2
spotify: Searching Spotify for 'Franks artist:Infected Mushroom album:Legend of the Black Shawarma'
spotify: Found 1 result(s) from Spotify for 'Franks artist:Infected Mushroom album:Legend of the Black Shawarma'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'The Rope (Le Youth remix) feat. Polica artist:Lane 8 album:The Rope'
spotify: Found 1 result(s) from Spotify for 'The Rope (Le Youth remix) feat. Polica artist:Lane 8 album:The Rope'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'Mishaps Happening artist:Quantic album:'
spotify: Found 0 result(s) from Spotify for 'Mishaps Happening artist:Quantic album:'
spotify: Searching Spotify for 'The Ghost of Tom Joad artist:Rage Against The Machine album:'
spotify: Found 0 result(s) from Spotify for 'The Ghost of Tom Joad artist:Rage Against The Machine album:'
spotify: Searching Spotify for 'Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:'
spotify: Found 0 result(s) from Spotify for 'Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:'
spotify: Searching Spotify for 'The Alcoholic artist:Royksopp album:'
spotify: Found 0 result(s) from Spotify for 'The Alcoholic artist:Royksopp album:'
spotify: Searching Spotify for 'Back To Jazz artist:sometimes jah album:Just Feel'
spotify: Found 1 result(s) from Spotify for 'Back To Jazz artist:sometimes jah album:Just Feel'
spotify: Spotify track(s) found, count: 1
spotify: Searching Spotify for 'Blues artist:STAND HIGH PATROL album:'
spotify: Found 0 result(s) from Spotify for 'Blues artist:STAND HIGH PATROL album:'
spotify: Searching Spotify for 'STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:'
spotify: Found 0 result(s) from Spotify for 'STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:'
spotify: Searching Spotify for 'Lightweight artist:Total Science album:'
spotify: Found 0 result(s) from Spotify for 'Lightweight artist:Total Science album:'
spotify: 23 track(s) did not match a Spotify ID:
spotify: track: Senegal Fast Food artist:Amadou & Mariam album:
spotify: track: La Realite artist:Amadou & Mariam album:
spotify: track: Je pense a toi artist:Amadou & Mariam album:Sou Ni Tile
spotify: track: Deep Jungle Walk artist:Astrix album:
spotify: track: Ephemeral artist:Bassnectar album:
spotify: track: Street Blues artist:Bellaire album:
spotify: track: Redemption artist:Boris Brejcha album:
spotify: track: Vaunce of the Flowers (Waltz of the Flowers) artist:Tchaikovsky album:
spotify: track: Walkin' artist:Miles Davis album:Autumn Leaves
spotify: track: Autumn Leaves artist:Miles Davis album:Autumn Leaves
spotify: track: So What artist:Miles Davis album:Autumn Leaves
spotify: track: 'Round Midnight artist:Miles Davis album:Autumn Leaves
spotify: track: All of You artist:Miles Davis album:Autumn Leaves
spotify: track: Ready or Not (Champion Bootleg) artist:Fugees album:
spotify: track: Amanece artist:EttoreTechnoChannel album:
spotify: track: Free Tibet (Vini Vici remix) artist:Hilight Tribe album:
spotify: track: Mishaps Happening artist:Quantic album:
spotify: track: The Ghost of Tom Joad artist:Rage Against The Machine album:
spotify: track: Wake Up (Rasticles drum n bass remix) artist:Rage Against the Machine album:
spotify: track: The Alcoholic artist:Royksopp album:
spotify: track: Blues artist:STAND HIGH PATROL album:
spotify: track: STAND HIGH PATROL: Boat People artist:STAND HIGH PATROL album:
spotify: track: Lightweight artist:Total Science album:
spotify: 
https://open.spotify.com/track/5qVNRfUrsJToWrIsdxytec
https://open.spotify.com/track/6zTbuAt90GFVTn3F651K5u
https://open.spotify.com/track/7toata3kAxgvAPer5XCwHE
https://open.spotify.com/track/0lYGEP22F5i29vvFZ8yuMW
Sending event: cli_exit

-> Only 4 of 27 songs matched (14%).

However, there should be a lot more matches, if not all.
For instance Miles Davis' "Autumn Leaves" is definitely known by Spotify.

Something seems to be wrong with the way the Spotify search query is constructed. I briefly tried to run some of the generated queries against the Spotify API here and they returned proper results. Not sure why the Beets plugin isn't finding the songs. I suspect some difference in the way the Spotify query is constructed but I didn't spot it yet.

...
On the other hand I did not configure Spotify authentication myself but when using the Spotify API on the documentation page, I was logged with my personal account. Is it possible that certain songs can be found only when logged in using a personal account?

Setup

• OS: Alpine 3.19 container within an Ubuntu 22.02 host.
• Python version: 3.12.2
• beets version: 1.6.1 + latest commits on main branch
• Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

importfeeds:
    relative_to: !!binary |
        L2RhdGEvaW1wb3J0ZmVlZA==
    formats: m3u
    dir: /data/importfeed
    m3u_name: imported.m3u
    absolute_path: no
directory: /data/music
# --------------- Main ---------------

library: /data/beets/library.db
paths:
    default: Albums/%title{$albumartist}/$album%aunique{}/$track $title
    singleton: Singles/%title{$first_artist}/$title
    comp: Compilations/$album%aunique{}/$track $title

# --------------- Plugins ---------------

plugins:
- describe
- web
- webm3u
- webrouter
- beetstream
- types
- smartplaylist
- importfeeds
- edit
- play
- mbsync
- spotify
- fromfilename
- lastgenre
- autogenre
- ftintitle
- inline
- rewrite
- ihate
- chroma
- xtractor
- mpdstats
- convert
- bpm
- duplicates
- bareasc
- fuzzy
- info
- ytimport
- badfiles
- unimported
- random
webrouter:
    host: 0.0.0.0
    port: 8337
    routes:
        /:
            plugin: web
        /favicon.ico:
            plugin: webrouter.favicon
        /subsonic:
            plugin: beetstream
            config:
                never_transcode: yes
                playlist_dir: /data/playlists
        /m3u:
            plugin: webm3u
    cors: ''
    cors_supports_credentials: no
    reverse_proxy: no
    include_paths: no
    readonly: yes
beetstream:
    host: 0.0.0.0
    port: 8080
    never_transcode: yes
    cors: '*'
    cors_supports_credentials: yes
    reverse_proxy: no
    include_paths: no
    playlist_dir: ''
web:
    host: 0.0.0.0
    port: 8337
    reverse_proxy: yes
    readonly: yes
    cors: ''
    cors_supports_credentials: no
    include_paths: no
webm3u:
    host: 0.0.0.0
    port: 8339
    cors: ''
    cors_supports_credentials: no
    reverse_proxy: no
    include_paths: no
    playlist_dir:
    uri_format:

import:
    write: yes
    copy: yes
    move: no
    link: no
    hardlink: no
    incremental: yes
    quiet: no
    quiet_fallback: skip
    log: /data/beets/import.log
ytimport:
    directory: /data/ytimport

    import: yes
    reimport: no
    format: bestaudio/best
    url_file: ''
    likes: no
    max_likes: 15
    auth_headers:
    set: {}
    min_length: 60
    max_length: 7200
    max_length_nochapter: 900
    split_tracks: yes
    group_albums: yes
spotify:
    source_weight: 0.7
    mode: list
    tiebreak: popularity
    show_failures: yes
    artist_field: albumartist
    album_field: album
    track_field: title
    region_filter:
    regex: []
    client_id: 4e414367a1d14c75a5c5129a627fcab8
    client_secret: REDACTED
    tokenfile: spotify_token.json
xtractor:
    auto: no
    dry-run: no
    write: yes
    threads: 4
    force: no
    quiet: no
    keep_output: no
    keep_profiles: no
    output_path: /tmp/xtractor
    essentia_extractor: /usr/local/bin/essentia_streaming_extractor_music
    high_level_targets:
        genre_rosamerica_probability:
            path: highlevel.genre_rosamerica.probability
            type: float
        genre_electronic:
            path: highlevel.genre_electronic.value
            type: string
        genre_electronic_probability:
            path: highlevel.genre_electronic.probability
            type: float
        timbre:
            path: highlevel.timbre.value
            type: string
        tonal_atonal:
            path: highlevel.tonal_atonal.value
            type: string
        key_edma:
            path: tonal.key_edma.key
            type: string
        key_edma_scale:
            path: tonal.key_edma.scale
            type: string
        danceable:
            path: highlevel.danceability.all.danceable
            type: float
            required: yes
        gender:
            path: highlevel.gender.value
            type: string
            required: yes
        is_male:
            path: highlevel.gender.all.male
            type: float
        is_female:
            path: highlevel.gender.all.female
            type: float
        genre_rosamerica:
            path: highlevel.genre_rosamerica.value
            type: string
            required: yes
        voice_instrumental:
            path: highlevel.voice_instrumental.value
            type: string
            required: yes
        is_voice:
            path: highlevel.voice_instrumental.all.voice
            type: float
        is_instrumental:
            path: highlevel.voice_instrumental.all.instrumental
            type: float
        mood_acoustic:
            path: highlevel.mood_acoustic.all.acoustic
            type: float
            required: yes
        mood_aggressive:
            path: highlevel.mood_aggressive.all.aggressive
            type: float
            required: yes
        mood_electronic:
            path: highlevel.mood_electronic.all.electronic
            type: float
            required: yes
        mood_happy:
            path: highlevel.mood_happy.all.happy
            type: float
            required: yes
        mood_sad:
            path: highlevel.mood_sad.all.sad
            type: float
            required: yes
        mood_party:
            path: highlevel.mood_party.all.party
            type: float
            required: yes
        mood_relaxed:
            path: highlevel.mood_relaxed.all.relaxed
            type: float
            required: yes
        mood_mirex:
            path: highlevel.moods_mirex.value
            type: string
            required: yes
        mood_mirex_cluster_1:
            path: highlevel.moods_mirex.all.Cluster1
            type: float
        mood_mirex_cluster_2:
            path: highlevel.moods_mirex.all.Cluster2
            type: float
        mood_mirex_cluster_3:
            path: highlevel.moods_mirex.all.Cluster3
            type: float
        mood_mirex_cluster_4:
            path: highlevel.moods_mirex.all.Cluster4
            type: float
        mood_mirex_cluster_5:
            path: highlevel.moods_mirex.all.Cluster5
            type: float
    extractor_profile:
        highlevel:
            svm_models:
            - /var/lib/essentia/svm-models/beta5/danceability.history
            - /var/lib/essentia/svm-models/beta5/gender.history
            - /var/lib/essentia/svm-models/beta5/genre_rosamerica.history
            - /var/lib/essentia/svm-models/beta5/genre_electronic.history
            - /var/lib/essentia/svm-models/beta5/mood_acoustic.history
            - /var/lib/essentia/svm-models/beta5/mood_aggressive.history
            - /var/lib/essentia/svm-models/beta5/mood_electronic.history
            - /var/lib/essentia/svm-models/beta5/mood_happy.history
            - /var/lib/essentia/svm-models/beta5/mood_sad.history
            - /var/lib/essentia/svm-models/beta5/mood_party.history
            - /var/lib/essentia/svm-models/beta5/mood_relaxed.history
            - /var/lib/essentia/svm-models/beta5/moods_mirex.history
            - /var/lib/essentia/svm-models/beta5/voice_instrumental.history
            - /var/lib/essentia/svm-models/beta5/tonal_atonal.history
            - /var/lib/essentia/svm-models/beta5/timbre.history
            compute: 1
        outputFormat: json
        outputFrames: 0
        lowlevel:
            frameSize: 2048
            hopSize: 1024
        chromaprint:
            compute: 0
    low_level_targets:
        bpm:
            path: rhythm.bpm
            type: integer
            required: yes
        danceability:
            path: rhythm.danceability
            type: float
        beats_count:
            path: rhythm.beats_count
            type: integer
        average_loudness:
            path: lowlevel.average_loudness
            type: float
            required: yes
ftintitle:
    auto: yes
    drop: no
    format: feat. {0}
item_fields:
    first_artist: "import re\nm = re.match('^(.+?)\\\\s+(x\\\\s|\\\\(?(feat(\\\\.?|uring)|(ft|vs)\\\\.))', artist, flags=re.IGNORECASE)\nif m:\n  return m[1]\nreturn artist\n"
bareasc:
    prefix: '#'
fuzzy:
    threshold: 0.7
    prefix: '~'
chroma:
    auto: yes
autogenre:
    pretend: no
    all: no
    force: no
    lastgenre: yes
    xtractor: yes
    remix_title: yes
    parent_genres: yes
    from_title: yes
    genre_rosamerica_strong: 0.8
    genre_electronic_strong: 0.8
    genre_electronic_prepend: 0.5
    genre_electronic_append: 0.45
lastgenre:
    auto: no
    prefer_specific: yes
    count: 4
    source: album
    min_weight: 15
    canonical: /etc/beets/genre-tree.yaml
    whitelist: /data/beets/genres.txt
    fallback:
    force: yes
    separator: ', '
    title_case: yes
ihate:
    warn: ['artist:rnb']
    skip:
    - genre:Comedy
    - artist::^Adele$
    - artist::^AYLIVA$
    - artist::^Sia$
convert:
    never_convert_lossy_files: yes
    embed: no
    hardlink: no
    dest:
    pretend: no
    link: no
    threads: 16
    format: mp3
    id3v23: inherit
    formats:
        aac:
            command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
            extension: m4a
        alac:
            command: ffmpeg -i $source -y -vn -acodec alac $dest
            extension: m4a
        flac: ffmpeg -i $source -y -vn -acodec flac $dest
        mp3: ffmpeg -i $source -y -vn -aq 2 $dest
        opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
        ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
        wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
    max_bitrate:
    auto: no
    auto_keep: no
    tmpdir:
    quiet: no
    paths: {}
    no_convert: ''
    copy_album_art: no
    album_art_maxwidth: 0
    delete_originals: no
    playlist:
duplicates:
    delete: no
    album: no
    checksum: ''
    copy: ''
    count: no
    format: ''
    full: no
    keys: []
    merge: no
    move: ''
    path: no
    tiebreak: {}
    strict: no
    tag: ''
bpm:
    max_strokes: 3
    overwrite: yes
play:
    command: play-playlist
    use_folders: no
    relative_to:
    raw: no
    warning_threshold: 100
    bom: no
smartplaylist:
    auto: no
    output: extm3u
    fields: [id]
    playlist_dir: /data/playlists
    relative_to: /data/playlists
    playlists:
    -   name: all.m3u
        query: ''
    -   name: all-by-album-genre.m3u
        query: albumartist+ year+ album+ disc+ track+ genre+ artist+ title+
    -   name: jazz-blues.m3u
        query:
        - genre::^Jazz
        - genre::^Blues
        - genre:'Jazz Fusion'
        - genre:'Swing Jazz'
    -   name: reggae.m3u
        query: ['genres:Reggae', 'genres::''^Dub$''', 'genres:Ska']
    -   name: dub.m3u
        query: ['genre::''^Dub$''']
    -   name: ambient.m3u
        query: ['genre:Ambient']
    -   name: chillout.m3u
        query:
        - genre:Ambient
        - genre:Downtempo
        - genre:'Nu Jazz'
        - genre:'Trip Hop'
        - genre:Electronica
        - genre::'^Electro$'
        - genre:'Easy Listening'
    -   name: idm.m3u
        query: ['genre:Idm', 'genre:Breakcore']
    -   name: downtempo.m3u
        query: ['genre:Downtempo', 'genre:''Nu Jazz''']
    -   name: nu-jazz.m3u
        query: ['genre:''Nu Jazz''']
    -   name: world-music.m3u
        query:
        - genre:Blues
        - genres:African
        - genres:Afrobeat
        - genres:'Rhythm And Blues'
        - genres::'(^|, )Funk(,|$)'
        - genres::'(^|, )Soul(,|$)'
        - genres:'Caribbean And Latin American'
        - genre:Brazilian
        - genre:Balkan
        - genre:Klezmer
        - genres:'East European'
    -   name: african.m3u
        query: ['genres:African', 'genres:Afrobeat']
    -   name: afrobeat.m3u
        query: ['genres:Afrobeat']
    -   name: balkan-klezmer.m3u
        query: ['genre:''East European''', 'genre:Balkan', 'genre:Klezmer']
    -   name: classical.m3u
        query: ['genre:Classical']
    -   name: dnb.m3u
        query:
        - genres:'Drum And Bass'
        - genre:Breakbeat
        - genre:Jungle
        - genre:Neurofunk
        - genre:'Liquid Drum And Bass'
        - genre:'Ragga Drum And Bass'
    -   name: neurofunk.m3u
        query: ['genre:Neurofunk']
    -   name: jungle.m3u
        query: ['genre:Jungle']
    -   name: uk-garage.m3u
        query: ['genres:''Uk Garage''', 'genres:Dubstep', 'genre:Grime']
    -   name: dubstep-grime.m3u
        query: ['genre:''Dark Dubstep''', 'genre:Dubstep bpm:0..90', 'genre:Grime']
    -   name: electronic.m3u
        query: ['genres:Electronic']
    -   name: house-techno.m3u
        query:
        - genre:House
        - genre:Techno
        - genre:Minimal
        - genre:Progressive
    -   name: deep-house.m3u
        query: ['genre:''Deep House''', 'genre:''Nu Jazz'' genres:House']
    -   name: house.m3u
        query: ['genre:House', 'genre:''Nu Jazz'' genres:House']
    -   name: tech-house-minimal.m3u
        query: ['genre:''Tech House''', 'genre:Minimal']
    -   name: techno.m3u
        query: ['genre:Techno', 'genre:Minimal']
    -   name: minimal.m3u
        query: ['genre:Minimal']
    -   name: trance.m3u
        query: ['genre:Trance']
    -   name: goa.m3u
        query: ['genre:''Goa Trance''']
    -   name: ska.m3u
        query: ['genre:Ska']
    -   name: rock.m3u
        query:
        - genre::'^Rock'
        - genre::'^New Wave'
        - genre::'^Folk'
        - genre:'Fun Metal'
    -   name: punk-rock.m3u
        query: ['genre:Punk', 'genre:''Fun Metal''']
    -   name: pop.m3u
        query:
        - genre:Pop
        - genre:'New Wave'
        - genre:'Balkan Beats'
        - genre::'^Electro$'
        - genres::'^Electronic Rock, (?!Drum And Bass)'
    -   name: hiphop.m3u
        query: ['genre:''Hip Hop''']
    -   name: liked/1 or more.m3u
        query: play_count:1..
    -   name: liked/5 or more.m3u
        query: play_count:5..
    -   name: liked/loved.m3u
        query: loved:1
    -   name: latest/last day.m3u
        query: added- added:-1d..
    -   name: latest/last 3 days.m3u
        query: added- added:-3d..
    -   name: latest/last 7 days.m3u
        query: added- added:-7d..
    -   name: latest/last 14 days.m3u
        query: added- added:-14d..
    -   name: latest/last 30 days.m3u
        query: added- added:-30d..
    -   name: latest/last 60 days.m3u
        query: added- added:-60d..
    uri_format:
    forward_slash: no
    prefix: REDACTED
    urlencode: no
    pretend_paths: no
unimported:
    ignore_extensions: []
    ignore_subdirectories: []
rewrite: {}
pathfields: {}
album_fields: {}
mpd:
    music_directory: /data/music
    strip_path: ''
    rating: yes
    rating_mix: 0.75
    host: localhost
    port: 6600
    password: REDACTED
edit:
    albumfields: album albumartist
    itemfields: track title artist album
    ignore_fields: id path
describe:
    auto: no

[arsaboo]

I have had similar experiences. As discussed in #4997, we should move to the Spotipy library, which does a great job of interfacing with Spotify.