Make queries fast, filter all flexible attributes #5240
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will be help with testing each of the documents which do not any more depend on the 'global' `current_app` and `request`. These two can now be provided at the time the objects are instantiated.
|
Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry. |
3 tasks
Use `json_group_object` SQLite function to aggregate flexible attributes into `flex_attrs` field. Register SQLite converter `json.loads` to automatically convert the JSON string to a Python dictionary. Remove the code that had this task previously.
For a flexible attribute query, replace the `col_name` property with a function call that extracts that attribute from the `field_attrs` field introduced in the earlier commit. Additionally, for boolean, numeric and date queries CAST the value to NUMERIC SQLite affinity to ensure that our queries like 'flex:1..5' and 'flex:true' continue working fine. This removes the concept of 'slow query', since every query for any field now has an SQL clause.
Unify query creation logic from - queryparse.py:construct_query_part, - Model.field_query, - DefaultTemplateFunctions._tmpl_unique to a single implementation under `LibModel.field_query` class method. This method should be used for query resolution for model (flex)fields. Allow filtering item attributes in album queries and vice versa by merging `flex_attrs` from Album and Item together as `all_flex_attrs`. This field is only used for filtering and is discarded after.
It seems like previously filtering by flexible attributes did not work
- I'd receive '{"data": []}' trying to GET `/aura/tracks?filter[play_count]=11`
Now this works, not only for tracks, but for `/aura/artists` and
`/aura/albums` too.
Additionally, this improves `/aura/tracks` response time significantly.
I tried loading the default of 500 tracks from my library:
On `master`, it took ~20s
After this commit, it takes under 1s.
snejus
force-pushed
the
only-fast-filtering
branch
from
ad2ea5a
to
9ceffb6
Compare
|
I'm using the |
snejus
force-pushed
the
only-fast-filtering
branch
from
3c293fd
to
bdb7fd9
Compare
snejus
commented
May 9, 2024
| @@ -1019,7 +1019,7 @@ def find_duplicates(self, lib): | |||
| # temporary `Item` object to generate any computed fields. | |||
| tmp_item = library.Item(lib, **info) | |||
| keys = config["import"]["duplicate_keys"]["item"].as_str_seq() | |||
| dup_query = library.Album.all_fields_query( | |||
| dup_query = library.Item.match_all_query( | |||
There was a problem hiding this comment.
It seemed to me this was supposed to be Item instead of Album?
|
I'll be able to review in a week or two, just end of semester push at the moment. |
In order to include the table name for fields in this query, use the `field_query` method. Since `AnyFieldQuery` is just an `OrQuery` under the hood, remove it and construct `OrQuery` explicitly instead.
snejus
force-pushed
the
only-fast-filtering
branch
from
bdb7fd9
to
070c87f
Compare
|
Hey, also just chiming in to say that it will take some time for me to go through the current batch of PRs. I won't be able to keep up the response times I had last week, but I'll slowly work through all of them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Another and (hopefully) final attempt to improve querying speed.
Fixes #4360 #3515 and possibly more issues to do with slow queries.
This PR supersedes #4746.
What's been done
The
albumanditemtables are joined, and corresponding data fromitem_attributesandalbum_attributesis merged and made available for filtering. This enables to achieve the following:beet list -a path::some/pathbeet list play_count:10beet list -a title:somethingbeet list artpath:coverbeet list -a art_source:somethingBenchmarks
You can see that now querying speed is more or less constant regardless of the query, and the speed is mostly influenced by how many results need to be printed out
Compare this with what we had previously
To Do
docs/to describe it.)docs/changelog.rstnear the top of the document.)Later
lslimit