New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create and use a separate table (parts_by_lcsc) to index FTS5 parts table by rowid to speed up. #444
base: main
Are you sure you want to change the base?
Conversation
…id instead of `LCSC Part` to significantly speed up performance. +3.3% DB size.
Can we avoid rebuilding the list to reduce the impact of list rebuilding being expensive? I get having a separate index table and considered such things. Imo it isn't worth the complexity if the cost of a few seconds is only when users alter the sort list of a component list with 100+ components. But again, I'm likely biased. |
I don't think it's possible if we want to keep the component list synchronised to the database. Even at a cost of a major refactor, you'll still need to do it every time the plugin starts, which is quite annoyingly slow right now. In the view of the FTS5 change, which expanded the database four-fold (and if you download the second time, watch for the multi-gigabyte bak file), I think a db size increase of 3.3% and an extra SELECT is not such a big thing. |
I second the observation that plugin startup has been very slow since the fts5 change. The plugin also freezes for about 20 seconds whenever I select a part which sort of defeats the purpose of fts5 in the first place. Search as you type is no fun if you then have to wait for the changes to commit. I was just in the process of doing some profiling to troubleshoot the issue but it looks like @gyohng may have found it already. (I'm using MacOS and KiCAD 8 BTW) |
Agree any freezes should be eliminated or reduced.
For the freezing, what is the action that results in the freezing for you?
I’m assuming it’s something other than changing the parts sort order.
…On Wed, Apr 24, 2024 at 1:02 PM Caleb Whiting ***@***.***> wrote:
I second the observation that plugin startup has been very slow since the
fts5 change. The plugin also freezes for about 20 seconds whenever I select
a part which sort of defeats the purpose of fts5 in the first place. (I'm
using MacOS and KiCAD 8 BTW)
I was just in the process of doing some profiling to troubleshoot the
issue but it looks like @gyohng <https://github.com/gyohng> may have
found it already.
—
Reply to this email directly, view it on GitHub
<#444 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJH4AFQEC3PGHRBNEN3JD3Y67QQ5AVCNFSM6AAAAABGRLPCMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGQZDQNZUGQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
The slowness seems to come from the Time taken in the
|
That helps. I'll take a look to see what might be going on there. We know
the fts5 stuff is a lot of overhead (in exchange for the free text
searching), there may be some straightforward wins to cut that time down on
those steps you mention taking a long time.
In terms of performance, I wrote a rust program that does profiling
against the old and new databases, and various versions in between,
relative to search queries, and it was helpful. I'll do the same here for
these paths.
…On Wed, Apr 24, 2024 at 2:21 PM Caleb Whiting ***@***.***> wrote:
The slowness seems to come from the populate_footprint_list function. I
added some log statements to print the execution time whenever the function
was called. Here are some measurements I got in standalone mode:
Time taken in the populate_footprint_list function:
- Startup - 15.94s
- Click "Select part" from the part picker dialog - 13.74s
- Sort by stock in the main window - 13.40s.
—
Reply to this email directly, view it on GitHub
<#444 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJH4ACEQFFZLQ4EUYPEBITY67ZZ5AVCNFSM6AAAAABGRLPCMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGU3DEMBXGY>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
Hi! :) I am actually a fan of your previously submitted PR with LIKE optimisation, but now FTS5 took over... This PR is essentially a replica of 5eb59e2. The main problem with 5eb59e2 (indexing by I went even as far as creating an index on the internal |
Hi @gyohng! I thought I already replied but I can't find the comment so I'll reply again. I think this comes down to workflow. You and I make heavy use of parametric search, so #439 was really all we needed and FTS5 has a high bar to meet. @chmorgan and probably a lot of other people prefer a single search-box interface hence the project going in that direction. But I think we all agree that the UI freezing for 15 seconds or more every time a part is selected is not acceptable to anyone's workflow. You may be interested in the discussion under #451, where we are trying to figure out where the slowdown is coming from. And #450 which is a totally diffferent idea for a parts database. |
The purpose of this PR is to increase the speed of any operation that involves list rebuilding, which is basically any operation, such as part selection, sorting, hiding/showing BOM parts, or even invoking the plugin window.
The database increase due to this is +3.3%, which I think is a reasonable trade-off until a better way is found to integrate it with FTS5 directly while maintaining the speed.
The problem is illustrated in this YouTube video (the principal part of the commit is initially commented out and then uncommented to show the difference in performance impact).
Youtube video with demonstrating the problem
The commit includes the following changes:
Creation of a New Table (
parts_by_lcsc
): A new table is created in the database to hold mappings between part identifiers (partsId
) and theLCSC Part
numbers. This table is designed to optimise lookup times by indexing parts based on theirLCSC Part
number rather than through direct queries on theparts
table, which seems to be slow.Modification in
get_part_details
Function:partsId
values from the newparts_by_lcsc
table using the providedLCSC Part
numbers.partsId
, it then fetches the parts details directly using these IDs, which is much faster as it leverages the row IDs for direct access instead of scanning through text fields.LCSC Part
numbers from theparts
table.Database Indexing in the
download
Method:download
method is the population of theparts_by_lcsc
table and the removal of an old index if it exists.parts
table, inserts or updates the mapping in theparts_by_lcsc
table, and updates a progress indicator, which communicates the operation's progress to the user interface.lcsc
column of theparts_by_lcsc
table to speed up the lookups.User Interface Updates: The code updates a gauge control that reflects the progress of the database indexing operation. On a modern M1 MacBook Pro it finishes under 5 seconds and is significantly faster than the downloading.