Am I the only one noticing a significant slow done in (building embeddings) new version? #595
Comments
|
Similar issues have been reported in the chat. I started making some changes to address this. The changes are now available in While the changes don't exactly speed things up, they will prevent any really long-running batches. 🌴 |
|
Hmm. Did not fix it, still gets stuck at 600 then goes slow. Also now won't accept my Gemini API KEY... Says invalid no matter which I try. |
|
Looks like Claude also broken.
|
|
@bbecausereasonss wow, sorry to hear that you're having so many issues Is anything being logged to the console when stuck at 600? Any batch that takes longer than 10 seconds should dump which files are taking too long in the console. Both Gemini and Anthropic should work after disabling and re-enabling the plugin. It has to do with the settings not being properly reflected after changing models, I'm working on a fix now. And, also with Gemini, I've been having more issues because of their "1 per minute" API request limit for free accounts, which b basically makes it unusable in the Smart Chat when using notes as context. 🌴 |
|
I'm on Gemini's paid plan. Okay I'll try enable/disable. So the time issue is due to ingesting over 1k embeddings (separately). Although that fixed the single large embedding corruption issue it introduced a super slow to create any embedding or load issue. |
|
Also no console logs during pause at 600... just a pause, then it continues. |
|
The "making smart connections" isn't doing any embedding. It's simply scanning for changes to trigger embeddings in the subsequent process. I'm also not sure I understand what you're saying here
|
|
I see. Interesting I thought it was scanning all of the premade embeddings.
Not sure why the scanning is taking so long. Especially on a blading fast
ssd. 😀
Sent from Mobile.
…On Fri, May 10, 2024, 5:56 PM WFH Brian ***@***.***> wrote:
The "making smart connections" isn't doing any embedding. It's simply
scanning for changes to trigger embeddings in the subsequent process.
I'm also not sure I understand what you're saying here
So the time issue is due to ingesting over 1k embeddings (separately).
Although that fixed the single large embedding corruption issue it
introduced a super slow to create any embedding or load issue.
—
Reply to this email directly, view it on GitHub
<#595 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE2TIUASIIG2XNFTZIMWF7TZBU7CJAVCNFSM6AAAAABHPCHIM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGMYTOMJXGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Could be a bottle neck of RAM, which even if you have a lot of it, the underlying electron architecture implements an arbitrary limit, which can be raised but it would be reasonable for Obsidian still to maintain a lower limit for lack of perceived need. I should be able to offload this to the Smart Connect software, similar to what's done with the embeddings, to improve the performance that way. There are also other things I can try to optimize in the implementation, but adding it into Smart Connect will likely be first because it also serves a few other things I'm working on. 🌴 |
|
I've got 64ram and 24 vram
Sent from Mobile.
…On Fri, May 10, 2024, 9:23 PM WFH Brian ***@***.***> wrote:
Especially on a blading fast
ssd. 😀
Could be a bottle neck of RAM, which even if you have a lot of it, the
underlying electron architecture implements an arbitrary limit, which can
be raised but it would be reasonable for Obsidian still to maintain a lower
limit for lack of perceived need.
I should be able to offload this to the Smart Connect software, similar to
what's done with the embeddings, to improve the performance that way.
There are also other things I can try to optimize in the implementation,
but adding it into Smart Connect will likely be first because it also
serves a few other things I'm working on.
🌴
—
Reply to this email directly, view it on GitHub
<#595 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE2TIUC6HYP2R7JDLG74OFDZBVXHPAVCNFSM6AAAAABHPCHIM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGQZTCOJVGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
|
Hi, I have the similar issue. I get the message "Making smart connections", the counter is stuck for some time at 730-740 of 1655, freezing the interface for a while, then quickly goes through all the notes up to 1655 and works. This re-scan happens every time Obsidian starts (and sometimes when new notes are created or existing notes rearranged). It seems at a first glance that some embeddings data might became corrupted and cannot be properly read by the plugin. The message. Screenshot from the console. |
|
Following the issue. Interesting though, why does the loader has trouble with some particular notes. I also noticed, that all Cyrillic characters in the note filenames are replaced by underlines in smart_connections note file names. Can this cause any trouble later? |
|
Mine is still oddly pausing at around 600 mark with no real logs :/ Should I just nuke my folder and redo all the embeddings? |
|
Redid them, did not make a difference :( |
Slow-Example.mp4
Everytime I open Obsidian, the SC plugin now takes about 2-3x as long during the initial ingest period. 600 (stuck around here) to 1000 notes. Used to just fly, but the new version I'm experiencing a significant slow down, really not sure what it's caused by.
The text was updated successfully, but these errors were encountered: