New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure IO is properly closed when importing NewPipe subscriptions #4346
base: master
Are you sure you want to change the base?
Ensure IO is properly closed when importing NewPipe subscriptions #4346
Conversation
src/invidious/user/imports.cr
Outdated
|
||
user.watched += db.query_all("SELECT url FROM streams", as: String) | ||
.map(&.lchop("https://www.youtube.com/watch?v=")) | ||
temp = File.tempfile(".db") do |tempfile| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would we maybe want to randomly generate part of the file file's name? (ex: .db-{UUID}) I imagine this might cause some issues if invidious starts to use Concurrency
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that might be a huge problem! The ideal option would be ti use in-memory databases and use sqlite3_serialize
, but that requires to make a PR to crystal-sqlite3
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that would be even better! I imagine there might need to be a PR made here as well (if serialization is supported by other databases): crystal-db
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In memory-databases are already supported in crystal-sqlite3 like so:
DB.open "sqlite3::memory" do | db |
db.exec "create table contacts (name text, age integer)"
db.exec "insert into contacts values (?, ?)", "John Doe", 30
end
The only thing that needs to be added is a way to actually access an in-memory database.
There's no need for a PR directly imo, an issue would be enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
src/invidious/user/imports.cr
Outdated
Invidious::Database::Users.update_subscriptions(user) | ||
end | ||
end | ||
temp.delete |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I realized that the file will not be deleted in the case of a database error (i.e if DB.open
raises an exception). You'll need to wrap the entire thing in a begin
/ (rescue
) / ensure
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done 😄
8c15384
to
67f98cb
Compare
File.open accepts a second argument which will close the initial stream (
IO::Memory.new(body)
) when the File stream closes.I also made it so that the code is a bit more safe with closing connections. ex: if an error occurs. This may also fix some potential memory leaks