Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure IO is properly closed when importing NewPipe subscriptions #4346

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ChunkyProgrammer
Copy link
Contributor

File.open accepts a second argument which will close the initial stream (IO::Memory.new(body)) when the File stream closes.

I also made it so that the code is a bit more safe with closing connections. ex: if an error occurs. This may also fix some potential memory leaks

@ChunkyProgrammer ChunkyProgrammer requested a review from a team as a code owner December 20, 2023 05:25
@ChunkyProgrammer ChunkyProgrammer requested review from syeopite and removed request for a team December 20, 2023 05:25

user.watched += db.query_all("SELECT url FROM streams", as: String)
.map(&.lchop("https://www.youtube.com/watch?v="))
temp = File.tempfile(".db") do |tempfile|
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we maybe want to randomly generate part of the file file's name? (ex: .db-{UUID}) I imagine this might cause some issues if invidious starts to use Concurrency

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that might be a huge problem! The ideal option would be ti use in-memory databases and use sqlite3_serialize, but that requires to make a PR to crystal-sqlite3

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be even better! I imagine there might need to be a PR made here as well (if serialization is supported by other databases): crystal-db

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In memory-databases are already supported in crystal-sqlite3 like so:

DB.open "sqlite3::memory" do | db |
  db.exec "create table contacts (name text, age integer)"
  db.exec "insert into contacts values (?, ?)", "John Doe", 30
end

The only thing that needs to be added is a way to actually access an in-memory database.

There's no need for a PR directly imo, an issue would be enough.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SamantazFox SamantazFox added the need-testing This feature needs to be deployed and tested to see if it's working, and doesn't break something label Jan 7, 2024
Invidious::Database::Users.update_subscriptions(user)
end
end
temp.delete
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realized that the file will not be deleted in the case of a database error (i.e if DB.open raises an exception). You'll need to wrap the entire thing in a begin / (rescue) / ensure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need-testing This feature needs to be deployed and tested to see if it's working, and doesn't break something
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants