[Blocking]: List of things to do before working on new features

[bbilly1]

Background: This project has grown very fast, too fast, new features were implemented at the cost of reduced maintainability and reliability of the project. Functionality wise, this project has a lot going for it, now its time to revisit some of these early flaws, so:

This is a potentially none complete list of blocking things to figure out, before working on additional major features. Outlined solutions may or may not be good ideas.

• Fix startup process and checks
  • Create manage.py function to check environment
  • Create manage.py function to apply custom migrations
  • All single threaded, no locks needed, track exit status to stop startup on errors
• Fail prove process locks
  • Track process ids produced by Celery to be able to check if they are running independent from locks
• Enforce snapshots
  • Guarantee snapshots are configured and enabled on startup, disable json backup task
• Complete search processor
  • Finalize rewrite serialize classes for dynamically process values from ES
  • Move all functionality to API
• Complete API
  • Move all form and buttons POST request to API endpoints
• Split up settings page into separate sub pages
• Transfer queue storage to redis
  • Load all tasks queues into redis for celery to take from one by one
• Rewrite cron schedule validator
  • Use Celery to validate input first before storing
• Refactor Subscribe functionality
  • Make distinction between Playlist and Channel
  • Surface error notifications through new task backend
• Refactor get next item to download
  • Call get next on the ES queue to get whole video context
  • Use new task backend to interact with the queue
  • Implement priorities and auto download in ES
• Simplify media path building
  • Standardize media paths to /<channel-id>/<video-id>.mp4
  • Fix all these incompatibilities in one go
• Remove all usecase of Redis plugins
  • That has broken too many times, it's time to just stick to standard redis
  • make all redis data transient, use only ES for persistence.

---

Additionally, not strictly blocking but high priority:

• At least a minimal automatic testing suite that can be extended
• Create useful content for https://www.tubearchivist.com
• Prepare project for a All in one version:
  • Figure out how to pack this as efficiently as possible
  • Figure out automated build process
  • Make container internal base paths configurable
  • Make ES auth optional
• Implement the notification broker on https://members.tubearchivist.com to collect webhooks from YouTube for realtime subscription notifications.

---

This list will be updated as we go. Expect this to take as long as it takes. Any help with this is highly welcome. Reach out on Discord to coordinate.

If you want to work on new features or fixes not listed here, reach out on Discord first, to make sure your code doesn't interfere with problems outlined above.

[lamusmaser]

I'll help where I'm able and have the time.

[lamusmaser]

Should we add "Allow videos to fail to download, but continue the queue" with exception handling as an additional marker, as noted in #307? I know we don't want a ton of tasks here, but that seems to be one of the overarching problems that is causing the download lock to become stuck.
It could be replaced with "Allow exception handling with video failure to catch failure and remove lock before stopping download queue", too. Either way, we should be able to capture the failure, throw the exception into the logs, then either move on through the queue or fail out of the queue gracefully.

[bbilly1]

That is implemented on expected errors that can't be recovered from, e.g. if the video became unavailable after it got added to the queue but before it's downloaded, in that case, the queue will continue:

https://github.com/tubearchivist/tubearchivist/blob/master/tubearchivist/home/src/download/yt_dlp_handler.py#L197-L199

Additionally there are timeouts and retries from yt-dlp too, to recover from small networking interruptions. But the more severe problems like, connection gets interrupted beyond the timeouts and retries, or a container disappears, I think it makes sense to stop the queue there. I mean if the internet is gone, it makes no sense to try again with the next video.

But to recover from that, that's where Fail prove process locks on the task list comes in. Not sure if tracking the task IDs is the right approach, but that was the first thought coming to mind.

[anonamouslyginger]

I'm going to take a look at moving the config persistence to ES (and hopefully removing the need for the Redis JSON plugin).
I've been burned too many times by Redis breaking things so I'm invested.

It's a chunky bit of work so it might take me a bit of time.
Toying with the idea of just creating a draft PR so everyone can see progress...

[lamusmaser]

Sounds good. You are free to have the draft PR and also welcome to join the Discord and discuss on our #help-contribute channel.

For the most part, we should be able to re-initialize Redis on each TA restart. That should mean that all configurations should be in ES first, then TA should push those configurations across to Redis for ease of access during runtime. Changes to those configurations should then first be pushed to ES and then re-initiate a pull to Redis.

[bbilly1]

Most of that is now completed. Or at least on the way. There is still work to be done to migrate away from the redis plugins, we'll tackle that going forward.