v0.4.8

Hotfix published

There is a hotfix published for ARM64 builds where the ffmpeg binaries weren't downloaded correctly into the container. If you are affected, pull again.

Project updates

• This ships various fixes for playlist refresh limitations. Also see the docs here clarifying that functionality.
• This fixes a comment extraction problem due to a change on YT
  • See below for a manual fix for your comments
• This also partially fixes failed channel extraction, we are waiting for upstream fix, that’s tracked here
  • See below to manually trigger a refresh of your failed channels
• This ships a new schedule management system, fixing various previous limitations. Your existing schedules and notification URLs should migrate automatically at first start, monitor your startup logs and validate on your schedule settings page that everything transferred correctly.

Added

• Added delete and ignore for videos, docs #286
• Added new notification hook for add to queue task, #714
• Added thumbs clean up in Thumbnail Check task
• Added 404 handler for detail views
• Added better error handling for invalid urls
• Added a few unit tests

Changed

• Changed build to use python script to download best ffmpeg build
• Changed schedule store to use beats model, no longer requiring restart to apply schedule, docs
• Changed task notification management, docs
• Changed playlist matching logic in download post processing, docs
• Changed queue handling to use Redis as temporary store, #500
• Changed queue handling progress and position tracking to use score in ordered set
• Changed to cleaner and more concise seconds to date parser

Fixed

• Fixed comment extraction
• Fixed playlist subscription bool logic in template, #684
• Fixed playlist rescanning logic, ensuring refresh, #686
• Partial fix channel subs extraction with workaround, #701, upstream link
• Fixed adding existing playlist to queue again, #634
• Fixed playlist matching not removing deleted videos
• Fixed rescan negative total notification
• Fixed schedule cron parser, you are no longer able to enter an invalid schedule
• Fixed logout
• Fixed application startup dependency making tests unrunnable, #564

Manual updates

You don’t need to run these commands, the regular metadata refresh should fix these problems, at least for active videos. That’s just if you want to fix it now.

First command will “relink” comments with your video. Due to a failed comment extraction, that link got lost, but all your comments are still here. Run python manage.py ta_fix_comment_link in your Tube Archivist container, e.g.:

docker exec -it tubearchivist python manage.py ta_fix_comment_link

Depending on your index size and system performance, expect this to take around 15 seconds for 10k videos. You will see ✓ task completed once that’s done.

Second command will trigger a reindex for channels that have failed to extract before, run python manage.py ta_fix_channels in your Tube Archivist container, e.g.:

docker exec -it tubearchivist python manage.py ta_fix_channels

This will scan your channels and add faulty channels to the regular reindex queue. That should return in a few seconds, you will see ✓ task queued. Then you can monitor the reindex progress on your settings page and in your docker logs.

v0.4.7

Project Updates

• This will trigger a playlist migration in ElasticSearch, adding a playlist_type field. As always that happens at first start and should be automatic.
• This updates yt-dlp to the latest release. After a change on YT, comment extraction needs to be modified, that is tracked here. Please be patient.

Added

• Added user created custom playlists, by @greg321321, docs and docs
• Added better error message for cookie errors, by @Gentoli

Changed

• Changed disable video autoplay on the video detail page to avoid too many continue watching items.
• Changed first watch progress is now only sent after 10 seconds to avoid too many continue watching items.

Fixed

• Fixed failed empty subtitle return from YT, #663
• Fixed empty comment author return for some old comments, by @dot-mike
• Fixed edge cases for failed channel title building
• Fixed download progress bar for very small videos, by guarantee sending 100% progress
• Fixed download progress messages getting called multiple times in DOM
• Fixed failed legacy channel migration trigger by removing ta_migpath from startup, #687

v0.4.6

Project Updates

• This is a small maintenance release fixing some bugs
• This has a workaround for a memory usage problem in Celery, the task scheduler

Fixed

• Fixed a memory leak in celery by reducing the worker life time, by @PhuriousGeorge
• Fixed an issue where deleting a channel wouldn't clean up subtitles correctly
• Fixed reindexing item that has since been deleted locally after queue start
• Fixed throttling video redirect when reindexing by validating IDs and raising error on mismatch

v0.4.5

Project Updates

• This is a small release, mainly shipping small but crucial fixes
• Getting this out, then we can take our time for the next additions

Changed

• Changed temporary queue building interface, use redis sorted set instead of lists

Fixed

• Fixed version check comparison over multiple version diff
• Fixed an issue where version_check schedule builder will reset unexpectedly, #611, by @lamusmaser
• Fixed upstream channel about page parser, #614
• Fixed upstream like_count extraction for videos
• Fixed mobile scaling issue in chrome based browsers, #616
• Fixed auto delete day parsing incompatible data types, #622
• Fixed author_is_uploader extraction in comments after yt-dlp API change
• Fixed incompatibility between celery and redis library versions by downgrading redis

Hotfix

If you have updated within approximately the first 24h under certain circumstances you may have gotten an error during the version check at startup. Pull the image again, if you are affected.

v0.4.4

Project Updates

• This fixes various mapping problems, triggering a rebuild of the ta_video and ta_playlist index
  • As always, this should be mostly automatic, if not see here.
• This fixes an unfortunate problem where the periodic refresh task wouldn't add outdated videos

Added

• Added additional sort by duration and file size, #559, by @spechter
• Added fallback handling for missing keys in manual import, #581, by @lamusmaser
• Added additional tiles to the dashboard, changed dashboard layout
• [API] Added video stats endpoint
• [API] Added channel stats endpoint
• [API] Added playlist stats endpoint
• [API] Added download stats endpoint

Changed

• [API] Removed previous /api/stats/primary/ endpoint in favor of more detailed dedicated endpoints
• Changed footer text alignment, by @derekslenk

Fixed

• Fixed query not matching outdated documents for reindex
• Fixed reindex outdated pagination error
• Fixed comment extraction for members only videos not passing cookie correctly, #596
• Fixed auto delete watched on new blank index without watched videos, #592
• Fixed mapping problem for some playlists, #605
• Fixed various spelling errors on settings pages, by @pairofcrocs

v0.4.3

Project Updates

• This is the second step in moving away from redis persistence, refactoring the environment variable parsing
• This ships a workaround for the new channel about popup parsing failure

Added

• Added read only user functionality, docs, #529 by @stratus-ss
• Added store for volume change in session
• Added CSS styling functionality, docs, by @extome9000
• Added daily download media size to dashboard
• [API] Added zip file backup endpoints
• [API] Added user config endpoints

Changed

• Changed env var handling, Refactored Environment settings, by @anonamouslyginger
• Changed dashboard tile building, by @MerlinScheurer
• Changed daily download stats to be timezone aware
• Touch up the icons, fix coloring for theme, by @pairofcrocs

Fixed

• Fixed channel about page parsing, #587
• Fixed direct video download button, #566 by @OmarLaham
• Fixed off by one message in filesystem rescan

v0.4.2

Project update

• First step towards moving away from redis plugins and redis persistence. At application startup this will migrate your user configurations from Redis to ES.

Added

• Added search score to full text search results, docs
• Added mark as unwatched button for playlists and channels, docs and docs, #495 by @froseph
• Added env vars for wider range of ES configurations, docs, #462 by @ajgon
  • ES_SNAPSHOT_DIR: File path for where the snapshots will get stored
  • ES_DISABLE_VERIFY_SSL: Disable SSL verification for ES
• Added health check endpoint, example in compose, by @ajgon
• Added forward authentication to support auth proxies, docs, #403 by @ajgon
• Added years to duration strings for aggregations

Changed

• Changed biggest channel stats to three different sorted tables, by @MerlinScheurer
• Changed and consolidated duration parser to produce human readable string
• Changed and completed search processor and serializer to use API throughout
• Changed user configuration to store in ES, first step moving away from Redis plugins, by @anonamouslyginger
• Changed ES wrapper for better dynamic args builder
• Changed json backup to split into multiple smaller files, #406
• Old wiki is now removed, in favor of docs

Fixed

• Fixed URLs to settings docs, #545 by @MerlinScheurer
• Fixed yt-dlp playlist extraction
• Fixed various yt-dlp errors with newest release, #522

v0.4.1

Project updates

• Due to an unfortunate misunderstanding between epoch seconds and milliseconds, this will trigger an index rebuild
  • Depending on your archive size, this can take anywhere from a few seconds to a few minutes to complete.
  • Be patient, follow the logs and let it do its thing.
• Tube Archivist Companion, the browser extension also got an update to v0.2.0 with lots of great additions. See the release notes there.

Added

• Added archive statistic dashboard
• [API] Added playlist delete endpoint
• [API] Added channel ID search endpoint
• [API] Added archive aggregation endpoints
• Added f hotkey for fullscreen to video player, by @dmynerd78

Changed

• Changed settings pages, are now split up for easier navigation, by @anonamouslyginger
• Changed redis config save to trigger bgsave
• Changed to ignore eaDir files from synology, #523
• Changed to reset autostart videos when stopping queue, #520

Fixed

• Fixed final thumbnail error handling
• Fixed validation error when subscribing to video in playlist, #299
• Fixed download folder list not using ingore_filelist
• [API] Fixed sponsorblock endpoint not returning 404, #526
• Fixed wrong index of epoch seconds/milliseconds in ES

v0.4.0

Project updates, potentially Breaking Changes

• After trying and repeatedly failing to correctly handle Title and Channel Names for media files, we have decided to standardize that to <channel-id>/<video-id>.mp4.
  • Before updating, make sure Tube Archivist has all media files at the expected file system location.
  • There is a migration at startup that will reorganize all your files.
  • Only the default naming convention is supported for the migration. If you made changes there, this will likely fail and require you to migrate your files manually.
  • Monitor your startup logs for a progress message. Any errors will log there. If you see messages similar to did not find expected video/subtitle at ..., rename that manually to the expected path and restart for the migration to complete.
  • Also see FAQ.
• There is now a Jellyfin integration released: tubearchivist/tubearchivist-jf
  • This repos is renamed from tubearchivist/jellyfin to avoid confusion, that this is not a fork of jellyfin...
  • Shoutout to @jonasrosland for taking the initiative for the docker build
• The Docs now got a dedicated URLs page to document how TA treats URLs from YouTube, docs.
• As mentioned previously, Redis now publishes ARM builds for all it's extensions, including RedisJSON. To avoid confusion, I'll delete our now outdated custom build bbilly1/rejson, use redis/redis-stack-server instead.

Added

• Added Apprise integration for task notifications, docs
• Added days to duration converter
• Added error message for failed downloads with additional controls, docs
• Added new screenshots and showcasing, by @pairofcrocs
• Added support for case insensitive manual import, by @lamusmaser, #502
• [API] Added version number to ping API, by @lamusmaser

Changed

• Changed file system organization to <channel-id>/<video-id>.mp4.
• Changed trusted host configuration to add localhost and tubearchivist to defaults.
• Changed comment extraction to ignore errors by default, so the rest of comments can continue.
• [API] Date strings are now formatted in ISO format for consistency.

Fixed

• Fixed rescan process initiation without task
• Fixed startup race condition error in some cases
• Fixed retry logic for failing download videos, #477
• Fixed reindex progress message when dynamically adding to queue, #478
• Fixed handling missing tv art in thumbnail validator, #479
• Fixed redis retry logic at application start, #485
• Fixed missing auto subtitle extraction error handling, #496
• Fixed download cache file name building for better resume logic, #498

v0.3.6

Project Updates

• This update will automatically change and rebuild the video, channel and download queue indexes.
• Tube Archivist Companion browser extension also got an update to control auto start behavior: Release Notes
• If you are a sponsor, the real time monitor client also got an update to control auto start behavior: Release Notes
• It is recommend to run Rescan Filesystem to validate all media file paths before updating, monitor the logs to make sure, this does nothing unexpected.
• At first start, there is a migration command running to index additional metadata from your media files. That includes file size, codec, bitrate, resolution.
  • That process can take some time, depending on various factors, expect this to take around 1 minute per 1000 videos.
  • Log output will show progress and any error messages.
  • The interface will become available again, after that completes.
  • Be patient and grab some popcorn to watch the logs fly by.

Added

• Added video stream metadata indexing like codecs, bitrate, filesize
• Added channel metadata aggregation like total file size, total videos, total playback.
• Added start now for adding to download queue, docs
• Added auto start for subscriptions, docs
• Added extractor language configuration, docs
• Added --format-sort configuration, docs, by @dsander
• Added channel tags indexing for better search results
• [API] Added endpoints to control auto start behavior

Changed

• Changed channel metadata extraction to use yt-dlp instead of custom scraper for better reliability.
• Removed the limit_count config field, use queue control instead

Fixed

• Fixed backup run issue when not initiated with task
• Fixed playlist ID parser for members only playlists, by @mglinski

Hotfix

• I've pushed a quick hotfix dealing with #476, if you encounter that, please pull again.