New Videos Not Downloading, No apparent Log Error in Docker

[bwt615]

2024-02-11 22:15:29 {"_type": "url", "ie_key": "Youtube", "id": "OVnrL4we2NM", "url": "https://www.youtube.com/watch?v=OVnrL4we2NM", "title": "Angular Drag-and-Drop File Uploads to Firebase Storage", "description": "https://angularfirebase.com/lessons/drag-and-drop-file-uploads-to-firebase-storage/\n\nLearn how to create a dropzone-style file uploading feature in your Angular 4 app to Firebase Storage on...", "duration": 375.0, "channel_id": null, "channel": null, "channel_url": null, "uploader": null, "uploader_id": null, "uploader_url": null, "thumbnails": [{"url": "https://i.ytimg.com/vi/OVnrL4we2NM/hqdefault.jpg?sqp=-oaymwEbCKgBEF5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLBX7-xguIsyWX9jxQ4cf6VDeI_3AA", "height": 94, "width": 168}, {"url": "https://i.ytimg.com/vi/OVnrL4we2NM/hqdefault.jpg?sqp=-oaymwEbCMQBEG5IVfKriqkDDggBFQAAiEIYAXABwAEG&rs=AOn4CLDN0ht6ohTEVpnNIGXjIv8NoyihAQ", "height": 110, "width": 196}, {"url": "https://i.ytimg.com/vi/OVnrL4we2NM/hqdefault.jpg?sqp=-oaymwEcCPYBEIoBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLAK1xOo7-KdEyg0CKPZKoCDUKPAfA", "height": 138, "width": 246}, {"url": "https://i.ytimg.com/vi/OVnrL4we2NM/hqdefault.jpg?sqp=-oaymwEcCNACELwBSFXyq4qpAw4IARUAAIhCGAFwAcABBg==&rs=AOn4CLCGSp6LVK-Cyt2J8H6iWyQBDAK8OA", "height": 188, "width": 336}], "timestamp": null, "release_timestamp": null, "availability": null, "view_count": 38060, "live_status": null, "channel_is_verified": true, "__x_forwarded_for_ip": null, "webpage_url": "https://www.youtube.com/watch?v=OVnrL4we2NM", "original_url": "https://www.youtube.com/watch?v=OVnrL4we2NM", "webpage_url_basename": "watch", "webpage_url_domain": "youtube.com", "extractor": "youtube", "extractor_key": "Youtube", "playlist_count": 558, "playlist": "Fireship - Videos", "playlist_id": "UCsBjURrPoezykLs9EqgamOA", "playlist_title": "Fireship - Videos", "playlist_uploader": "Fireship", "playlist_uploader_id": "@fireship", "n_entries": 558, "playlist_index": 545, "__last_playlist_index": 558, "playlist_autonumber": 545, "epoch": 1707707729, "duration_string": "6:15", "_version": {"version": "2023.11.16", "current_git_head": null, "release_git_head": "24f827875c6ba513f12ed09a3aef2bbed223760d", "repository": "yt-dlp/yt-dlp"}}
error from daemon in stream: Error grabbing logs: invalid character 'l' after object key:value pair

[meeb]

This may need a yt-dlp update. This will be bundled into the latest release shortly.

[bwt615]

Awesome! Look forward to the release!

[AverTry]

I installed 13.4 yesterday, and it downloaded 12 video playlist and a 15 one, then I tried to do different lists each containing a few hundred. This morning only the directories were created but are empty, no errors. I tried again left it for a couple of hours and nothing?
Any help would be appreciated. Is youtube blocking larger pulls?

[meeb]

What's your tasks page showing?

[bwt615]

A restart seemed to fix, I also upped the workers and lowered the check frequency. I don’t know how it is parsing the channel data, but is there an opportunity to add an incremental sync that checks for new content in the last 24 hours? Then have a separate full sync?

[AverTry]

What's your tasks page showing?

[meeb]

OK, so you just have a massive number of scheduled tasks backlogged. I would strongly suggest you drop the worker count back to 1 as well, any more than that and it tends to annoy YouTube. Your backlog of tasks should be getting smaller. When you add a massive channel it has to slowly get each items metadata one item at a time which can take quite a while. Is the 3039 scheduled tasks going down in number?

[bwt615]

OK, so you just have a massive number of scheduled tasks backlogged. I would strongly suggest you drop the worker count back to 1 as well, any more than that and it tends to annoy YouTube. Your backlog of tasks should be getting smaller. When you add a massive channel it has to slowly get each items metadata one item at a time which can take quite a while. Is the 3039 scheduled tasks going down in number?

Isn't it two by default? I'm wondering if I should drop mine from 4...

[meeb]

Yes I would suggest you absolutely drop it from 4 to 1. The default is 1. I'll likely remove this option in the future and just fix it at 1.

[AverTry]

Tried to reset playlists, but the multiple playlists remained.
Deleted the db, then tried one large playlist of 458 videos, only 358 downloaded, upon closer inspection, from where my website displays by playlist, it shows 99 deleted videos and two private videos.
It tried downloading the private videos 9 times each, and the deleted, seem to just take up space in the TubeSync media tab.
I know in the you tube api, there is a way to differentiate, the private and deleted. So as an added feature request, it may be good to ignore adding these to the db, and just informing the end-user.
As it stands right now, thankyou for the heads up on one list at a time. But because of the network traffic still going on on this app, even though these videos are unavailable, I will download each playlist from a fresh db, thereby having a backup, that unfortunately cannot be scheduled. But at least I know I've got backups as YouTube seem to be deleting more and more stuff these days, as their restrictions or copyright become tighter.

Thank you for this app.

[meeb]

If you add a playlist that includes deleted or private videos there's not much else tubesync can do here really. The media item ID will be indexed then retrieving the metadata will fail. After enough failures it'll just mark the media item as failed and skip it.

tubesync, when calling yt-dlp, doesn't use the YouTube API at all. It effectively scrapes the front end of the public website. When a playlist is indexed there is no way to tell until you attempt to get an items metadata if it's private or deleted.

While deleted and private items will be visible in the front end, they won't stop any working media items from being downloaded and are just a visual annoyance.

[rpowel]

This may need a yt-dlp update. This will be bundled into the latest release shortly.

Updating to 0.13.4 fixed this issue for me, thanks!

[AverTry]

Thanks meeb, that makes sense. The playlists are just topical from the last 8 years, from different sources, not my own videos.

[meeb]

@bwt615 this issue has gone somewhat off-topic. Has updating tubesync fixed this for you?

[bwt615]

@bwt615 this issue has gone somewhat off-topic. Has updating tubesync fixed this for you?

Lol yes and no. I went down to 1 worker as suggested and have a pretty big scheduled backlog now.

Several downloads are failing now but that may have predated the worker downgrade.

Can't tell if this cause I had the workers up too high.

[meeb]

"Expected outfile does not exist" is a generic catch-all error for "we called yt-dlp, yt-dlp ran but when it finished the expected file doesn't exist on disk". This can be caused by basically anything, YouTube has blocked you, the media item isn't valid at YouTube, connection issues, DNS issues, your disk is full, etc. etc.

You can call yt-dlp manually in the container, probably the next advisable debug step.

$ docker exec -ti tubesync bash

Then:

$ cd /downloads
$ yt-dlp "https://www.youtube.com/watch?v=DSj9cM8KHFY"

See if that works directly without tubesync involved at all.

[bwt615]

I'm currently running a clean resync of my library with one worker, want to make sure it wasn't related to me upping to 4/8. I'll keep you posted! No errors so far.