Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TubeSync skipping all downloads #478

Open
martinkeat opened this issue Feb 29, 2024 · 23 comments
Open

TubeSync skipping all downloads #478

martinkeat opened this issue Feb 29, 2024 · 23 comments

Comments

@martinkeat
Copy link

Tubesync 14.3 in Docker, logs show the following. All videos from all sources (set to download 1080p - or next best format/codec - VP9, Opus)

2024-02-29 08:55:56,896 [tubesync/WARNING] Download task triggered for media: 1yfuCpcX7EU (UUID: 7a037a27-498e-44fa-a6b9-a9a06d6f2c0f) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:56,930 [tubesync/WARNING] Download task triggered for media: q98vhT0QhFM (UUID: 83a06363-b801-4301-ada6-82a1c6238003) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,032 [tubesync/WARNING] Download task triggered for media: 3FeserOHtgg (UUID: b1613aba-e13c-4678-a580-03c36e920a05) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,089 [tubesync/WARNING] Download task triggered for media: Gjzhi6aHU74 (UUID: 0d72f1d9-4e05-406c-974a-e32465cf74d5) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,150 [tubesync/WARNING] Download task triggered for media: lloVQPM8TOY (UUID: c81e9b58-3e44-4919-9658-821298f71c00) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,209 [tubesync/WARNING] Download task triggered for media: dqo2DxUBb-o (UUID: 6451a90c-75d6-40ab-b724-a13baed9e376) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,249 [tubesync/WARNING] Download task triggered for media: 05S6p-HkrtY (UUID: 80c52d99-4871-47f3-918a-a14cc0bcfbec) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,320 [tubesync/WARNING] Download task triggered for media: lC1UG_hrJeo (UUID: 6ad21505-0a39-4971-b489-61550cd1b4fe) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,362 [tubesync/WARNING] Download task triggered for media: 5wMQNIK9OIk (UUID: fb47e570-7495-4349-aa0c-22d506bb2469) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,448 [tubesync/WARNING] Download task triggered for media: rKHgSM-IK3s (UUID: 4a0b40b6-5a63-4cee-827c-d07487d89628) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,476 [tubesync/WARNING] Download task triggered for media: Xvz6MEOtwyo (UUID: 47adc696-0be4-41b6-b04b-d35d21024293) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,514 [tubesync/WARNING] Download task triggered for media: s0RK_LomzWg (UUID: 81feace4-08d1-4285-9800-19d9983517d6) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,578 [tubesync/WARNING] Download task triggered for media: TaD3SgPuccE (UUID: 865f6d4a-f6fa-4aa6-8801-bace69cdbb58) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,584 [tubesync/WARNING] Download task triggered for media: 15WDBuvovXo (UUID: 40796111-378b-4367-93d7-1eab2c3ab05b) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,687 [tubesync/WARNING] Download task triggered for media: r0yI78cFeVs (UUID: 2e9b26c3-5987-44a9-a698-c8b7e6c220a1) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,742 [tubesync/WARNING] Download task triggered for media: PB4MXzB-GZ4 (UUID: fcb9b84a-fbeb-46ba-b838-99d133aa7535) but it is now marked to be skipped, not downloading thumbnail
2024-02-29 08:55:57,842 [tubesync/WARNING] Download task triggered for media: Sw_jVHpGcjg (UUID: 98c5e7c7-2be7-4ca3-9333-ab503e22d30b) but it is now marked to be skipped, not downloading thumbnail
@martinkeat
Copy link
Author

Settings for one of the sources:

`

Fallback | Get next best resolution or codec instead -- | -- Skipping? |   Can download? |   Available formats | ID: sb3ID: sb2ID: sb1ID: sb0ID: 233ID: 234ID: 599 , audio:mp4a.40.5 @30.819k / 22050HzID: 600 , audio:opus @32.293k / 48000HzID: 139 , audio:mp4a.40.5 @48.816k / 22050HzID: 249 , audio:opus @47.248k / 48000HzID: 250 , audio:opus @61.261k / 48000HzID: 140 , audio:mp4a.40.2 @129.505k / 44100HzID: 251 , audio:opus @117.514k / 48000Hz (matched)ID: 597 , 144p (188x144), fps:13, video:avc1.4d400b @24.903kID: 602 , (188x144), fps:13.0, video:vp09.00.10.08 @74.014kID: 598 , 144p (188x144), fps:13, video:vp9 @17.875kID: 269 , (188x144), fps:25.0, video:avc1.4D400B @138.26kID: 160 , 144p (188x144), fps:25, video:avc1.4D400B @35.954kID: 603 , (188x144), fps:25.0, video:vp09.00.11.08 @153.937kID: 278 , 144p (188x144), fps:25, video:vp09.00.11.08 @64.746kID: 229 , (312x240), fps:25.0, video:avc1.4D400D @241.054kID: 133 , 240p (312x240), fps:25, video:avc1.4D400D @73.429kID: 604 , (312x240), fps:25.0, video:vp09.00.20.08 @251.379kID: 242 , 240p (312x240), fps:25, video:vp09.00.20.08 @67.968kID: 230 , (468x360), fps:25.0, video:avc1.4D4015 @622.653kID: 134 , 360p (468x360), fps:25, video:avc1.4D4015 @153.527kID: 18 , 360p (468x360), fps:25, video:avc1.42001E @282.633k , audio:mp4a.40.2 @nonek / 44100HzID: 605 , (468x360), fps:25.0, video:vp09.00.21.08 @586.728kID: 243 , 360p (468x360), fps:25, video:vp09.00.21.08 @117.215kID: 231 , (626x480), fps:25.0, video:avc1.4D401E @1027.605kID: 135 , 480p (626x480), fps:25, video:avc1.4D401E @285.895kID: 606 , (626x480), fps:25.0, video:vp09.00.30.08 @700.298kID: 244 , 480p (626x480), fps:25, video:vp09.00.30.08 @163.235k (matched) Matched formats | Combined: no matchAudio: 251 (exact match)Video: 244 (fallback) `

@meeb
Copy link
Owner

meeb commented Feb 29, 2024

Can you screenshot the status for one of the media items that are skipped? E.g. /media/7a037a27-498e-44fa-a6b9-a9a06d6f2c0f

@martinkeat
Copy link
Author

image
image
image

@martinkeat
Copy link
Author

image
image
image

@meeb
Copy link
Owner

meeb commented Mar 1, 2024

Your media items have failed to download enough times that they have been marked as failed and skipped. Make sure you've updated tubesync to the latest release and try using the "unskip" button on a single item of currently skipped media and see if that works.

As to why the downloads originally failed, no idea this could be because tubesync needed an update or a network connection issue or disk was full or YouTube limited your IP etc. etc.

@martinkeat
Copy link
Author

Hi @meeb

I actually googled the issue first before coming here. Manual skipping and unskipping does not resolve the issue. I tried a number from each source. Also, the version is the latest version and is set to update automatically each night.

The IP address might be something though. I'll check that and get back to you.

@meeb
Copy link
Owner

meeb commented Mar 1, 2024

You can test yt-dlp directly, drop into the tubesync container with:

$ docker exec -ti tubesync bash

Then you can test it with:

$ cd /downloads
$ yt-dlp "https://www.youtube.com/watch?v=1yfuCpcX7EU"

If that errors then it's your connection or some other global issue.

@martinkeat
Copy link
Author

Hi @meeb - sorry for the late reply. Ran that. It did not error. So not what we thought. Output from the terminal is below.

FYI, I am running it on unraid so didn't need to docker exec to drop into the container.

#  cd /downloads
# yt-dlp "https://www.youtube.com/watch?v=1yfuCpcX7EU"
[youtube] Extracting URL: https://www.youtube.com/watch?v=1yfuCpcX7EU
[youtube] 1yfuCpcX7EU: Downloading webpage
[youtube] 1yfuCpcX7EU: Downloading ios player API JSON
[youtube] 1yfuCpcX7EU: Downloading android player API JSON
[youtube] 1yfuCpcX7EU: Downloading m3u8 information
[info] 1yfuCpcX7EU: Downloading 1 format(s): 616+251
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 56
[download] Destination: Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f616.mp4
[download] 100% of   86.85MiB in 00:00:27 at 3.19MiB/s
[download] Destination: Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f251.webm
[download] 100% of    4.54MiB in 00:00:00 at 7.28MiB/s
[Merger] Merging formats into "Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].webm"
Deleting original file Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f616.mp4 (pass -k to keep)
Deleting original file Sade - Your Love Is King (Live 2011) [1yfuCpcX7EU].f251.webm (pass -k to keep)
#

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

When you unskip a skipped video are there any errors on your tasks page?

@martinkeat
Copy link
Author

@meeb Yes.

image

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

That's the media item page not the tasks page.

@martinkeat
Copy link
Author

Sorry. Wrong screenshot

image

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

Did you specifically unskip that YouTube item? jDvhwHSnUJQ? "No valid format available" means, assuming you don't have crazy source requirements, that the metadata is missing or invalid. Anything on the container logs themselves?

@martinkeat
Copy link
Author

Yup. I did it there and then. Hence why you see the "downloading thumbnail" directly below the error.

Source requirements aren't out there. Standard VP9 with OPUS 1080p set to get the next available quality or codec. Thats pretty much it.

@martinkeat
Copy link
Author

martinkeat commented Mar 3, 2024 •
edited by meeb

Log:

2024-03-03 09:19:17,205 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:01:26 later at 2024-03-03 09:20:43.212297+00:00
Unauthorized: /
2024-03-03 09:20:47,470 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:04:21 later at 2024-03-03 09:25:08.477666+00:00
10.0.10.11 - davie [03/Mar/2024:09:19:04 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:05 +0000] "GET /tasks HTTP/1.1" 200 4825 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:10 +0000] "GET /tasks HTTP/1.1" 200 3405 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:13 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:19 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:21 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:20:02 +0000] "GET / HTTP/1.1" 401 120 "-" "-"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/fontawesome/fa-regular-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-regular.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-bold.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

@martinkeat
Copy link
Author

martinkeat commented Mar 3, 2024 •
edited by meeb

Tried a different one:

sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:04:21 later at 2024-03-03 09:25:08.477666+00:00
10.0.10.11 - davie [03/Mar/2024:09:19:04 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:05 +0000] "GET /tasks HTTP/1.1" 200 4825 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:10 +0000] "GET /tasks HTTP/1.1" 200 3405 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:13 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:19 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:19:21 +0000] "GET / HTTP/1.1" 200 2395 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:20:02 +0000] "GET / HTTP/1.1" 401 120 "-" "-"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:19 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/fontawesome/fa-regular-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-regular.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:21:20 +0000] "GET /static/fonts/roboto/roboto-bold.woff HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
Unauthorized: /
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET / HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
Unauthorized: /tasks
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET /tasks HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
10.0.10.11 - - [03/Mar/2024:09:22:30 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
Unauthorized: /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47
10.0.10.11 - - [03/Mar/2024:09:22:31 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
Unauthorized: /sources
10.0.10.11 - - [03/Mar/2024:09:22:31 +0000] "GET /sources HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:52 +0000] "GET /source/f7dda3c8-0cbe-49b8-971a-4c998c583a47 HTTP/1.1" 200 3378 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:52 +0000] "GET /static/fonts/fontawesome/fa-solid-900.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:53 +0000] "GET /static/fonts/fontawesome/fa-brands-400.woff2 HTTP/1.1" 304 0 "https://tubesync.methven.app/static/styles/tubesync.css" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:55 +0000] "GET /tasks HTTP/1.1" 200 2773 "https://tubesync.methven.app/source/f7dda3c8-0cbe-49b8-971a-4c998c583a47" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:22:57 +0000] "GET /sources HTTP/1.1" 200 2449 "https://tubesync.methven.app/tasks" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:00 +0000] "GET /media HTTP/1.1" 200 2113 "https://tubesync.methven.app/sources" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:01 +0000] "GET /static/images/nothumb.png HTTP/1.1" 304 0 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:02 +0000] "GET /media?show_skipped=yes HTTP/1.1" 200 10490 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:09 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 3658 "https://tubesync.methven.app/media?show_skipped=yes" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:16 +0000] "GET /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2222 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:17 +0000] "POST /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 302 0 "https://tubesync.methven.app/media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:17 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped HTTP/1.1" 200 2671 "https://tubesync.methven.app/media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:20 +0000] "GET /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2179 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:21 +0000] "GET /static/styles/tubesync.css HTTP/1.1" 200 34554 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:21 +0000] "GET /static/images/favicon.ico HTTP/1.1" 304 0 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:24,229 [tubesync/INFO] Scheduling task to download metadata for: https://www.youtube.com/watch?v=foIjI685ino
10.0.10.11 - davie [03/Mar/2024:09:23:24 +0000] "POST /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 302 0 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:23:24 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=enabled HTTP/1.1" 200 2727 "https://tubesync.methven.app/media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:27,750 [tubesync/INFO] Downloading media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) to: "/downloads/video/Sade/..mkv"
Rescheduling Downloading media for "foIjI685ino"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 410, in download_media
    format_str, container = media.download_media()
                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1405, in download_media
    raise NoFormatException(f'Cannot download, media "{self.pk}" ({self}) has '
common.errors.NoFormatException: Cannot download, media "df6438a5-240c-4eb3-a53f-0004c88dfdec" (foIjI685ino) has no valid format available
Rescheduling task Downloading media for "foIjI685ino" for 0:00:06 later at 2024-03-03 09:23:33.755498+00:00
10.0.10.11 - davie [03/Mar/2024:09:23:29 +0000] "GET /media-enable/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2176 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:29,894 [tubesync/WARNING] Media: Sade / foIjI685ino is older than 2 days, skipping
2024-03-03 09:23:29,903 [tubesync/INFO] Scheduling task to download thumbnail for: Sade - Cherish The Day (Lovers Live) from: https://i.ytimg.com/vi/foIjI685ino/maxresdefault.jpg
2024-03-03 09:23:29,910 [tubesync/INFO] Saved 81205 bytes of metadata for: Sade / df6438a5-240c-4eb3-a53f-0004c88dfdec
10.0.10.11 - davie [03/Mar/2024:09:23:31 +0000] "GET /media-skip/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 200 2227 "https://tubesync.methven.app/media/df6438a5-240c-4eb3-a53f-0004c88dfdec" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:23:32,856 [tubesync/WARNING] Download task triggered for media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading thumbnail
2024-03-03 09:23:37,887 [tubesync/WARNING] Download task triggered for  media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading

@martinkeat
Copy link
Author 

2024-03-03 09:23:37,887 [tubesync/WARNING] Download task triggered for  media: foIjI685ino (UUID: df6438a5-240c-4eb3-a53f-0004c88dfdec) but it is now marked to be skipped, not downloading
Unauthorized: /media/df6438a5-240c-4eb3-a53f-0004c88dfdec
10.0.10.11 - - [03/Mar/2024:09:24:55 +0000] "GET /media/df6438a5-240c-4eb3-a53f-0004c88dfdec HTTP/1.1" 401 120 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
2024-03-03 09:25:09,616 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:10:30 later at 2024-03-03 09:35:39.621931+00:00

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

Well, I have zero idea why command line yt-dlp works but calling it over the API doesn't but the reason your downloads are failing is because they're all marked as claimed under the YouTube copyright claimed:

2024-03-03 09:25:09,616 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy

@martinkeat
Copy link
Author

Dunno if it is relevant but I just tried deleting and re-adding the source: https://www.youtube.com/@SadeSME

And noticed that it seems to be inserting a /c/ in the URL for the channel.

image

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

Yeah you need to convert that @ handle to a channel ID using something like:

https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/

For example SadeSME is UCegKDBFMUSQNeWe4FAB7aig then add a source by channel ID and use UCegKDBFMUSQNeWe4FAB7aig as the ID.

But, that won't make the media items which are failing to download due to copyright claims magically work.

@martinkeat
Copy link
Author

Just one problem with that, you'll note the name of the person as sarah something or other. Well if I disable/enable love thy neighbour, which has nothing to do with sade, and is also way outside the fair use for the UK, it is still supposedly copyrighted by the same person:

 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:40 +0000] "GET /media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 200 2229 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:43 +0000] "POST /media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 302 0 "https://tubesync.methven.app/media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:43 +0000] "GET /media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=skipped HTTP/1.1" 200 2698 "https://tubesync.methven.app/media-skip/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:56 +0000] "GET /media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 200 2177 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=skipped" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:33:59,128 [tubesync/INFO] Scheduling task to download metadata for: https://www.youtube.com/watch?v=Ik1C0aYvqVk
10.0.10.11 - davie [03/Mar/2024:09:33:59 +0000] "POST /media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2 HTTP/1.1" 302 0 "https://tubesync.methven.app/media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
10.0.10.11 - davie [03/Mar/2024:09:33:59 +0000] "GET /media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=enabled HTTP/1.1" 200 2716 "https://tubesync.methven.app/media-enable/ab04eb31-cd13-4260-bce1-98bf1f2888f2" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
**2024-03-03 09:34:01,855 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy**
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:00:06 later at 2024-03-03 09:34:08.681956+00:00
2024-03-03 09:34:15,386 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:00:21 later at 2024-03-03 09:34:36.552793+00:00
10.0.10.11 - davie [03/Mar/2024:09:34:28 +0000] "GET /media HTTP/1.1" 200 2113 "https://tubesync.methven.app/media/ab04eb31-cd13-4260-bce1-98bf1f2888f2?message=enabled" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
2024-03-03 09:34:43,634 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy
Rescheduling Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2"
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/background_task/tasks.py", line 43, in bg_runner
    func(*args, **kwargs)
  File "/app/sync/tasks.py", line 289, in download_media_metadata
    metadata = media.index_metadata()
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/sync/models.py", line 1424, in index_metadata
    return indexer(self.url)
           ^^^^^^^^^^^^^^^^^
  File "/app/sync/youtube.py", line 89, in get_media_info
    raise YouTubeError(f'Failed to extract_info for "{url}": No metadata was '
sync.youtube.YouTubeError: Failed to extract_info for "https://www.youtube.com/watch?v=Ik1C0aYvqVk": No metadata was returned by youtube-dl, check for error messages in the logs above. This task will be retried later with an exponential backoff.
Rescheduling task Downloading metadata for "ab04eb31-cd13-4260-bce1-98bf1f2888f2" for 0:01:26 later at 2024-03-03 09:36:10.336515+00:00
10.0.10.11 - davie [03/Mar/2024:09:34:46 +0000] "GET /media?show_skipped=yes HTTP/1.1" 200 7094 "https://tubesync.methven.app/media" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

Yeah you need to convert that @ handle to a channel ID using something like:

https://www.streamweasels.com/tools/youtube-channel-id-and-user-id-convertor/

For example SadeSME is UCegKDBFMUSQNeWe4FAB7aig then add a source by channel ID and use UCegKDBFMUSQNeWe4FAB7aig as the ID.

But, that won't make the media items which are failing to download due to copyright claims magically work.

@meeb
Copy link
Owner

meeb commented Mar 3, 2024 •
edited

Yeah, they all have the same error in your logs:

2024-03-03 09:34:01,855 [tubesync/ERROR] ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy**

There's obviously not much tubesync can do about YouTube returning that error.

I also get that error with that video:

$ yt-dlp "https://www.youtube.com/watch?v=Ik1C0aYvqVk"
[youtube] Extracting URL: https://www.youtube.com/watch?v=Ik1C0aYvqVk
[youtube] Ik1C0aYvqVk: Downloading webpage
[youtube] Ik1C0aYvqVk: Downloading ios player API JSON
[youtube] Ik1C0aYvqVk: Downloading android player API JSON
ERROR: [youtube] Ik1C0aYvqVk: Video unavailable. This video is no longer available due to a copyright claim by Sarah Ann Kennedy

Basically your error is because the videos aren't available on YouTube.

@meeb
Copy link
Owner

meeb commented Mar 3, 2024

Oh and if your "love thy neighbour" tool works like a VPN it's entirely possible the copyright claim is regional and might not be present in some countries which is why that might impact it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@meeb @martinkeat