Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Adding a playlist to the download queue doesn't work if the playlist has been subscribed to before #634

Closed
2 tasks done
zombachu opened this issue Dec 16, 2023 · 9 comments
Labels
bug Something isn't working

Comments

@zombachu
Copy link

I've read the documentation

Operating System

Synology

Your Bug Report

Describe the bug

I had a playlist I was subscribed to but was behind in archiving by a couple months, so I wanted to sync it all at once by adding it to the download queue. However, none of the missing videos were added and the playlist unsubscribed itself and marked itself as deactivated too. Logs show TubeArchivist was able to get the playlist from the url I provided but it seemed to only check the videos it already had. Resubscribing to the playlist and doing rescan subscriptions didn't pick up the unarchived videos either. I'm able to add the videos individually manually so I don't think it's an access issue but I really really do not want to need to do this.

Steps To Reproduce

  1. Have a subscribed playlist in TubeArchivist that is not fully archived
  2. Add the playlist to the download queue
  3. sadness

Expected behavior

The unarchived videos to be added to the download queue

Relevant log output

[pid: 3297|app: 0|req: 5593/5593] 172.17.0.1 () {52 vars in 1275 bytes} [Sat Dec 16 01:17:20 2023] POST /api/download/ => generated 104 bytes in 34 msecs (HTTP/1.1 400) 8 headers in 279 bytes (1 switches on core 0)
[2023-12-16 01:18:05,372: INFO/MainProcess] Task extract_download[f8510c88-5047-4262-ad0b-fe3c9ea0c60c] received
�[1;33m[2023-12-16 01:18:05,645: WARNING/ForkPoolWorker-2] extract_download create callback�[0m
�[1;33m[2023-12-16 01:18:19,162: WARNING/ForkPoolWorker-2] PLiasgr1hOV7crPuf1h0AzZOFUQqtOcI5H: get metadata from es�[0m
�[1;33m[2023-12-16 01:18:19,434: WARNING/ForkPoolWorker-2] K7_XGfcytPQ: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,435: WARNING/ForkPoolWorker-2] du0TU1ME6dk: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,436: WARNING/ForkPoolWorker-2] vJAJkaEQDXg: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,436: WARNING/ForkPoolWorker-2] PYNYpkZLhXY: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,436: WARNING/ForkPoolWorker-2] GnZCka4je4M: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,437: WARNING/ForkPoolWorker-2] OCLODPBxXyQ: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,437: WARNING/ForkPoolWorker-2] ccNn0aCu9QI: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,437: WARNING/ForkPoolWorker-2] s0rsOOD3RfQ: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,437: WARNING/ForkPoolWorker-2] OZIWgFgiFTI: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,438: WARNING/ForkPoolWorker-2] 1cXl4rAgGJY: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,438: WARNING/ForkPoolWorker-2] JSBw8twpEgI: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,439: WARNING/ForkPoolWorker-2] 6hb0o7rbMMo: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,439: WARNING/ForkPoolWorker-2] ppygXaXcOY8: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,439: WARNING/ForkPoolWorker-2] hn9LsLYKors: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,440: WARNING/ForkPoolWorker-2] B4GfL5wrjqs: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,440: WARNING/ForkPoolWorker-2] iWQB6SwzuE0: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] ekHm3ncQBlo: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] 0q2hdt4rLIo: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] Dud97ap37BU: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] G2uoPnS1oTE: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] dj7IFtaGZ_8: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,441: WARNING/ForkPoolWorker-2] 33uj9819wRc: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,442: WARNING/ForkPoolWorker-2] gjIi0zVvmP4: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,442: WARNING/ForkPoolWorker-2] iCAxJvWq-Uk: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,442: WARNING/ForkPoolWorker-2] rWKg8a0CEIY: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,442: WARNING/ForkPoolWorker-2] wVZNFz2cbx0: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,442: WARNING/ForkPoolWorker-2] io2hD2gj5mw: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,443: WARNING/ForkPoolWorker-2] rYk11hS_qM4: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,443: WARNING/ForkPoolWorker-2] 9oMx8ayW8UE: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,444: WARNING/ForkPoolWorker-2] 6z5wesZKuiI: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,444: WARNING/ForkPoolWorker-2] VU_Y1il1Cdk: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,444: WARNING/ForkPoolWorker-2] HI1-PJ7Sy1w: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,445: WARNING/ForkPoolWorker-2] CA_bmeTGZgE: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,446: WARNING/ForkPoolWorker-2] W5gXd1FzLK4: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,446: WARNING/ForkPoolWorker-2] 1eCre8nIlJ4: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,446: WARNING/ForkPoolWorker-2] XYH_lu5R3H0: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,447: WARNING/ForkPoolWorker-2] -RqrZrcVCOk: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,449: WARNING/ForkPoolWorker-2] qopdmnl_Riw: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,450: WARNING/ForkPoolWorker-2] EgJMMUgcvIo: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,451: WARNING/ForkPoolWorker-2] yMUz9efa1yE: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,452: WARNING/ForkPoolWorker-2] Vo9vuhsrItc: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,453: WARNING/ForkPoolWorker-2] wUrZNG7v5nA: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,454: WARNING/ForkPoolWorker-2] 8nu2qOdUzxQ: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,454: WARNING/ForkPoolWorker-2] Hw9RsOXyCqA: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,454: WARNING/ForkPoolWorker-2] JlrTu4ee8J0: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,455: WARNING/ForkPoolWorker-2] 2DhPCtjrYWc: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,455: WARNING/ForkPoolWorker-2] HFFgepMA6mg: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,455: WARNING/ForkPoolWorker-2] bm0nLJuRNbw: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,456: WARNING/ForkPoolWorker-2] kXsOynpQ9RM: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,456: WARNING/ForkPoolWorker-2] afc53yDtc64: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,456: WARNING/ForkPoolWorker-2] Mka0oUCh6XI: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,457: WARNING/ForkPoolWorker-2] tuADSGLjLB8: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,458: WARNING/ForkPoolWorker-2] Kpes_5zYy5c: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:19,458: WARNING/ForkPoolWorker-2] AbPgkv9x-dU: skipped adding already indexed video to download.�[0m
�[1;33m[2023-12-16 01:18:21,499: WARNING/ForkPoolWorker-2] PLiasgr1hOV7crPuf1h0AzZOFUQqtOcI5H: get metadata from es�[0m
�[1;33m[2023-12-16 01:18:32,129: WARNING/ForkPoolWorker-2] UCHVijOMzF7t5Ujust6Z65cw: get metadata from es�[0m
�[1;33m[2023-12-16 01:18:47,785: WARNING/ForkPoolWorker-2] f8510c88-5047-4262-ad0b-fe3c9ea0c60c success callback�[0m
[2023-12-16 01:18:47,789: INFO/ForkPoolWorker-2] Task extract_download[f8510c88-5047-4262-ad0b-fe3c9ea0c60c] succeeded in 42.15202007535845s: None
�[1;33m[2023-12-16 01:18:47,789: WARNING/ForkPoolWorker-2] f8510c88-5047-4262-ad0b-fe3c9ea0c60c return callback�[0m

Anything else?

Current playlist status, shows "Last refreshed: 2023-10-15" despite my efforts above
Screenshot 2023-12-16 at 1 34 22 AM

Playlist in TubeArchivist
Screenshot 2023-12-16 at 3 09 16 AM

Playlist in YouTube (unarchived videos highlighted)
Screenshot 2023-12-16 at 3 07 47 AM

@zombachu
Copy link
Author

I'm not exactly sure what the conditions are for it to fail like this so title is probably wrong

@bbilly1
Copy link
Member

bbilly1 commented Dec 21, 2023

Are these videos downloaded and just not in the playlist?

Also check your ignore list, to check you haven't ignored it before.

@bbilly1 bbilly1 added the question Further information is requested label Dec 21, 2023
@lamusmaser
Copy link
Collaborator

Checking in with @zombachu to confirm if this is now resolved for you.

@zombachu
Copy link
Author

zombachu commented Jan 6, 2024

Are these videos downloaded and just not in the playlist?

Also check your ignore list, to check you haven't ignored it before.

The videos haven't been downloaded or ignored.

Checking in with @zombachu to confirm if this is now resolved for you.

Haven't tried on 0.4.5 yet but is still an issue on 0.4.4. Will test and report back when I have some time.

@zombachu
Copy link
Author

zombachu commented Jan 7, 2024

@lamusmaser Still present on 0.4.5.

Do you guys want any logs or other info from me before I try deleting and readding the playlist in tubearchivist?

@salmonrumpnova
Copy link

salmonrumpnova commented Jan 20, 2024

I'm getting the same issue. It'll work the first time if you delete the playlist and re-add to the queue (without subscribing). But later when you add the playlist to the queue again, it'll parse the playlist and say it's adding the new videos to the queue in the docker logs, but the videos never shows up on the actual queue for some reason.

@lamusmaser
Copy link
Collaborator

Let's get some complete logs to review.

svuong-dev added a commit to svuong-dev/tubearchivist that referenced this issue Mar 3, 2024
Addresses the bug tubearchivist#634 
inside playlist.py
```
    def build_json(self, scrape=False):
...
        if scrape or not self.json_data:
```
self.json_data will only be empty if the playlist has never been indexed. Since `scrape` defaults to `False`, both conditions mean subsequent playlist downloads will never do any scraping. Adding `scrape=True` to the parent call fixes this scenario (tested on my own setup). 

Note that playlists don't seem to update via subscriptions scans still. I suspect we need a similar fix there, but I don't have the bandwidth to dig into that atm.
@bbilly1
Copy link
Member

bbilly1 commented May 11, 2024

I'm able to reproduce this. It gets confused because the playlist is already indexed, need to handle that separately.

@bbilly1 bbilly1 added bug Something isn't working and removed question Further information is requested labels May 11, 2024
@bbilly1 bbilly1 added the pending-release Fixed and pending release label May 11, 2024
@bbilly1 bbilly1 removed the pending-release Fixed and pending release label May 22, 2024
@bbilly1
Copy link
Member

bbilly1 commented May 22, 2024

That should now be fixed with v0.4.8, please update.

@bbilly1 bbilly1 closed this as completed May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants