Problem with cookies.

[sirodus]

I've been using it for a very long time.
I use this to make download lists for videos from MANY channels every day. Recently, a cookie message pops up every time the browser opens, which destroys my script, and I have over 50 youtube channels from which I download videos every day and I have to confirm cookies every time when browser opens for every channel. Is there any solution?
I thought yt-videos-list would open a browser with my profile where I'm logged in instead of creating a "temporary profile". How to do it? I am asking for help or at least a little hint.

[shailshouryya]

Hey sirodus, thanks for starting this discussion!

Can you provide a screenshot of what's happening next time this cookie message pops up? I can add a check to see if any messages popup when yt-videos-list goes to the channel so that the program closes the message automatically, but the check would be a bit different depending on the type of message that pops up, for example:

### if the pop-up is a new window ###
driver.switch_to.window(driver.window_handles[1])
driver.close() # closes currently active window
driver.switch_to.window(driver.window_handles[0]) # switch back to main window

### if the pop-up is more like a text box overlaid on top of the page ###
driver.switch_to.alert.accept()

This would probably go in the logic.py file after loading the page with the driver.get(url) line. I found a Stack Overflow post that I think might be similar to the message you're getting here. When you get the cookie message, is the pop-up you see similar to the type pop-up the original poster there encounters?

Regarding the opening

browser with my profile where I'm logged in instead of creating a "temporary profile"

I don't think yt-videos-list actually opens a window where you're logged in, since selenium opens a new window that is separate from your regular Firefox/Opera/Safari/Chrome/Brave/Edge browser and then executes the code in this separate session. NOTE that this session is not shared with your regular, non-selenium session.

I'm curious about how this was working before for you though, since I never tested this program (or even thought about using it) with a session that had some settings enabled. When you were running the program before, could you tell if you were logged into your YouTube/Gmail account?

Also, another thing to note is if you run the program on many channels at once in a short period of time, YouTube might temporarily throttle the results/connection to your IP address. This usually goes away after a while (couple hours?), so maybe you can wait a bit and try again later. If this is what's happening, I'd suggest adding a sleep timer in your script after scraping, say every 10 channels, to sleep for a couple minutes before resuming so that it's less likely you get throttled by YouTube.

A final workaround that might work if the problem is your IP address being throttled is to use a VPN to connect to somewhere else before running the script every day, so that you're less likely to get throttled.

This is an interesting problem, and one that I haven't encountered before, so let me know what happens. 🙂 Also, if you're able to, can you attach a screenshot of the cookie message next time you run into it? I just tried running the program again and the program worked fine (I didn't get a cookie message), so this makes me think the problem might be because YouTube detects lots of activity from your IP address and is temporarily flagging it for bot-like activity.

[sirodus]

Thank you for your answer.
I attach ss the message that appears to me.
After accepting it, the program continues to run.
Unfortunately, my script will call yt-videos-list multiple times and I always have to accept cookies. Previously, this problem was not there, it started to appear about a week ago. I've always used this on a clean chrome browser which I don't use, I know selenium creates "temporary sessions" every time and doesn't save cookies and anything so this is why I get this message every time, but it wasn't like that before.
I have modified yt-videos-list so that selenium uses mine google chrome profile I'm logged on to solved my problem.
Google has already blocked my IP address several times. Now I am careful.

[shailshouryya]

Hey sirodus, thanks for attaching the screenshot! I also see this message when browsing on YouTube sometimes in my own browser (non-Selenium browser), but haven't seen this when running yt-videos-list.

Unfortunately, my script will call yt-videos-list multiple times and I always have to accept cookies. Previously, this problem was not there, it started to appear about a week ago. I've always used this on a clean chrome browser which I don't use, I know selenium creates "temporary sessions" every time and doesn't save cookies and anything so this is why I get this message every time, but it wasn't like that before.

My guess is this wasn't a problem before but is a problem now because YouTube has to follow new GDPR rules. This would make sense because YouTube now asks for your permission before tracking a bunch of things about you. I tried to find a specific article that mentions when this went into effect but couldn't find anything, but my guess is this went into effect pretty recently since I too started noticing this recently.

I have modified yt-videos-list so that selenium uses mine google chrome profile I'm logged on to solved my problem.

So does this mean everything works now? 🙂 Or is there still some changes you need to make to yt-videos-list before it works? For the specific cookie message you get (the message in your screenshot), I could add a check to see if such a notification pops up and accept it within the program - this should be pretty simple for this case since the url changes to consent.youtube.com, so anytime yt-videos-list runs, the program can check the current url using something like:

# function to check if there's a cookie pop-up
def accept_cookies_if_prompted():
    if 'consent.youtube.com' in driver.current_url:
        # click on accept
# proceed with program

This should work in cases like the one you run into! I also noticed other times when I get prompted to accept cookies, but the url doesn't change to consent.youtube.com - an in-browser pop-up shows up instead. This check won't work in such cases, and might need something like what I posted earlier:

# check to see if there's an in-browser pop-up
### if the pop-up is a new window ###
driver.switch_to.window(driver.window_handles[1])
driver.close() # closes currently active window
driver.switch_to.window(driver.window_handles[0]) # switch back to main window

### if the pop-up is more like a text box overlaid on top of the page ###
driver.switch_to.alert.accept()

Google has already blocked my IP address several times. Now I am careful.

😅😅

I also found this when trying to look for solutions, so perhaps this might also be useful: https://stackoverflow.com/questions/66934826/accept-cookies-consent-from-youtube

Also another workaround might be to VPN to a location in the Americas, since it seems that YouTube is lagging on enforcing GDPR-like regulations there, and you get no pop-ups about cookies at all. 🤔🙂

[sirodus]

The only change I made to yt-videos-list was to add my chrome profile to selenium. It would be nice to add this option, using a chrome profile with selenium solves any problems with cookies. It also seems like a good idea to add auto-approval for cookie messages.

[shailshouryya]

How did you add the chrome profile for selenium? I checked the Capabilities & ChromeOptions page for chromedriver, but I don't see any options that seem to be directly related to cookies.

I also tried the following to see if I could find hints in the docstring from the selenium package itself, but didn't find anything useful:

from selenium import webdriver

driver = webdriver.Chrome()
options = webdriver.ChromeOptions()

help(options)
# not much here

options.capabilities
# {'browserName': 'chrome', 'version': '', 'platform': 'ANY'}

options.experimental_options
# {}

options.extensions
# []

driver.capabilities
# dictionary with attributes for driver instance, but didn't see anything that looked like cookie configs

When you have the chance, can you include the option(s) you set for the chromedriver or include an example of the code you modified to enable your chrome profile to bypass the cookie message?

And I'l add the auto-accept cookie feature in the next release (after the current version I'm working on, so this will be in 0.5.6)!

[sirodus]

My solution is very primitive, I wanted to quickly start up to my needs.
It involved removing many functionalities and simplifying the code so that it would work for my needs.
Don't laugh at me, I'm not a programmer :) Maybe it will help you somehow.
I reworked the file "execute.py" as follows.

`import sys
import time
import traceback
import contextlib
import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from . import program
from .download.user_os_info import determine_user_os
from .notifications import Common, ModuleMessage, ScriptMessage
from .custom_logger import log

def logic(channel, channel_type, file_name, log_silently, txt, csv, markdown, reverse_chronological, headless, scroll_pause_time, execution_type):
common_message = Common()
module_message = ModuleMessage()
script_message = ScriptMessage()

def check_user_input():
    nonlocal channel, channel_type
    base_url = 'https://www.youtube.com'
    videos = 'videos'
    url = f'{base_url}/{channel_type}/{channel}/{videos}'
    if txt is False and csv is False and markdown is False:
        print(common_message.not_writing_to_any_files)
        print(module_message.not_writing_to_any_files_hint) if execution_type == 'module' else print(
            script_message.not_writing_to_any_files_hint)
        sys.exit()
    return url

def determine_file_name():
    if file_name is not None:
        return file_name.strip('.csv').strip('.txt').strip('.md')
    else:
        channel_name = driver.find_element_by_xpath(
            '//yt-formatted-string[@class="style-scope ytd-channel-name"]').text.replace(' ', '')
        suffix = 'reverse_chronological_videos_list' if reverse_chronological else 'chronological_videos_list'
        return f'{channel_name}_{suffix}'

@contextlib.contextmanager
def yield_logger(file_name):
    log_file = f'{file_name}.log'
    with open(log_file, 'a', encoding='utf-8') as output_location:
        if log_silently is True:
            yield (output_location,)
        else:
            yield (output_location, sys.stdout)

user_os = determine_user_os()
url = check_user_input()
program_start = time.perf_counter()
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data")
PATH = r"C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(executable_path=PATH, options=options)
with driver:
    driver.get(url)
    driver.set_window_size(780, 800)
    driver.set_window_position(0, 0)
    wait = selenium.webdriver.support.ui.WebDriverWait(driver, 9)
    try:
        wait.until(
            EC.element_to_be_clickable((By.XPATH, '//yt-formatted-string[@class="style-scope ytd-channel-name"]')))
    except selenium.common.exceptions.TimeoutException as error_message:
        common_message.display_selenium_unable_to_load_elements_error(error_message)
        traceback.print_exc()
        sys.exit()
    file_name = determine_file_name()
    with yield_logger(file_name) as logging_locations:
        log('>' * 50 + 'STARTING PROGRAM' + '<' * 50, logging_locations)
        log(f'Now scraping {url}', logging_locations)
        program.determine_action(url, driver, scroll_pause_time, reverse_chronological, file_name, txt, csv,
                                 markdown, logging_locations)
        program_end = time.perf_counter()
        total_time = program_end - program_start
        log(f'This program took {total_time} seconds to complete.', logging_locations)
        log('>' * 50 + 'PROGRAM COMPLETE' + '<' * 50, logging_locations)
return file_name`

[shailshouryya]

Got it, this makes sense!

• options.add_argument("user-data-dir=SOME_PATH_TO_PROFILE") is a flag that allows you to tell selenium where to look for your profile
• PATH = r"C:\Program Files (x86)\chromedriver.exe" assigns the path to your selenium binary to launch the selenium window
• driver = webdriver.Chrome(executable_path=PATH, options=options) creates an instance of the selenium object with what you specified in your options flag(s) and PATH

I wasn't sure how to add an option or flag for enabling/disabling cookies or using your profile, but. based on this code sample it looks like we can do that with the user-data-dir argument for options.

Thanks for providing this code, I appreciate it! I tried looking into this further and found some articles that I found interesting. I'm not sure if it'll be useful for this program at all, but just in case it is, I'm adding them here for future reference:

• https://chromium.googlesource.com/chromium/src/+/master/docs/user_data_dir.md
• https://piprogramming.org/articles/How-to-make-Selenium-undetectable-and-stealth--7-Ways-to-hide-your-Bot-Automation-from-Detection-0000000017.html

[sirodus]

I tried to add it in the original code (and I didn't know exactly where.) but it didn't work.
options = webdriver.ChromeOptions() options.add_argument("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data") driver = webdriver.Chrome(executable_path=PATH, options=options)

I haven't had time to analyze exactly how your program works. I'm a python beginner.

[shailshouryya]

This is pretty good for a beginner! 🙂🤓

I'm not exactly sure why the code snippet you tried didn't work in the existing code, but after looking at the code sample you provided above, I tried to think of how I could add your modification to the code and I couldn't really think of a straightforward way to include the change you made with how the program is structured right now. Or in other words: the current structure of execute.py makes it really hard to make modifications like what you were trying to do.

In order to make this work with the yt-videos-list program the way execute.py is currently structured, I think we would need to add the code snippets for options and PATH into the check_driver() function AND provide them as arguments to webdriver.Chrome, BUT execute.py would fail at the open_driver() function since the open_driver() function tries to run seleniumdriver(). The reason the program fail here is because the program tries to instantiate seleniumdriver inside open_driver() - make note of the parentheses () after seleniumdriver here! I'll point this out below too.

check_driver() right now:

    def check_driver():
        nonlocal user_driver
        if user_driver is None:
            print(module_message.running_default_driver) if execution_type == 'module' else print(script_message.running_default_driver)
            print(module_message.show_driver_options)    if execution_type == 'module' else print(script_message.show_driver_options)
            user_driver = 'firefox'
        if   'firefox' in user_driver: return webdriver.Firefox
        elif 'opera'   in user_driver: return webdriver.Opera
        elif 'chrome'  in user_driver: return webdriver.Chrome    # NOTICE THERE ARE NO PARENTHESES AFTER webdriver.Chrome HERE! THIS MEANS WE DON'T INSTANTIATE THE SELENIUM OBJECT RIGHT NOW!
        elif 'brave'   in user_driver: return configure_bravedriver
        elif 'edge'    in user_driver: return configure_edgedriver
        elif 'safari'  in user_driver: return configure_safaridriver
        else:
            print(common_message.invalid_driver)
            sys.exit()

The problem with the current structure of execute.py is that check_driver() only returns a webdriver.Firefox or webdriver.Opera or webdriver.Chrome or configure_SOMEdriver function (and if we look at the configure_SOMEdriver functions, the functions themselves return the webdriver.SOMEdriver object from inside), and the REALLY SUBTLE thing we need to notice is that the check_driver() function returns an UNinstantiated selenium object (more on this below). Also, the check_driver() function doesn't allow enabling additional options for any drivers here, which is what we're trying to do (adding the custom user profile).

So now, let's try to change things to use your profile with the code you provided in the code blocks below. What we'll probably have had to do if we want to keep everything in the program (the only difference is under the elif 'chrome' in user_driver: code):

    def check_driver():
        nonlocal user_driver
        if user_driver is None:
            print(module_message.running_default_driver) if execution_type == 'module' else print(script_message.running_default_driver)
            print(module_message.show_driver_options)    if execution_type == 'module' else print(script_message.show_driver_options)
            user_driver = 'firefox'
        if   'firefox' in user_driver: return webdriver.Firefox
        elif 'opera'   in user_driver: return webdriver.Opera
        elif 'chrome'  in user_driver: 
            options = webdriver.ChromeOptions()
            options.add_argument("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data")
            PATH = r"C:\Program Files (x86)\chromedriver.exe"
            return webdriver.Chrome(executable_path=PATH, options=options)    # NOTICE THERE **are** PARENTHESES AFTER Chrome HERE! THIS MEANS WE INSTANTIATE THE SELENIUM OBJECT  **right here**!!
        elif 'brave'   in user_driver: return configure_bravedriver
        elif 'edge'    in user_driver: return configure_edgedriver
        elif 'safari'  in user_driver: return configure_safaridriver
        else:
            print(common_message.invalid_driver)
            sys.exit()

Notice how we instantiate the selenium object inside check_driver() function now since we had to provide the PATH and options arguments to the webdriver.Chrome line here. THIS WILL CAUSE PROBLEMS NOW, SINCE AFTER check_driver() finishes, it assigns the returned value to the seleniumdriver variable in this part of the code:

    verify_writing_to_at_least_one_file()
    user_os             = determine_user_os()
    url                 = process_url()
    seleniumdriver      = check_driver()

And THEN when it tries to run the open_driver function here:

driver = open_user_driver()

the program fails since the program tries to instantiate the selenium object inside the open_driver() function. This is because execute.py is currently set up to expect a function or UNinstantiated selenium object from check_driver() function, which execute.py then instantiates within the open_driver() function. WHY IS IT FAILING HERE? Well, this is because we already instantiated the selenium object inside the check_driver() function in this line:

return webdriver.Chrome(executable_path=PATH, options=options)    # NOTICE THERE **are** PARENTHESES AFTER Chrome HERE! THIS MEANS WE INSTANTIATE THE SELENIUM OBJECT  **right here**!!

so when open_driver() runs in the driver = open_user_driver() line, the program fails because we can't instantiate an object that we already instantiated. For easy reference, this is the open_driver() function right now:

    def open_user_driver():
        if headless is False:
            return seleniumdriver()      # NOTICE HOW THE PROGRAM TRIES TO "INSTANTIATE" THE OBJECT WITH THE PARENTHESES HERE - THIS FAILS SINCE WE ALREADY INSTANTIATED THE SELENIUM OBJECT USING webdriver.Chrome(executable_path=PATH, options=options) above
        else: # headless is True
            # AND IF YOU RUN yt_videos_list WITH THE headless=True OPTION, THE set_up_headless_SOMEdriver() FUNCTION BELOW INSTANTIATES THE SELENIUM OBJECT INSIDE THAT FUNCTION!
            if   user_driver == 'firefox': return set_up_headless_firefoxdriver()
            elif user_driver == 'opera':   return set_up_headless_operadriver()
            elif user_driver == 'safari':  return set_up_headless_safaridriver()
            elif user_driver == 'chrome':  return set_up_headless_chromedriver()
            elif user_driver == 'brave':   return set_up_headless_bravedriver()
            elif user_driver == 'edge':    return set_up_headless_edgedriver()

As you can see, this is a really messy problem with the way I've currently structured execute.py, so I'll refactor the code in this file to make it easier to make additions like the one you made in the future since I think the modification you made will be useful for the program. And the modification you made also highlights that the program isn't structured properly right now to allow such changes easily! 😅

I'll link the change I make that should make it easier to make modifications like adding flags to options here after I figure out how to make this simpler, and I'll add another link here after I include changes that should enable yt-videos-list to aoutomatically use your local browser profile like C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data or its variations for other operating systems (see here).

Also, you mentioned that you were a beginner in python, so if you're not familiar with object-oriented programming (OOP), the "instantiate" and "object" terminology I used might be confusing. To better understand these terms, I'd suggest looking into differences between classes and instances/objects. This was a confusing topic for me, so I'd recommend you watch other people/read articles other people wrote to explain it since they'll probably do a better job than I will. Some videos that might be useful:

• Corey Schafer's OOP series
• Python 3's init(), self, Class and Instance Objects Explained Concisely
• OOP Instantiation Tutorial

P.S. My summary is: "instantiate" is a concise way to say "create an instance of a class," and the way you instantiate something is by adding the parentheses () after the name of the class.

• E.g.
  • webdriver.Chrome() instantiates the selenium webdriver object
  • webdriver.Chrome refers to the selenium webdriver class but does not create an instance of webdriver.Chrome since we didn't add the parentheses after!

Thanks again for looking into this issue, this is a very sneaky bug, so I appreciate you taking the time to come up with a fix for this yourself! 🤓

[sirodus]

Thank you very much for such a comprehensive explanation. I am familiar with OOP programming in python. I use Python irregularly, sometimes I have long breaks in programming, so I need to refresh some OOP issues.
I spent some time trying to add this functionality to your program until I finally found it too complex for me. I simplified it (primitive, I know xD) through trial and error.
As you confirmed (and I came to similar conclusions) the current structure of execute.py is difficult to modify. I think if I had worked on it a day or 2 I would have been able to do it but unfortunately I don't have time and needed a quick fix.
I am very glad that you are going to add options to use the google chrome profile. Until then, I will stay with my modification :)

[shailshouryya]

Simplified the logic to add options in commit d90c29f (see the configure_chromedriver() function in logic.py directly for a more clear idea).

Adding options like what you were trying to do should be significantly easier now, as we just need to add the option (such as options.add_argument("user-data-dir=C:\\Users\\user\\AppData\\Local\\Google\\Chrome\\User Data") to the configure_{specific}driver() function now, and the program should be able to launch selenium using whatever options are specified.

Now that this change is committed to the repo, you should have a much easier time adding new options or modifying existing options! Note that I will add this functionality to be able to use local settings as an optional argument in ListCreator eventually, but getting this set up for the different drivers for both macOS and Windows will take some time, so I will add the functionality in a future release.