Block urls at UC

[JimKarvo]

Hello,

I am trying to block some urls from loading in order to have fast rendering of web page (css, js, images)

Eg, I want to load example.com and block all example.com/assets/* requests.

In order to have a proxyless system, are there any solution that I can follow?

[mdmintz]

For blocking all images, use the block_images arg. Eg.

from seleniumbase import SB

with SB(uc=True, block_images=True) as sb:
    sb.open("https://github.com/readme")
    breakpoint()

For blocking specific content, (eg. ads), see: SeleniumBase/examples/test_cdp_ad_blocking.py

Additionally, you could set host_resolver_rules. More info: #2281

[mdmintz]

That info is here: SeleniumBase/examples/test_cdp_ad_blocking.py (Make modifications as needed!)

Eg. Blocking all JS, CSS, and PNG before the page load, and then removing all <svg> elements after the page load:

from seleniumbase import BaseCase
BaseCase.main(__name__, __file__)

class CDPNetworkBlockingTests(BaseCase):
    def test_cdp_network_blocking(self):
        self.execute_cdp_cmd(
            'Network.setBlockedURLs', {"urls": [
                "*.js",
                "*.css",
                "*.png",
            ]})
        self.execute_cdp_cmd('Network.enable', {})
        self.open('https://www.w3schools.com/jquery/default.asp')
        self.remove_elements("svg")
        source = self.get_page_source()
        self.assert_true("doubleclick.net" not in source)
        self.assert_true("google-analytics.com" not in source)
        self.post_message("Blocking was successful!")

And here's the script just for blocking all JS:

from seleniumbase import BaseCase
BaseCase.main(__name__, __file__)

class CDPNetworkBlockingTests(BaseCase):
    def test_cdp_network_blocking(self):
        self.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.js",]})
        self.execute_cdp_cmd('Network.enable', {})
        self.open('https://www.w3schools.com/jquery/default.asp')
        source = self.get_page_source()
        self.assert_true("doubleclick.net" not in source)
        self.assert_true("google-analytics.com" not in source)
        self.post_message("Blocking was successful!")

[JimKarvo]

Thank you for your reply! Code seems that it is working, but has no effect at undetected browser (uc=True)

Maybe there is an issue with UC?

Edit: To be more specific, it looks like the "execute_cdp_cmd" is not working at all (it doent's throw any error, but the settings that i am executed are ignored)

[mdmintz]

You have to use sb.driver.uc_open(url) because the other URL-opening methods of UC Mode (and the default way) will open a URL in a new tab, which means that the changes made by using sb.execute_cdp_cmd() from the current tab will be lost. Try this:

from seleniumbase import SB

with SB(uc=True) as sb:
    sb.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.js",]})
    sb.execute_cdp_cmd('Network.enable', {})
    sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
    source = sb.get_page_source()
    sb.assert_true("doubleclick.net" not in source)
    sb.assert_true("google-analytics.com" not in source)
    sb.post_message("Blocking was successful!")

[JimKarvo]

Thank you @mdmintz

Finally I found the problem.
The first time I use the uc_open we are ok, i get the page without js,
the second time i use again the uc_open, seems like that the settins are lost

the code:

`from seleniumbase import SB

with SB(uc=True) as sb:
sb.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.css",]})
sb.execute_cdp_cmd('Network.enable', {})
sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
sb.save_screenshot("test1.png")
sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
sb.save_screenshot("test2.png")
source = sb.get_page_source()
sb.assert_true("doubleclick.net" not in source)
sb.assert_true("google-analytics.com" not in source)
sb.post_message("Blocking was successful!")
`

the first screenshot is without css, the second have the css rendered.

[mdmintz]

driver.execute_cdp_cmd() is only good for the next page load. If you need it again, use it again:

from seleniumbase import SB

with SB(uc=True) as sb:
    sb.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.js",]})
    sb.execute_cdp_cmd('Network.enable', {})
    sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
    sb.sleep(3)
    sb.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.js",]})
    sb.execute_cdp_cmd('Network.enable', {})
    sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
    sb.sleep(3)
    source = sb.get_page_source()
    sb.assert_true("doubleclick.net" not in source)
    sb.assert_true("google-analytics.com" not in source)
    sb.post_message("Blocking was successful!")

driver.execute_cdp_cmd() doesn't come from SeleniumBase, so I can't control how it works.

[baturinivan]

You have to use sb.driver.uc_open(url) because the other URL-opening methods of UC Mode (and the default way) will open a URL in a new tab, which means that the changes made by using sb.execute_cdp_cmd() from the current tab will be lost. Try this:

from seleniumbase import SB

with SB(uc=True) as sb:
    sb.execute_cdp_cmd('Network.setBlockedURLs', {"urls": ["*.js",]})
    sb.execute_cdp_cmd('Network.enable', {})
    sb.driver.uc_open('https://www.w3schools.com/jquery/default.asp')
    source = sb.get_page_source()
    sb.assert_true("doubleclick.net" not in source)
    sb.assert_true("google-analytics.com" not in source)
    sb.post_message("Blocking was successful!")

Strange..
Can't get this example working. program stops on sb.assert_true("doubleclick.net" not in source). So, "doubleclick.net" is in source.
Also, I have my own example with cdp_cmd filter in undetected_chromedriver and it filters all necessary urls as needed. Changing it to SB with uc=True according to provided examples gives loading of all content without filtering.
I have tried open, uc_open, open with disconnect... same behavior.

[mdmintz]

SeleniumBase/help_docs/uc_mode.md

Line 180 in e693775

driver.default_get(url) # Faster, but Selenium can be detected

Since UC Mode disconnects the driver on uc_open methods, (which resets CDP changes), you may need to use that instead.

[baturinivan]

driver.default_get(url) # Faster, but Selenium can be detected

Perfect. Thank you!