Playwright error: no filters were found - wrong user-agent

[codedesperate]

Describe the bug
When switching from selenium to playwright, all checks get the error:
Warning, no filters were found, no change detection ran - Did the page change layout? update your Visual Filter if necessary.

Version
0.45.1

To Reproduce

Steps to reproduce the behavior:
1:
Use the following docker-compose:

version: '3.2'
services:
    changedetection:
      image: ghcr.io/dgtlmoon/changedetection.io
      container_name: changedetection
      hostname: changedetection
      network_mode: host
      environment:
        - PORT=5001
        - PLAYWRIGHT_DRIVER_URL=ws://127.0.0.1:3000/?stealth=1&--disable-web-security=true
#        - WEBDRIVER_URL=http://127.0.0.1:4444/wd/hub
      volumes:
        - changedetection-data:/datastore
      restart: unless-stopped
      depends_on:
#        browser-chrome:
#          condition: service_started
        playwright-chrome:
          condition: service_started

    playwright-chrome:
      hostname: playwright-chrome
      image: browserless/chrome
      network_mode: host
      restart: unless-stopped
      environment:
        - SCREEN_WIDTH=1920
        - SCREEN_HEIGHT=1024
        - SCREEN_DEPTH=16
        - ENABLE_DEBUGGER=false
        - PREBOOT_CHROME=true
        - CONNECTION_TIMEOUT=300000
        - MAX_CONCURRENT_SESSIONS=10
        - CHROME_REFRESH_TIME=600000
#        - DEFAULT_BLOCK_ADS=true
        - DEFAULT_STEALTH=true

#    browser-chrome:
#      hostname: browser-chrome
#      image: selenium/standalone-chrome-debug:3.141.59
#      network_mode: host
#      environment:
#        - VNC_NO_PASSWORD=1
#        - SCREEN_WIDTH=1920
#        - SCREEN_HEIGHT=1080
#        - SCREEN_DEPTH=24
#      volumes:
#          - /dev/shm:/dev/shm
#      restart: unless-stopped

volumes:
  changedetection-data:

Set up the following monitor:
https://changedetection.io/share/d0qlVgD8rgsa

In global "Fetch Method" have wait seconds set to 20.

Expected behavior
To have the monitor run, and successfully get the CSS Selector's value.

Desktop (please complete the following information):

• OS: Ubuntu Server 22.04.3
• Browser: Firefox

[dgtlmoon]

more than likely playwright couldnt get the page for some reason ,check the logs, maybe some JS didnt run or other

[dgtlmoon]

also set "wait seconds" to 10 not 20, see if it helps

[codedesperate]

@dgtlmoon I have now tried setting the wait seconds to 10 instead of 20. I still get the error.
The docker logs show the following:

Filter for 04be0299-fbb2-498c-b4d2-5ee72c262b82 not found, consecutive_filter_failures: 3
Saving JSON..
> Queued watch UUID 04be0299-fbb2-498c-b4d2-5ee72c262b82 last checked at 1694625017 queued at 1694625078.99 priority 1694625078 jitter 0.00s, 61.99s since last checked
> Processing UUID 04be0299-fbb2-498c-b4d2-5ee72c262b82 Priority 1694625078 URL https://www.royalcaribbean.com/booking/landing?groupId=EX10MIA-2854647895&sailDate=2025-01-16&shipCode=EX&packageCode=EX10D077&destinationCode=CARIB&selectedCurrencyCode=DKK&country=DNK&loyaltyNumber=385903959&CAS=true

I've also now tried going to the monitor to see the preview, and then clicking "screenshot". I can the the page is not loading correctly:
https://i.imgur.com/2ZvmseZ.png

Do you have any idea as to why?

[codedesperate]

I've now tried to install changedetection and playwright using the pip install method. I am getting the same error here.
I've found if I in the browser steps section choose "Wait for text in element", then I'll get the error:

Timeout 60000ms exceeded. =========================== logs =========================== <ws connecting> ws://127.0.0.1:3000/ ============================================================ 

Setting the timeout to 300000ms (5min) produces the following error:

TypeError: Cannot read properties of null (reading 'innerText') at eval (eval at predicate (eval at evaluate (:201:30)), <anonymous>:1:46) at eval (<anonymous>) at predicate (eval at evaluate (:201:30), <anonymous>:11:37) at next (eval at evaluate (:201:30), <anonymous>:32:31) at eval (eval at evaluate (:201:30), <anonymous>:42:11) at UtilityScript.evaluate (<anonymous>:203:17) at UtilityScript.<anonymous> (<anonymous>:1:44) 

I have noticed this is not an issue on every website. Only on royalcaribbean.com. It is a heavy JS page. Maybe playwright can't handle it?

I don't know why, I'm hoping someone can tell me :)

[codedesperate]

@dgtlmoon is your paid service able to load this URL using playwright? https://www.royalcaribbean.com/booking/landing?groupId=JW15AMS-3530067439&sailDate=2024-05-15&shipCode=JW&packageCode=JW15U019&destinationCode=EUROP&selectedCurrencyCode=DKK&country=DNK&loyaltyNumber=385903959&CAS=true

In browser steps the page loads fine, but when actually running it, the only text returned is "oops we are having problems" from the website

[codedesperate]

Ok, I figured it out.
I needed to set a user agent, and this solved my issues. I used this method to apply to all my monitors: https://github.com/dgtlmoon/changedetection.io/wiki/Adding-headers-from-an-external-file

[dgtlmoon]

@codedesperate ah amazing, so I bet if you dont override any user-agent, you get a default user-agent of "'Mozilla/5.0'" if you test it with http://www.xhaus.com/headers

can you confirm?

[dgtlmoon]

"'Mozilla/5.0'" is a bad default.. it should let playwright use its own inbuilt one

[codedesperate]

Just tested and you are right. With no override that site shows me user agent "Mozilla/5.0" and nothing else.

Here is before and after:
https://i.imgur.com/B2lPDV4.png

[dgtlmoon]

@codedesperate ok , and now you switch to mac?

[codedesperate]

@dgtlmoon In the diff picture, the first is with default, the second I have the following in headers.txt.
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

[dgtlmoon]

I just realised the content fetcher is setting a pretty ... not perfect user agent

user_agent=request_headers.get('User-Agent', 'Mozilla/5.0'),

[codedesperate]

Ahhh, okay. I only had this problem on royalcaribbean.com and not on other websites I monitor. So maybe royalcaribbean is just very picky about the user agents. When using the default content fetcher agent, royalcaribbean didn't properly load the website, and instead wrote:

  oops... looks like royalcaribbean.com is on vacation

  don't let that stop your vacation planning

  Call us
  (866) 562-7625
  +1 (305) 341-0204

  Our Brand Specialists are waiting

Maybe bot detection that setting a real user agent circumvents?

[dgtlmoon]

hmmm the bot detection will use a mix of strategies..

[dgtlmoon]

I created #1809

thanks so much for your patience! I'll look at this one soon, I bet this could affect many people

[codedesperate]

And thank you, for all your hard work on this project. You helped me save some money on a holiday :)

I want to give you a donation as a token of my appreciation. However, is your sponsor thing setup correct? When I click the "sponsor" button I am redirected to your profile page, with no option for sponsoring you. With other projects, I'm redirected to a page I can enter an amount after clicking "sponsor".

[dgtlmoon]

@codedesperate try https://www.paypal.com/paypalme/changedetectionio :)

[dgtlmoon]

or you can take out a subscription with our paid accounts, that also helps a lot!!!

[codedesperate]

You just got one new subscription! :)

[dgtlmoon]

Just merged that fixed, also found that it should handle a mix of case, ie uSer-Agent should override it properly too, thanks again! will make a new minor release in the next few days

[dgtlmoon]

#1811

[codedesperate]

@dgtlmoon
Hi again, have there been a recent change with how the User-Agent and headers work?
With:
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0 in headers.txt in /datastore, several websites (when I add them as a watch) report user agent as:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

I've tried several different headers for user-agent in headers.txt, but it seems to no longer have an effect?