Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiled list of anti-bot security measures by google #593

Open
2 of 4 tasks
JijaProGamer opened this issue Sep 17, 2023 · 17 comments
Open
2 of 4 tasks

Compiled list of anti-bot security measures by google #593

JijaProGamer opened this issue Sep 17, 2023 · 17 comments

Comments

@JijaProGamer
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

I'm submitting a ...

Description

I'm providing here a list of anti-bot POST network requests made by youtube.

I am using firefox since my bot is also based on firefox, but these should help you out no matter what browser you use (even though you should use firefox)

Environment

- OS : Windows 10
- Python: Not applicable
- Script version: not applicable
- Browser: Firefox v117

config.json

Not applicable
@JijaProGamer
Copy link
Author

JijaProGamer commented Sep 17, 2023

First network request when opening any youtube website. Used to track basic info about your fingerprint:

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "2078",
            "referer": "https://www.youtube.com/",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "x-youtube-bootstrap-logged-in": "false",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "same-origin",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; __Secure-YEC=REDACTED; VISITOR_PRIVACY_METADATA=REDACTED; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest"
        },
        "url": "https://www.youtube.com/youtubei/v1/guide?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8&prettyPrint=false"
    },

postData:

{
  "context": {
    "client": {
      "hl": "en",
      "gl": "RO",
      "remoteHost": "<redacted IP>",
      "deviceMake": "",
      "deviceModel": "",
      "visitorData": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0,gzip(gfe)",
      "clientName": "WEB",
      "clientVersion": "2.20230914.04.00",
      "osName": "Windows",
      "osVersion": "10.0",
      "originalUrl": "https://www.youtube.com/",
      "platform": "DESKTOP",
      "clientFormFactor": "UNKNOWN_FORM_FACTOR",
      "configInfo": {
        "appInstallData": "COHTm6gGENShrwUQ26-vBRDZya8FEJfn_hIQ6sOvBRDuoq8FEN3rrwUQp-r-EhDd6P4SEKbs_hIQ1-mvBRDr6P4SENPhrwUQ1eWvBRDks_4SELzrrwUQ5OavBRDi1K4FENzjrwUQzK7-EhDE3a8FELzM_hIQ1eqvBRCG6q8FEMzfrgUQuIuuBRC9tq4FEOe6rwUQiOOvBRCst68FEKTerwUQjMuvBRD6vq8FELTJrwUQx-avBRD65P4SEInorgUQpcL-EhC1pq8FEPOorwUQtOavBQ%3D%3D"
      }
    },
    "browserName": "Firefox",
    "browserVersion": "102.0",
    "acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "deviceExperimentId": "ChxOekkzT1RjMk16QXdNVEEzTVRrek9EWTBOdz09EOHTm6gGGOHTm6gG",
    "screenWidthPoints": 1280,
    "screenHeightPoints": 720,
    "screenPixelDensity": 1,
    "screenDensityFloat": 1,
    "utcOffsetMinutes": 180,
    "userInterfaceTheme": "USER_INTERFACE_THEME_LIGHT",
    "mainAppWebInfo": {
      "graftUrl": "https://www.youtube.com/",
      "webDisplayMode": "WEB_DISPLAY_MODE_BROWSER",
      "isWebNativeShareAvailable": false
    },
    "timeZone": "Europe/Bucharest"
  },
  "user": {
    "lockedSafetyMode": false
  },
  "request": {
    "useSsl": true,
    "internalExperimentFlags": [
      {
        "key": "force_enter_once_in_webview",
        "value": "true"
      }
    ],
    "consistencyTokenJars": []
  },
  "adSignalsInfo": {
    "params": [
      {
        "key": "dt",
        "value": "1694951906097"
      },
      {
        "key": "flash",
        "value": "0"
      },
      {
        "key": "frm",
        "value": "0"
      },
      {
        "key": "u_tz",
        "value": "180"
      },
      {
        "key": "u_his",
        "value": "1"
      },
      {
        "key": "u_h",
        "value": "900"
      },
      {
        "key": "u_w",
        "value": "1440"
      },
      {
        "key": "u_ah",
        "value": "860"
      },
      {
        "key": "u_aw",
        "value": "1440"
      },
      {
        "key": "u_cd",
        "value": "24"
      },
      {
        "key": "bc",
        "value": "31"
      },
      {
        "key": "bih",
        "value": "720"
      },
      {
        "key": "biw",
        "value": "1263"
      },
      {
        "key": "brdim",
        "value": "4,4,4,4,1440,0,1292,811,1280,720"
      },
      {
        "key": "vis",
        "value": "1"
      },
      {
        "key": "wgl",
        "value": "true"
      },
      {
        "key": "ca_type",
        "value": "image"
      }
    ]
  },
  "fetchLiveState": true
}

@JijaProGamer
Copy link
Author

JijaProGamer commented Sep 17, 2023

Request made when visiting a video. Possibly to fingerprint as well. Many details are the same as the first important request.

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "2244",
            "referer": "https://www.youtube.com/",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "x-youtube-bootstrap-logged-in": "false",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=REDACTED; __Secure-YEC=REDACTED; VISITOR_PRIVACY_METADATA=REDACTED; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=REDACTED"
        },
        "url": "https://www.youtube.com/youtubei/v1/att/get?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8&prettyPrint=false"
    }

postData:

{
  "context": {
    "client": {
      "hl": "en",
      "gl": "RO",
      "remoteHost": "<redacted IP>",
      "deviceMake": "",
      "deviceModel": "",
      "visitorData": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
      "userAgent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0,gzip(gfe)",
      "clientName": "WEB",
      "clientVersion": "2.20230914.04.00",
      "osName": "Windows",
      "osVersion": "10.0",
      "originalUrl": "https://www.youtube.com/",
      "platform": "DESKTOP",
      "clientFormFactor": "UNKNOWN_FORM_FACTOR",
      "configInfo": {
        "appInstallData": "COHTm6gGENShrwUQ26-vBRDZya8FEJfn_hIQ6sOvBRDuoq8FEN3rrwUQp-r-EhDd6P4SEKbs_hIQ1-mvBRDr6P4SENPhrwUQ1eWvBRDks_4SELzrrwUQ5OavBRDi1K4FENzjrwUQzK7-EhDE3a8FELzM_hIQ1eqvBRCG6q8FEMzfrgUQuIuuBRC9tq4FEOe6rwUQiOOvBRCst68FEKTerwUQjMuvBRD6vq8FELTJrwUQx-avBRD65P4SEInorgUQpcL-EhC1pq8FEPOorwUQtOavBQ%3D%3D"
      },
      "playerType": "UNIPLAYER",
      "tvAppInfo": {
        "livingRoomAppMode": "LIVING_ROOM_APP_MODE_UNSPECIFIED"
      }
    },
    "browserName": "Firefox",
    "browserVersion": "102.0",
    "acceptHeader": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "deviceExperimentId": "ChxOekkzT1RjMk16QXdNVEEzTVRrek9EWTBOdz09EOHTm6gGGOHTm6gG",
    "screenWidthPoints": 1280,
    "screenHeightPoints": 720,
    "screenPixelDensity": 1,
    "screenDensityFloat": 1,
    "utcOffsetMinutes": 180,
    "userInterfaceTheme": "USER_INTERFACE_THEME_LIGHT",
    "mainAppWebInfo": {
      "graftUrl": "https://www.youtube.com/",
      "webDisplayMode": "WEB_DISPLAY_MODE_BROWSER",
      "isWebNativeShareAvailable": false
    },
    "timeZone": "Europe/Bucharest"
  },
  "user": {
    "lockedSafetyMode": false
  },
  "request": {
    "useSsl": true,
    "internalExperimentFlags": [
      {
        "key": "force_enter_once_in_webview",
        "value": "true"
      }
    ],
    "consistencyTokenJars": []
  },
  "adSignalsInfo": {
    "params": [
      {
        "key": "dt",
        "value": "1694951920095"
      },
      {
        "key": "flash",
        "value": "0"
      },
      {
        "key": "frm",
        "value": "0"
      },
      {
        "key": "u_tz",
        "value": "180"
      },
      {
        "key": "u_his",
        "value": "1"
      },
      {
        "key": "u_h",
        "value": "900"
      },
      {
        "key": "u_w",
        "value": "1440"
      },
      {
        "key": "u_ah",
        "value": "860"
      },
      {
        "key": "u_aw",
        "value": "1440"
      },
      {
        "key": "u_cd",
        "value": "24"
      },
      {
        "key": "bc",
        "value": "31"
      },
      {
        "key": "bih",
        "value": "720"
      },
      {
        "key": "biw",
        "value": "1263"
      },
      {
        "key": "brdim",
        "value": "4,4,4,4,1440,0,1292,811,1280,720"
      },
      {
        "key": "vis",
        "value": "1"
      },
      {
        "key": "wgl",
        "value": "true"
      },
      {
        "key": "ca_type",
        "value": "image"
      }
    ]
  },
  "engagementType": "ENGAGEMENT_TYPE_PLAYBACK",
  "ids": [
    {
      "playbackId": {
        "cpn": "EljjkHNU6y-SeKJt"
      }
    }
  ]
}

@JijaProGamer
Copy link
Author

Request made when accessing any *.youtube.com page, made for initializing the GRPC connection

network request data:

    {
        "method": "POST",

        "headers": {
            "host": "jnn-pa.googleapis.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json+protobuf",
            "content-length": "24",
            "referer": "https://www.youtube.com/",
            "x-goog-api-key": "AIzaSyDyT5W0Jh49F30Pqqtyfdf7pDLFKLJoAnw",
            "x-user-agent": "grpc-web-javascript/0.1",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "cross-site",
            "connection": "keep-alive"
        },
        "url": "https://jnn-pa.googleapis.com/$rpc/google.internal.waa.v1.Waa/Create"
    }

postData:

[\"<REDACTED>\"]

@JijaProGamer
Copy link
Author

Weird POST with encrypted data, internally named "LOG_EVENT", my best guess is that it's used to log stuff like hovering over suggestions, pausing, clicking, etc.

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "4505",
            "referer": "https://www.youtube.com/",
            "x-goog-request-time": "1694951924455",
            "x-goog-eom-visitor-id": "CgttS0tnak81WWZkWSjh05uoBjIGCgJSTxIA",
            "content-encoding": "gzip",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951906097&flash=0&frm&u_tz=180&u_his=1&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c"
        },
        "url": "https://www.youtube.com/youtubei/v1/log_event?alt=json&key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
    }

postData:

\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0003�\\\u000bo�Ȗ�+\u0011ڻڽJ'.���\u001di�\u0018�\u0004\u001b����\u00112؀���m����{�LH&=\r=Yw��t[i�\u001cW��wN�2��V��Q��R���$\f��^��R��G��Ҍ\\�-���\u001a��/���Ǯ��A\u001c�\u0018��eX�Ȉ�a��!3�h\u001a̴h\u001a\u0013��j�Ei�a��\\�����0k��5O�N�\u0015>m�fm�w��z?�Fs�#�-B[�O@3pBƭ>%��y�\u0013ڜ�>�S2\u000e}Z�q\t�\u0019m�\u000f�=B[�#�5\u000ft.�r\t-@\u000f\u001c�;<\u0012��A$�TL�6\u000f:��?�8�!\u0010��a�̜�Z[��&gO0��\u000bdn�\"�)i&��\u0007�'�G}Md\b\u001b2�i�\u0013��\u0013�+�D?-�\t�դ\tr\u0015�\"�ڭ��ˈ~���\u001a�P\u000f��B���g톐t��f�O��G��$\u0004D�pG�8��>3\u0004\"_�D�C; 4�����ֲ��g�\u001fg\u0002����r?R_H\u0002�Qk�\t-\\S~�DB��-�1m��\u0006�\u0003�{\u0000��/�mw1��5�3�e+�q�4\u000e��êY��k��Ѹ.��24��Dݻ��r�s�\u0001�q=6���=�Wmv�\u0018���it\u0013׉9w��\u001cd̛�������Mǵ���_9^��zjX\u001d�M�o�\\��9�J�0u�aP���hҨo:}#�\"9�H3�j�\u000b�Ӗs�:=��P�g�W�ΡZ��+�X���Rb�}#\u0019,��I�ʺ\f����\u001d/P����c���t�6Zs����N��\u000e�a�7\u001e;�8���\u000ej\u001c��EXi���#?�K��Q5t���X���a\u000f,�衃&\r/��۽ۭ\u000bݻ������^����Io^��u\u0005R�jw\rcx�:C\\����a��)�Ϭ������\u001fB�s�a��3���k1���\u000f\u0005�qX�\u0017�\u001b��=��}��W��\u0012p�8���\u000e��5;!|\u001eg���ΰ;���mu{��\u0003F�\u0017\u001cА�{�3�:�\u0006�÷�փ��G�\u0016�\u0006�S�f���~�Т��'��ɔ�\u001e }!���\b�Gm�\u0001ڃH�=\u000euLh�����GhO-:w�\u0013�vX\u0011ڪ�\u0010\u001a\u0016\b�\u0019%��>��u\u0011R�����}\u0011�o\u0015\u0001\bG�rد\"]]m�ƀ5�\u001a��ad6����*\u001e��w\u0017\u000e\u000f��C\u001cn�}�i׼Ⱥ�1\u001d{�\u001d2uی4NW�$d�_�֩�Tft�(��!\bC���a���\u0017D^�M�\f�\n17��\\%�2b�\u001b濯\u001a�d\u0011߲\fb�\u0007]Ճğƻ[z\u0017`�t���Z����r��ǵ ]��^�=XrJ=�:�iV�Y\u0019��VM\u001dU�V�RM��8\u0001�~�/N�#�҉��J=ˋ��أʔ�z\u0018\u001d\u0005ҳi�,�RS�\u0007��&�\u0012�i��t�\f`�L\"7Tw+?\t���z���R����\u0016�\u001e���?�ɸQ\u001c�e\u0010���M�oA��\r�D�,�_���;p�708�@/m`��g$Ȝ�#��X`��_5I�(�\u000b��\\�&�o��;�,��,[�����v{����z��L��-�_l�n��V7g{�9z�E×�\u001d\\�ftD\\�2\u0014\u0002�嘩�[�f��Y��(�\b\u0017<#�\u0012G.8\u0016\u000b�5\u000f@�\u0004I�\u0014��xBa\u0004̳9E�\u0018J�\u000b���$��\u0012$^ cxV`�\u0002˲�P�\u0000W�|\u001c��t�\b,&�Y���� \u0011}X��9zK����\u0016�\u0019B�x,�\t\f���\u000f�yBA\u0012G5�\u0002+��\u0007K<H'\f9,c���xQ���`3��E�a�\u0002q\f���<�H�:\u0003CLg���\f��3Ce�X�\u0006J���-,�\u0000\u0013�`0\u0007n�\u0006�����3\"fs?c^�BY�\u001d�\u0000�r,\"&K���\u0010\b<�\u0006Btd�\u0010+\u0018����2��YH`e�\u0006��\u0006\u000eL\u0015\u0010�\u0007��\u0019ư�(I�\u0005�0\u000f�$PY\b!1w8f�<�\u0002�\t\u001f\u0001��6\u001ae���C\u0012\f��\u0018\u0016�\u0012�&\u000b\f� a8\u0010G(\u0002x��`e$Q���(w&D��5�����<�\u001cx��\u001b�\"�����1��hP\u0000\u0000��\u0003\u0012X�\u00100�H9�(�G\u001c�t0#������P�\n2��\u0011�\u001f��\"�g��/��c��\u0005�:,s\u0002G�L\u0010���\u0014�\\\r�\u0000\u0012܂y�r\ra6���RG�?�(�$\t�X\u0005U��\u0000�)��_��\f\u0011:� �2Xx�8\u0016�F�b��\u0014c\u00020Dy��P\u0007G\u0011��,\u0010G�\t��y,\u0000ct0\u0002\u001cѼ\u0010\u0011`�0\u0014��s�Ҵ�P\n\u001cu\u0014\u000f�@�`,��\u001c�\u0001\u0000�D���\u001cQH\u00121� \u0014\rȫ<O9�:p��\u0007�\u0011�j���)6\u0010�\u0011D\u001a\u001d�\u0011��X���\u0002���\n�?�P�X�З#�U��F�9+7q��\r�閟l��o\u0005�����ki�t����/�\"��\"�g��d�'��N�0��:k�N�`���&�֒0��/��?��_����U\u0012{���JwZ45��A/��TϞ��\u00154N0<��)\t�+�\u000bBO�9��T����T�O��+\u0014�֨c��U\u0017�\tw��)�N��\u000f�\r��S�;��iA{�|�u�T\tNk�\tׯ��KI E�Wh `�=Y؉;�i�����u:q3�+�I�q�,2��z=��,��k]CzKZ2?�u�\u001e\u001a\u000b>��<\n��f3\u0005Xe�~4r�۱��7�i\u0019��eN�\u001c�3��\u001f\u0005�0WF,�\u001c�y\u0005rŌPf\u0011p��,n<h@�l2WI#V�KxY(c\t��L�B��2�>GY�\\YPV�]�IEr\u0016!�,�s�\u0007�r�eQ�\u001c��ݹ�\u0017�� ��u\u0018��HK��H��R\u0006\u0011���h;+���,�\u0011�c\u0017�������2t�9���<a[(S\u0011��s��o`���l�\u0002V�\u000fU�� J�T�\u0011�\u001e��s�\u0014� Q\u0014P\u0019V�ϑV$R\u0010Ǘ9�ߣ��f� ���]�5\u000bqBY�ڲp�-㰅)�m�LٲD�J�P�P��/�Q44���\u000bb��%\t�\u0005O\u001b\u0005��2��<�7��E�l\u0014�`\u00114�\u0010�V�(�\u0018\u001a����_\u0012d�bA\u0006``�J��%)�7�$د\u0000o�H�9Y\u0014���\u001cU�\u0005�q�\u0019�\f��Yg�z~���խ7��\u0013�an\u001f�q\u0010ݦ���C�O�y�\u0010\bk\\F��n�|\u0001h��&�k\u0017�2_� ����5��\u00157\fǰ����\u0000��/A�}F���Q��[\"�#\u0015\u001f`�,\u0001Է�;����*Q�t� �ꉻ��b�+B������%�D��0���e\u0010=�qv\fIq\u0001�G:q���k�\u001cʿ�S���Ŵz~/p���9J�դwq�r\u0019\t������;B}Amؼ2\u00000�A��\u0002��,�\u0016n\t\b\u0013A���Uw�z�}<�뒎��*\u000e�K��~�C��\r�Q0\u0001\u0000<+pq\u001c]\u0002%�1&��\u001b՚�\u0004q\u0012d��AK\u0016\\�9\u001a��\u0011�B5�e��`=\u001b��wM?F\u0017L\u001fP��x�0�'�v�\u0006�\u0018�>ZC@\u0017�a͛�\u001f�-��e\u0001�|��7�\u001b7<%�2�h��Q\u0002OB��\u0011\u0013�\r~�T��\u0005V�􃫏ċlY\u0000M&�%���\u0011��E�\u001e�^�}������\u00027�^��>�\\h����,\u000f�LV\u0016�_��\u0014�tHe�D[)��\u0000\u001bCX�V���1�)W�߀��\u0001;\f(�\u001fʇ���\fO�\u0015,�WN\u000b� �h�$^�K&����\r\u0019�1�8�o|`�#T\"��*�����\u0007��#�Tl{��؏P�!��>8,�`�Or�'�$�yS\u0018�č�f��+\u0010[�f��}$��Ҥ\u0010�\fz���`%\t\u000be�Aj��K� �����W\u000f��c�a<��4ug�\u0014L�o|r\u0016V5͖9j�]�9�UL�Ќ\u00069}�<�d�&WS\u0017dxW�ڿ���<��'�n�_W����W�u��|u�O�W�A?)��>�,���^�Rw�\n��\u001f��Y���r�޹�\tYe�\u0005� ۓc�%�'��9��^�6A�vC5���)k\u001eo#��NJ�.�\n��\u001f5N�;\u0007�6�\u0019�\u000eg\u001c4^?tn�),�\u000fȸ�\"�fmz����yp\u0014��n���%�=�&�ݬ�w��N��x�l�~��t�\rIK,�`�x�\u0010N@�ɐj���K���w��^*�\u0012��\u0018K\u0016?�\u0012^�>ir�\u0017m\t>o����b��\u0014m�Ĝ3�ce�Ɛ���\u0013���ۖ\n�a���%[f��ܚ�fتYQl�e���F\u0003V�QC�l�TG����\u0007C55e�45�t\t�gm'g�Ki\u0018��hUr��2�g3�S_����aXrZ6%�_^���qZs?\f��\u000b���w�Q�u�\t6nh�A��G��6J\u000f�uFBƼ\u001a��ٚ�j9fW�V�#ˮ؎�\u001b[ij]�PL��J��\u00185��\u0019jm�XF���$N�\\��\"l5��Q��Ѡ��PzG�5\u001dK�,��6�ӈ�K�Xj�<�HVG�Q;�s��7@%�Q\u0014`Aވ���n����\u001c��W4cTi��V�J�\u0018��͡\u0002\f%����m����;i�5�4\u0017}�\u0016�����9�\u0003{�\u0019�8\bR�bWFmS�T��&!����-wv\u0017����kA8\u0013�/�zG�NA8���\u0005|v\u001d��TAOsd�U�Q�PG�f8��M\u001e\\�U��֡yу_J�U�(��\u0010ѻ��A��\\\u0016�B�*p�2\u001f@��64�6+_ͭ����:�\r7?.�����Tx?f_��g�,X�p����ӽ!*.�p�\u0002\u0010�~g���\u0012�j\u0013� ���Z�M\u0016Ngf����t�\"�ߚX�hMX���B��������~���9\n��~�/0���:���4O�^��]�Р;\rA�h�'\u0013��\u000e&��7��긧w��\u001f��G\u0019�^�A��\u0012��>-lI�W\u0006��~�g/��]�w��Y\u0012�#�·�n컔�4H�rC\u001c:)V�'k�^?�����튝�~����\u0003\u0018\u0018D���]�5U���懙���\u0018��ʒ\u0000���7�[P/�\u0016�I�;\u0015ҳ\n�\u001e٦�h\u0000ȡѫV����e��\u0005Å7ۄ��������\u001c�+,�ܛm�pQó;�\\ß��WTÂ�Պ_�\u000bl�ߗ�g\u001f�Ahd\"7���A�&\t�O�ĩ�\u0004n\u0018\u001c|�t�\u0018���(m\u0006�n5��L\u001f\u001b�dٝ������Ȼ�\u0006CK^\r\u001bݵ�\b\u0019ߒc���}v�\u001f��\b�<�]�q�ʉ��N'K�\u0019��՘��jd�k4�k�2K\u001f\u0014'l9Ȭ��nV5E��Am\u001b�+3��u��vUu�i��/�\\N��Ϥ8�g���%O�aOۀ7\u0002�7�\f�����p\u0012��\u0000�e�S��.n\rXy�a�g��A�>\u001d�:�R��FE�*�Y����\u0005+\u0011y\"\u001c�a�݇4���J|�X\u0000苫���-l�˪���X��Ec8\b�J����\u000fڃ�U\u0016�ʬ@)�h=ey�#2�\u000f\u0005��9�I\u0000\u0000

@JijaProGamer
Copy link
Author

JijaProGamer commented Sep 17, 2023

Request sent every few seconds while watching a video, to show youtube your progress on the video:

network request data:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/x-www-form-urlencoded",
            "content-length": "226",
            "referer": "https://www.youtube.com/",
            "x-goog-visitor-id": "CgtGQjF3Vm1vcEhfYyj105uoBjIGCgJSTxIA",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951960794&flash=0&frm&u_tz=180&u_his=1&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c; GPS=1"
        },
        "url": "https://www.youtube.com/api/stats/qoe?fmt=243&cpn=h4fYjUTUAKyV87Qy&el=detailpage&ns=yt&fexp=v1%2C23983296%2C2730%2C18618%2C2602%2C73492%2C54572%2C233521%2C2567%2C381%2C12995%2C54587%2C84737%2C25387%2C973%2C8870%2C1087%2C3781%2C1193%2C976%2C322%2C3200%2C125880%2C4132%2C26306256%2C26%2C171%2C136%2C840%2C146%2C2225%2C536%2C1253%2C677%2C612%2C243%2C5869%2C5694&cl=564805759&seq=1&docid=XaK1ht8poRc&ei=Q-sGZevOKNjigAeziomIDA&event=streamingstats&feature=g-high-rec&plid=AAYFjNNZaVEN45oR&referrer=https%3A%2F%2Fwww.youtube.com%2F&sdetail=p%3A%2F&sourceid=y&qclc=ChBoNGZZalVUVUFLeVY4N1F5EAE&embargoed=0&cbr=Firefox&cbrver=102.0&c=WEB&cver=2.20230914.04.00&cplayer=UNIPLAYER&cos=Windows&cosver=10.0&cplatform=DESKTOP&vps=0.000:N,0.006:N&cat=streaming&vfs=0.006:243:243::r&view=0.006:371:209&bwe=0.006:130000&vis=0.006:0&cmt=0.006:0.000&bh=0.006:0.000"
    }

postData:

session_token=REDACTED

@JijaProGamer
Copy link
Author

Event POST message. This affirms my theory that youtube sends your key strokes and mouse movements to their server to detect bot activity:

    {
        "method": "POST",
        "headers": {
            "host": "www.youtube.com",
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.5",
            "accept-encoding": "gzip, deflate, br",
            "content-type": "application/json",
            "content-length": "3643",
            "referer": "https://www.youtube.com/watch?v=XaK1ht8poRc",
            "x-goog-request-time": "1694952268990",
            "x-goog-visitor-id": "CgtGQjF3Vm1vcEhfYyj105uoBjIGCgJSTxIA",
            "x-youtube-client-name": "1",
            "x-youtube-client-version": "2.20230914.04.00",
            "x-youtube-device": "cbr=Firefox&cbrver=102.0&ceng=Gecko&cengver=102.0&cos=Windows&cosver=10.0&cplatform=DESKTOP",
            "x-youtube-page-cl": "565485865",
            "x-youtube-page-label": "youtube.desktop.web_20230914_04_RC00",
            "x-youtube-utc-offset": "180",
            "x-youtube-time-zone": "Europe/Bucharest",
            "x-youtube-ad-signals": "dt=1694951925740&flash=0&frm&u_tz=180&u_his=2&u_h=900&u_w=1440&u_ah=860&u_aw=1440&u_cd=24&bc=31&bih=720&biw=1263&brdim=4%2C4%2C4%2C4%2C1440%2C0%2C1292%2C811%2C1280%2C720&vis=1&wgl=true&ca_type=image",
            "origin": "https://www.youtube.com",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "connection": "keep-alive",
            "cookie": "YSC=zW3r7T6c4V4; VISITOR_PRIVACY_METADATA=CgJSTxIA; CONSENT=PENDING+373; PREF=tz=Europe.Bucharest; SOCS=CAISEwgDEgk1NjUxODgxOTQaAmVuIAEaBgiAg5moBg; VISITOR_INFO1_LIVE=FB1wVmopH_c; GPS=1"
        },
        "url": "https://www.youtube.com/youtubei/v1/log_event?alt=json&key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8"
    },

postData:

{
   "context":{
      "client":{
         "hl":"en",
         "gl":"RO",
         "clientName":1,
         "clientVersion":"2.20230914.04.00",
         "configInfo":{
            "appInstallData":"CPXTm6gGELWmrwUQ1uqvBRDM364FEIjjrwUQteavBRCX5_4SEMTdrwUQzK7-EhDuoq8FENXlrwUQjMuvBRDk5q8FEPq-rwUQ1KGvBRDqw68FEOLUrgUQp-r-EhDr6P4SEPrk_hIQ2cmvBRC0ya8FENPhrwUQ57qvBRDbr68FEKPerwUQhuqvBRCJ6K4FEKy3rwUQu-uvBRClwv4SEKXs_hIQ8qivBRDks_4SELzM_hIQ3ej-EhC4i64FEL22rgUQx-avBRDd668FEPTnrwUQ3fWvBQ%3D%3D",
            "coldConfigData":"CPXTm6gGEO-6rQUQvbauBRDi1K4FEPKorwUQyLmvBRCfx68FEObLrwUQg8yvBRCj3q8FEMLerwUQxd-vBRCj4q8FELXmrwUQ5OavBRD0568FEOHprwUQhuqvBRDW6q8FELvrrwUQgu6vBRD9768FEM7wrwUQ0PCvBRDY8K8FEN31rwUaMkFPakZveDNzTk1udld1eDloZUxyUy1wSlcwRV8tbjdzeURUdTJwWkdERjFoVVZKZmRRIjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUSpEQ0FNU0xnMFZncGFvQXNnV19nV21INTBLdlFfOUJ1QWVGUi1TZ3RBTXMwZWd0Z2FPSHJKRzNtS2RMNURaQlBPcHpBcz0%3D",
            "coldHashData":"CPXTm6gGEhMxNzg2MTM0NzkyODA4NzE3NTM0GPXTm6gGMjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUToyQU9qRm94M3NOTW52V3V4OWhlTHJTLXBKVzBFXy1uN3N5RFR1MnBaR0RGMWhVVkpmZFFCRENBTVNMZzBWZ3Bhb0FzZ1dfZ1dtSDUwK3c4cXVPUmdNQlhOTFJiS3k0cTRKcWc4QUFRNkFJVVNvTHVaZjFMc1lpLTV3c0ZFUkx4akozNEdvYmp2N0hrcnlUbWVqdElkYVRuMkdoeFhxR0RSTVhoVlZLaW1sZ0YycklqSkJUQW5oT0kxN3lOSXlTcjdtbElkakg1bWFzdndydVVRdVR2dW80V2hsVEhKdk00SjRxN2R3TWFJSkN3aVZNSjNtc2NzWG1rbk1lNUlqa1dvVzFSZ2xvVGhBQT09",
            "hotHashData":"CPXTm6gGEhQxNTQ0MjIwMTg5MjY0MTk4NTQ1Mxj105uoBiiU5PwSKNuT_RIoxrL9EiiqtP0SKKXQ_RIonpH-Eiiarf4SKIuu_hIoyMr-EiiUzf4SKN3O_hIowd3-Eiio4f4SKJnm_hIojOn-Eij26f4SKKTq_hIopez-EijE7P4SMjJBT2pGb3gzc05NbnZXdXg5aGVMclMtcEpXMEVfLW43c3lEVHUycFpHREYxaFVWSmZkUToyQU9qRm94M3NOTW52V3V4OWhlTHJTLXBKVzBFXy1uN3N5RFR1MnBaR0RGMWhVVkpmZFFCMENBTVNJQTBMb3RmNkZiNF95Z0NvT2NBUkZSZmR6OElNcUpzQ2ktNEI4ZUFPdFlBRQ%3D%3D"
         },
         "userAgent":"Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0",
         "mainAppWebInfo":{
            "webDisplayMode":"WEB_DISPLAY_MODE_BROWSER"
         },
         "browserName":"Firefox",
         "browserVersion":"102.0",
         "osName":"Windows",
         "osVersion":"10.0",
         "platform":"DESKTOP"
      },
      "request":{
         "internalExperimentFlags":[
            {
               "key":"force_enter_once_in_webview",
               "value":"true"
            }
         ]
      }
   },
   "events":[
      {
         "eventTimeMs":1694952267942,
         "latencyActionTicked":{
            "tickName":"att_s",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"124"
         }
      },
      {
         "eventTimeMs":1694952267942,
         "latencyActionTicked":{
            "tickName":"bg_v",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"125"
         }
      },
      {
         "eventTimeMs":1694952267984,
         "latencyActionTicked":{
            "tickName":"bg_s",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"167"
         }
      },
      {
         "eventTimeMs":1694952267984,
         "latencyActionTicked":{
            "tickName":"att_f",
            "clientActionNonce":"2LRg0P--wguss09S"
         },
         "context":{
            "lastActivityMs":"168"
         }
      },
      {
         "eventTimeMs":1694952268374,
         "visualElementShown":{
            "csn":"MC43OTc4MDk3NDUwMjU2MTU0",
            "ve":{
               "trackingParams":"CEQQkKwCIhMIjbeSnM2xgQMV0lrgCh2t0AQ-"
            },
            "eventType":1,
            "clientData":{
               "adsClientData":{
                  "adClientDataEntry":{
                     "slotData":{
                        "type":"SLOT_TYPE_IN_PLAYER",
                        "controlFlowManagerLayer":"CONTROL_FLOW_MANAGER_LAYER_CORE",
                        "entryTriggerType":"TRIGGER_TYPE_LAYOUT_ID_ENTERED",
                        "debugData":{
                           "slotId":"RviyELkxejtZQxJX",
                           "slotEntryTriggerData":{
                              "type":"TRIGGER_TYPE_LAYOUT_ID_ENTERED",
                              "triggerSourceData":{
                                 "associatedLayoutId":"Dfplaq7bsu7NA5qN"
                              }
                           },
                           "fulfillmentTriggerData":[
                              {
                                 "type":"TRIGGER_TYPE_SLOT_ID_ENTERED",
                                 "triggerSourceData":{
                                    "associatedSlotId":"RviyELkxejtZQxJX"
                                 }
                              }
                           ],
                           "expirationTriggerData":[
                              {
                                 "type":"TRIGGER_TYPE_ON_NEW_PLAYBACK_AFTER_CONTENT_VIDEO_ID"
                              },
                              {
                                 "type":"TRIGGER_TYPE_SLOT_ID_EXITED",
                                 "triggerSourceData":{
                                    "associatedSlotId":"RviyELkxejtZQxJX"
                                 }
                              }
                           ]
                        }
                     },
                     "layoutData":{
                        "type":"LAYOUT_TYPE_MEDIA_LAYOUT_PLAYER_OVERLAY",
                        "controlFlowManagerLayer":"CONTROL_FLOW_MANAGER_LAYER_CORE",
                        "debugData":{
                           "layoutId":"ywBBjxutchYq5Xz1"
                        }
                     }
                  }
               }
            }
         },
         "context":{
            "lastActivityMs":"557"
         }
      }
   ],
   "serializedClientEventId":{
      "serializedEventId":"9ekGZdzSCcux6dsP-e6PqAk",
      "clientCounter":"19839"
   }
}

@kattstof
Copy link

kattstof commented Sep 17, 2023

I could be wrong but isn't this the automation detection that was bypassed in undetected_chromedriver v3.4.5?
it could just be that this uses a deprecated and hence detectable version (3.2 vs the current 3.5) my recommendation of switching user agents with chrome properly causes it to keep some views but delete others, but firefox geckodriver seems to keep all of them so far. Maybe its a case of needing to update undetected_chromedriver? i'll try in the next few days when i get time and update with anything i find. cheers!

@JijaProGamer
Copy link
Author

I could be wrong but isn't this the automation detection that was bypassed in undetected_chromedriver v3.4.5? it could just be that this uses a deprecated and hence detectable version (3.2 vs the current 3.5) my recommendation of switching user agents with chrome properly causes it to keep some views but delete others, but firefox geckodriver seems to keep all of them so far. Maybe its a case of needing to update undetected_chromedriver? i'll try in the next few days when i get time and update with anything i find. cheers!

No...?

This isn't even using chrome.

This is standard Firefox. These are just the network requests google sends to fingerprint the user.

@kattstof
Copy link

No, i know it's using firefox But these mitigation techniques are what borked undetected_chromedriver a few months ago and caused the need for a complete overhall i'm pretty sure.

@JijaProGamer
Copy link
Author

No, i know it's using firefox But these mitigation techniques are what borked undetected_chromedriver a few months ago and caused the need for a complete overhall i'm pretty sure.

But undetected_chromedriver has nothing to do with this.

This is just a compiled list of every POST request made by YouTube.

undetected_chromedriver does nothing to change the user agent, the window size, the CPUs and memory shown by the user agent, etc.

Even IF undetected_chromedriver was randomising the fingerprint, it would still have nothing to do with my list of POST requests

@kattstof
Copy link

most of my work is based on selenium (geckodriver) and requests ( shout out to everyone else helping with v3n0m ;) )

the reason i'm bring up undetected_chromedriver is not because of your POST request LOL I'm pointing out that the automation detection that these allude to borked undetected_chromedriver 3.2 firefox doesn't have this problem, undetected_chromedriver does.

@JijaProGamer
Copy link
Author

most of my work is based on selenium (geckodriver) and requests ( shout out to everyone else helping with v3n0m ;) )

the reason i'm bring up undetected_chromedriver is not because of your POST request LOL I'm pointing out that the automation detection that these allude to borked undetected_chromedriver 3.2 firefox doesn't have this problem, undetected_chromedriver does.

So if your project has nothing to do with my post, why did you even comment?

@kattstof
Copy link

Are you dense? I'm stating that your post requests allude to mitigation techniques that have been bypassed in a newer version of undetected_chromedriver so my comment has everything to do with this thread. LOL

@JijaProGamer
Copy link
Author

Are you dense? I'm stating that your post requests allude to mitigation techniques that have been bypassed in a newer version of undetected_chromedriver so my comment has everything to do with this thread. LOL

Please keep it profesionional.

And it seems like your project has nothing about this, which anti fingerprinting depends on: "undetected_chromedriver does nothing to change the user agent, the window size, the CPUs and memory shown by the user agent, etc."

The bot isn't detected because it's using a old chromedriver, it's because it doesn't even try to anti fingerprint.

@kattstof
Copy link

yes because you not understanding me is unprofessional lol
let me dumb this down
the use of the undetected_chromedriver package used in this script is used for bypassing known anti-bot detection mechanisms. catch my drift? so it is part of the solution if we insist on using chrome

@JijaProGamer
Copy link
Author

JijaProGamer commented Sep 17, 2023

yes because you not understanding me is unprofessional lol let me dumb this down the use of the undetected_chromedriver package used in this script is used for bypassing known anti-bot detection mechanisms. catch my drift? so it is part of the solution if we insist on using chrome

That part of the solution is useless. Undetected_chromedriver doesn't live up to it's name, considering it gets detected every 2 weeks, and you still have to fix a lot of stuff by yourself.

Firefox has everything done for you, using less resources, with a neater API. You just have to do the anti fingerprinter.

@kattstof
Copy link

Trust me i know, i've been repeatedly saying to use geckodriver on deaf ears to the point where i'm probably just going to port it to firefox myself and submit a PR. But, The use of chrome seems to be what mshawon would prefer -- so im trying to keep that in mind when discussing it.

"firefox would be better, but if chrome is insisted upon..." is my line of thinking currently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants