Set up HTTPS testing on Browserstack

[Finesse]

At the moment the tests run in insecure environment (http://localhost). Browsers behave differently in secure (HTTPS) environment and the agent is designed to work in secure environment.

A solution that will probably work:

1. Set protocol: 'https' in the Karma configuration
2. Make a couple of dummy SSL certificates, put them to the repository and add them to the httpsServerOptions setting of the configuration
3. Set browserStack: { acceptSslCerts: true } in the Karma configuration

[makma]

I wanted to help with this issue and tried to solve it, nevertheless, I was not able to run tests properly without some deeper refactoring :/.
However, I reckon my findings might help to save some time for someone or find alternative ways to run browser tests on HTTPS (but probably not using Karma).

One can run tests easily with
npm run test:bowserstack BROWSER_STACK_USERNAME=<yourname> BROWSER_STACK_ACCESS_KEY=<yourkey>

Self-signed cert (and key) can be generated with
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

Moreover, with the settings below, I was able to run tests on BrowserStack, however, I haven't found a way to bypass the browser's check for certificate validity - ERR_CERT_AUTHORITY_INVALID.

Updated karma.browserstack.config

  config.set({
    reporters: [...config.reporters, 'BrowserStack'],
    browsers: Object.keys(customLaunchers),
    customLaunchers,
    concurrency: 5,
    protocol: 'https',
    httpsServerOptions: {
      key: fs.readFileSync('server.key', 'utf8'),
      cert: fs.readFileSync('server.cert', 'utf8')
    },
    browserStack: {
      project: 'FingerprintJS',
      // A build number is required to group testing sessions in the BrowserStack UI.
      // GitHub Actions will add a value for GITHUB_RUN_ID. More on the environment variables:
      // https://docs.github.com/en/free-pro-team@latest/actions/reference/environment-variables#default-environment-variables
      build: process.env.GITHUB_RUN_ID || makeBuildNumber(),
      // The timeout is reduced for testing sessions to not hold the BrowserStack queue long in case of problems.
      timeout: 120,
      acceptSslCerts: "true",
      acceptInsecureCerts: "true"  // wannabe attempt working for Appium https://www.browserstack.com/docs/app-automate/appium/advanced-features/accept-insecure-certs
    },
  })

Note changes in config:
protocol: 'https'

httpsServerOptions: {
  key: fs.readFileSync('server.key', 'utf8'),
  cert: fs.readFileSync('server.cert', 'utf8')
},

acceptSslCerts: "true", and
acceptInsecureCerts: "true"

Browser args bypass

Also, I tried to bypass validations explicitly for browser with args.

For Chrome:

// eslint-disable-next-line no-unused-vars
const chromeDisableSslValidationCapabilities = {
  'goog:chromeOptions': {
    args: ['--ignore-certificate-errors', '--disable-web-security'], // https://superuser.com/a/1036062 https://stackoverflow.com/a/3177718/1692651
  },
}

For Firefox:

// eslint-disable-next-line no-unused-vars
const firefoxDisableSslValidationCapabilities= {
  'moz:firefoxOptions': {
    prefs: {
      'security.insecure_field_warning.contextual.enabled': false, // https://support.mozilla.org/en-US/questions/1159525
    },
  },
}

All of the above does not work and Browserstack still ignores params and validates self-signed certificates.

Possible solutions or workarounds

• Test could contain logic that will click on the Advanced button and clicks to proceed unsafely. It might be necessary to implement this logic for each browser and it will be probably expensive to maintain.
• Environment might be deployed with a real domain and certificate. Therefore Browserstack could perform tests on this deployed site.
• Migrate tests from Karma to Espresso, Selenium or another framework that can bypass cert checks.

[Finesse]

@makma Thank you for your research! My experience shows that karma-browserstack-launcher is a very limited tool. It doesn't allow sending browser arguments, and many BrowserStack options don't work. I faced similar issues when I tried to run the tests in incognito mode.

Using Selenium directly gives much more possibilities. I'd prefer to lean to Selenium to fix both this, the incognito testing and the browser detection issues. Karma is a high level tool that allows running tests in BrowserStack with little configuration, meanwhile Selenium is a low level tool that only controls a browser. Selenium requires additional tools to transpile the code for browser, run a local server, tunnel the server to the public internet, collect test results and print them to the console. Combining these tools together means creating a complete test runner like Karma. Do you know if there is a ready solution? Probably, we will make our own testing solution that combines these tools and is shipped as a separate library.

If we want to fix only this issue, I will prefer the simplest solution possible as a temporary solution that will be replaced with a full-featured solution described above.

[makma]

Hi @Finesse,
From my point of view, it might be unnecessarily expensive to create and maintain a custom test runner just for these HTTPS tests. If I understand it correctly, the main goal is to run tests on the HTTPS environment - it doesn't have to be done on Browserstack at all - am I right?

I believe GitHub Actions might be great for this task. Actions are for free for open source repositories and one can choose the platform - Ubuntu, Windows, or even macOS. In addition, it seems you can also generate and trust self-signed certificates - I assume this since one has sudo/administrator privileges on these machines. The cherry on the top - default virtual environments come with a lot of preinstalled browsers, mobile Safari might be problematic, though.

I've created the first PoC test for Chrome (with --ignore-certificate-errors arg), without trusting the certificate, yet. Please, take a look at this pull request with a passing test and let me know your thoughts.
Cheers!

[Finesse]

Hi @makma, HTTPS testing isn't the main goal of testing. The main goal is to run the tests in a large variety of browsers, that's why we use BrowserStack. The second important goal is running a test that checks that incognito and regular modes of all supported browsers give the same result. Running the tests in HTTPS is the same important.

Running browsers in GitHub Actions in addition to BrowserStack will achieve the goals partially because it will cover only some desktop browsers. But it's better than nothing, so we can implement it. We used to run browsers in GitHub Actions in v2.

[entoncescomo1001]

Hi @makma, HTTPS testing isn't the main goal of testing. The main goal is to run the tests in a large variety of browsers, that's why we use BrowserStack. The second important goal is running a test that checks that incognito and regular modes of all supported browsers give the same result. Running the tests in HTTPS is the same important.

Running browsers in GitHub Actions in addition to BrowserStack will achieve the goals partially because it will cover only some desktop browsers. But it's better than nothing, so we can implement it. We used to run browsers in GitHub Actions in v2.

Duplicate of #
Duplicate of #

[Finesse]

HTTPS has been set up for almost all browsers by #874. Closed in favor of fingerprintjs/broyster#36. The test tools are now a scope of another repository.