Add HTTPS Everywhere #128

diracdeltas · 2016-01-09T01:06:14Z

The rulesets.sqlite file is taken from the tip of https://github.com/efforg/https-everywhere/tree/stable. Eventually we need a script to automatically fetch this and run npm run-script preload-httpse per release or so.

The rewrite logic is basically:

On app start, open a connection to the rulesets sqlite DB and load the pre-computed mapping of ruleset ID's to hostnames (httpse-targets.json).
In webrequest.onBeforeRequest, check if the URL has been blacklisted due to causing an infinite redirect loop. If not, look up applicable rulesets from the DB by ID and apply the first-matching rewrite rule. (This can be made more efficient by caching recently-used rulesets in memory.)
Redirect the request to the rewritten URL.

diracdeltas · 2016-01-11T05:19:09Z

Update: thanks to the Electron maintainers who pointed out my very-obvious typo, this is now actually doing redirects. :)

bbondy · 2016-01-11T14:25:52Z

app/filtering.js

@@ -9,7 +9,7 @@ module.exports.register = (wnd, filteringFn) => {
  filteringFns.push(filteringFn)
  if (!wnds.has(wnd)) {
    wnds.add(wnd)
-    wnd.webContents.session.webRequest.onBeforeRequest((details, cb) => {
+    wnd.webContents.session.webRequest.onBeforeSendHeaders((details, cb) => {


Is it any slower to block onBeforeSendHeaders instead of onBeforeRequest?

I'd expect them to be roughly the same

bbondy · 2016-01-11T14:27:12Z

For js/data/rulesets.sqlite I think this is GPL so should probably be loaded remotely. See adblock.js and trackingProtection.js which use the dataFile module. I can throw it on S3 if you want, just ping me on slack.

Please push back if you think it's ok to include it as is.

bbondy · 2016-01-11T14:28:49Z

app/httpsEverywhere.js

+  if (dbErr) {
+    console.log('error loading httpse rulesets.sqlite')
+  } else {
+    console.log('loaded httpse ruleset db')


We can probably comment this one out, and other non error cases.

Good catch, this was just here to prove that it is doing something for subresource loads

bbondy · 2016-01-11T14:33:15Z

Looking great, I'd love to get this in before the beta. The main thing before merging is just to check if the DB is also GPL'ed or get permission for including that statically.

bbondy · 2016-01-11T14:49:00Z

Please also add a switch in app/appConfig.js for enabled in case we need to disable this last minute before the 20th.

diracdeltas · 2016-01-11T20:58:17Z

For js/data/rulesets.sqlite I think this is GPL so should probably be loaded remotely. See adblock.js and trackingProtection.js which use the dataFile module. I can throw it on S3 if you want, just ping me on slack.

Yep, it's GPL so this sounds right.

diracdeltas · 2016-01-11T20:58:58Z

Please also add a switch in app/appConfig.js for enabled in case we need to disable this last minute before the 20th.

will do. will these switches be exposed to users eventually or even for the beta?

bbondy · 2016-01-11T21:00:20Z

The config is just for us currently but anything in there should be moved to app settings once we have them.

bbondy · 2016-01-11T21:01:05Z

I'd love to have settings for beta but I don't think we'll have enough time for 0.7 maybe shortly after though.

diracdeltas · 2016-01-12T22:37:22Z

I've updated this so HTTPS Everywhere also uses dataFile.init

bbondy · 2016-01-13T16:31:49Z

Great, good to merge?

diracdeltas · 2016-01-13T19:30:58Z

Will resolve and merge for now!

Note: this is not quite working yet due to a possible Electron webRequest bug

Registering multiple listeners on onBeforeRequest caused the request to be completed before HTTPS Everywhere got the chance to rewrite it. This may be a bug in how Electron handles conflict resolution between multiple request listeners.

It seems that usually the rulesets have loaded by the time the first network request happens. So instead of aborting if the rulesets haven't loaded, load the listeners anyway.

Add HTTPS Everywhere

diracdeltas changed the title ~~Add non-working HTTPS Everywhere module~~ Add HTTPS Everywhere Jan 11, 2016

bbondy reviewed Jan 11, 2016
View reviewed changes

diracdeltas force-pushed the feature/httpse branch from 726dde8 to 37c74e8 Compare January 12, 2016 00:34

diracdeltas added 11 commits January 13, 2016 12:36

Add current stable rulesets and ruleset preloading script

e153a28

Load https everywhere when a new window opens

7c30eea

Pre-convert rulesets from XML into JSON

1222dc8

Hook up HTTPS Everywhere URL rewriting

039fd60

Note: this is not quite working yet due to a possible Electron webRequest bug

Move HTTPS Everywhere to app folder

9cc2cb6

s/redirectUrl/redirectURL

ff35e97

Update rulesets.sqlite to v5.1.2

f485446

Generalize datafile init so it can be applied to HTTPS Everywhere

380b6f0

Use datafile init for https everywhere

405c6e4

load https-e listeners even if db hasn't finished loading

df71e4e

It seems that usually the rulesets have loaded by the time the first network request happens. So instead of aborting if the rulesets haven't loaded, load the listeners anyway.

diracdeltas force-pushed the feature/httpse branch from 366dae0 to df71e4e Compare January 13, 2016 20:54

diracdeltas added 2 commits January 13, 2016 12:54

Delete unused https-everywhere files

e481798

Update preload-httpse instructions now that rules are in AWS

158a10c

Merge branch 'master' into feature/httpse

2392023

bbondy added a commit that referenced this pull request Jan 13, 2016

Merge pull request #128 from brave/feature/httpse

349395c

Add HTTPS Everywhere

bbondy merged commit 349395c into master Jan 13, 2016

diracdeltas deleted the feature/httpse branch January 14, 2016 21:01

This was referenced Sep 15, 2021

[Snyk] Fix for 1 vulnerabilities velkynosss/browser-laptop#10

Open

[Snyk] Fix for 1 vulnerabilities terrorizer1980/browser-laptop#1

Open

[Snyk] Fix for 1 vulnerabilities San1212/browser-laptop#9

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HTTPS Everywhere #128

Add HTTPS Everywhere #128

diracdeltas commented Jan 9, 2016

diracdeltas commented Jan 11, 2016

bbondy Jan 11, 2016

diracdeltas Jan 11, 2016

bbondy commented Jan 11, 2016

bbondy Jan 11, 2016

diracdeltas Jan 11, 2016

bbondy commented Jan 11, 2016

bbondy commented Jan 11, 2016

diracdeltas commented Jan 11, 2016

diracdeltas commented Jan 11, 2016

bbondy commented Jan 11, 2016

bbondy commented Jan 11, 2016

diracdeltas commented Jan 12, 2016

bbondy commented Jan 13, 2016

diracdeltas commented Jan 13, 2016

Add HTTPS Everywhere #128

Add HTTPS Everywhere #128

Conversation

diracdeltas commented Jan 9, 2016

diracdeltas commented Jan 11, 2016

bbondy Jan 11, 2016

Choose a reason for hiding this comment

diracdeltas Jan 11, 2016

Choose a reason for hiding this comment

bbondy commented Jan 11, 2016

bbondy Jan 11, 2016

Choose a reason for hiding this comment

diracdeltas Jan 11, 2016

Choose a reason for hiding this comment

bbondy commented Jan 11, 2016

bbondy commented Jan 11, 2016

diracdeltas commented Jan 11, 2016

diracdeltas commented Jan 11, 2016

bbondy commented Jan 11, 2016

bbondy commented Jan 11, 2016

diracdeltas commented Jan 12, 2016

bbondy commented Jan 13, 2016

diracdeltas commented Jan 13, 2016