Implement Fathom page classification module #78

victwj · 2021-05-06T16:54:57Z

Description

Resolves #27. Integrates Mozilla Fathom to enable ML classification of pages or DOM elements within a page. At the element level, example applications include recognizing which DOM elements represent overlays, ads, login forms, privacy policies, and cookie banners. At the page level, example applications include recognizing shopping pages, login pages, and news article pages. The user must first train their own model. This library then makes it simple to deploy the trained model through a Rally extension.

Technical

The fathom-web package is installed as devDependency. If the user wants to use this module, they must install this package in their own extension. The broad architecture of this module follows pageManager and pageText. A summary of how it works:

The fathom.js background script exports the webScience.fathom.onFathomData event. The user must call webScience.fathom.onFathomData.addListener() from their own background script, specifying their callback and URL match pattern. Fathom will only be run on URLs which match the specified pattern.
The user must also call the global function window.webScience.fathom.addTrainees() from their own content script to specify their trained Fathom model.
When a tab is opened, the fathom background script sends an webScience.fathom.isClassifiable message to the content script running on the tab. This boolean is determined by the user's URL match patterns.
Upon receiving an isClassifiable == true message, the Fathom content script will proceed to classify the page based on the user's rulesets, which were added via the global window.webScience.fathom.addTrainees().
Once the Fathom content script is done, results are sent to the background script as a webScience.fathom.fathomData message. The Fathom background script then processes the results and calls the user's callbacks on the results.

Discussions

This module currently supports the most basic functionality: run Fathom at most once upon page load, then return the classification results. In reality, the element the user is looking for may be dynamically loaded at some point after the page is loaded. Should a rerunning function based on MutationObserver be implemented in the content script? This would be difficult to do on the user's end, so might be worth doing in this module.
Once an element is classified, should this module handle tracking the amount of time the element is in the user's screen? This should be doable on the user's end and is more study specific, but is likely to be a very commonly used feature, making it worth handling in the module. Most other types of element tracking (such as whether or not the element was clicked) are study specific and would fit better as a user implementation. This module can return the classified DOM elements, after which the user may monitor it as they see fit.

Todos

Memoize page contents to optimize running multiple Fathom rulesets.
Make border drawing a user configurable option.
Potentially change the use of onUpdated as the main event to trigger Fathom runs.
Write jsdoc comments.
Remove console.log lines, which are all temporary and only used for development purposes.
Define a proper classification results schema.

jonathanmayer · 2021-06-08T18:01:04Z

package.json

@@ -18,6 +18,7 @@
        "eslint-plugin-import": "^2.22.1",
        "eslint-plugin-mocha": "^8.1.0",
        "eslint-plugin-node": "^11.1.0",
+        "fathom-web": "^3.7.3",


Please merge in the latest changes on main, since the dependencies changed.

jonathanmayer · 2021-06-08T18:01:34Z

src/content-scripts/fathom.content.js

+
+import {ruleset, type} from "fathom-web";
+
+(function() {


You don't need an IIFE wrapper. Rollup will add that automatically.

jonathanmayer · 2021-06-08T18:03:00Z

src/content-scripts/fathom.content.js

+    if ("webScience" in window) {
+        if ("fathom" in window.webScience) {
+            return;
+        }
+        else {
+            window.webScience.fathom = {
+                trainees: new Map(),
+                addTrainees: addTrainees,
+            }
+        }
+    }
+    else {
+        // Else, this is the first webScience initialization
+        window.webScience = {};
+        window.webScience.fathom = {
+            trainees: new Map(),
+            addTrainees: addTrainees,
+        }
+    }


Could condense this a bit.

jonathanmayer · 2021-06-08T18:03:31Z

src/content-scripts/fathom.content.js

+    let pageManagerLoaded = false;
+
+
+    // Function to initalize fathom once conditions are fulfilled 


Say more about what this function does.

jonathanmayer · 2021-06-08T18:03:47Z

src/content-scripts/fathom.content.js

+        }
+    }
+
+    // Whether the elements of this page have been classified.


Nit: would be consistent about punctuation.

jonathanmayer · 2021-06-08T19:20:09Z

src/fathom.js

+/**
+ * A callback function to add a fathomDataListener.
+ * @param {fathomDataCallback} listener - The listener that is being removed.
+ * @param {Object} options - Options for the listener.
+ * @param {string[] options.matchPatterns} matchPatterns - The match patterns 
+ * for pages where the listener should be notified.
+ * @private
+ */


Firefox recently added dynamic imports for content scripts. That would be a good way to handle rulesets and avoid the awkwardness with content script globals. Something like...

A parameter to addListener specifies a path for one or more ruleset JS files and names for each ruleset.

The ruleset JS file must be a module that exports a ruleset() function.

When a page visit starts, check the URL against the match pattern set for each listener. Send an array of ruleset names and file paths to the content script.

In the content script, import the relevant rulesets, run them, and send back the results.

jonathanmayer · 2021-06-08T19:20:46Z

src/fathom.js

+        // When a tab is updated, send it a message if the page should be 
+        // classified with Fathom
+        // TODO: onUpdated may not be the right event
+        browser.tabs.onUpdated.addListener((tabId, changeInfo, tab) => {


Would use the pageManager.onPageVisitStart event so you can include a pageId in the message.

jonathanmayer · 2021-06-08T19:21:13Z

src/fathom.js

+    const contentScript = await browser.contentScripts.register({
+        matches: matchPatterns,
+        js: [{
+            code: inline.dataUrlToString(fathomContentScript)


We changed how content script imports work.

jonathanmayer · 2021-06-08T19:22:51Z

src/fathom.js

+function messageListener(fathomDataObject) {
+    console.log("messageListener called");
+    for (const [listener, listenerRecord] of fathomDataListeners) {
+        listener(fathomDataObject);


Don't want to notify the listener unless there's a match.

jonathanmayer · 2021-06-08T19:23:32Z

src/fathom.js

+        if (listenerRecord.matchPatternSet.matches(fathomDataObject.url)) {
+            listener(fathomDataObject);
+        }


Don't need to do this matching twice. Could associate each listener with a unique ID and include that in the message to the content script and the response.

Victor Ongkowijaya added 14 commits May 3, 2021 15:33

Install fathom-web as devDependency

630a8c3

Add fathom background script

9020b8b

Add fathom content script

45d88bd

Add fathom module export

f629dff

Stop fathom from running multiple times

1219e16

Add framework for fathom in content script

ca8cf3c

Temporary structure, need to fix ruleset() exception

1650f97

Implement one-time fathom classification on page

dbc1a1f

Temporary fix of race condition using setTimeout

4d62cba

Minor cleanup in fathom background script

01242d1

More cleanup of fathom scripts

e7618c8

Remove timeout-based race condition handling

1f5b322

Fix results sending to the background script

aedfdb1

Cleanup for first review

427ff35

victwj marked this pull request as ready for review May 6, 2021 19:16

Fix outdated jsdoc comment

706e8bc

jonathanmayer self-requested a review June 8, 2021 17:59

jonathanmayer assigned victwj Jun 8, 2021

jonathanmayer added the enhancement New feature or request label Jun 8, 2021

jonathanmayer reviewed Jun 8, 2021

View reviewed changes

victwj closed this Nov 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Fathom page classification module #78

Implement Fathom page classification module #78

victwj commented May 6, 2021 •

edited

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

jonathanmayer Jun 8, 2021

		let pageManagerLoaded = false;


		// Function to initalize fathom once conditions are fulfilled

Implement Fathom page classification module #78

Implement Fathom page classification module #78

Conversation

victwj commented May 6, 2021 • edited

Description

Technical

Discussions

Todos

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

victwj commented May 6, 2021 •

edited