Skip to content

Latest commit

 

History

History
337 lines (213 loc) · 15.9 KB

GETTING-STARTED.md

File metadata and controls

337 lines (213 loc) · 15.9 KB
Raw

Contributing

Installation

This assumes a basic knowledge of git, npm and usage of the command line.

Please ensure you are running on an environment using a v10.x release of Node. As of writing, this is currently Node v10.17.0.

Newer versions could result in a failing yarn install as they haven't been assessed just yet.

First steps:

Clone this repo:

$ git clone https://github.com/WorldBrain/Memex

Install yarn:

We recommend intalling the latest stable version of Yarn by following the instructions specific to your operating system on the Yarn installation page.

Run yarn to install dependencies

in the cloned folder Memex run yarn, This could take a while....

$ yarn

Clone Storex submodules that we depend on

git submodule update --init --recursive

Now run yarn watch to compile incremental builds

$ yarn watch

You can also enable optional OS build notifications in watch mode

$ yarn watch:notif

Subsequent development builds are sped up via use of caching. If any odd side-effects are encountered between builds, you can clear the cache to ensure your next build is completely fresh.

$ yarn cache:clean

Running The Extension

As of now it should work in most modern browsers except Safari (we mainly use Chrome for testing so if any inconsistencies are found in Firefox, Opera or any other browser please create an issue and submit a fix)

Note: It is highly recommended to create a new browser profile for dev purposes especially if you currently use the extension and would like to develop without interfering with its daily use

Creating a New Browser Profile

Chrome:

  1. In the top right corner of your browser click the user icon (above the menu). Alternatively go into settings
  2. Now click Manage People, in settings: Under People click Manage other people
  3. Now click Add People in the lower left
  4. Choose a name such as Worldbrain Test and any icon.
  5. All Finished! 🎉

Firefox:

  1. In firefox url type in about:profiles
  2. Click Create a new Profile
  3. Hit continue
  4. Now enter a name such as Worldbrain Test and hit continue
  5. You should see the new Profile and can now click Launch profile in new browser
  6. All Finished! 🎉 For more info see: creating multiple profiles

Running + Debugging

Chrome:

  1. Open a New Tab
  2. Type chrome://extensions into the address bar
  3. At this point, it is recommended to bookmark Extensions for ease of use in development
  4. Check developer mode box
  5. Click Load unpacked extension...
  6. Now navigate to the folder where you cloned the repo and there should be a new folder named extension (this was created by [yarn watch]) go into this folder then click select this folder
  7. Everything should be all loaded! 😃
  8. To view developer tools go to the Extension Page and under inspect views: click background page

Note: This is only for debugging the background scripts. Content-Script works within the reg dev tools of any given tab and the Options, Overview and Popup UI dev tools can be accesed by right click Inspect on the given element.

Firefox:

  1. Enter about:debugging into the address bar
  2. Check the Enable add-on debugging box
  3. Click Load Temporary Add-on
  4. Now navigate to the folder where you cloned the repo and there should be a new folder named extension (this was created by [yarn watch]) go into this folder select the manifest.json file and then click open
  5. Everything should be all loaded! 😃
  6. To view the developer tools simply click Debug under the Worldbrain Extension in about:debugging

Creating your own branches to work on

We recommend creating a forked repo on your GitHub account, which you can create branches to push your commits to, and easily keep up-to-date with our upstream changes.

Clone your forked repo:

$ git clone https://github.com/${GH_USERNAME}/Memex

Add & fetch our repo as a remote named upstream:

$ git remote add -f upstream https://github.com/WorldBrain/Memex

Create and move to a new local branch based on our current develop branch:

$ git checkout -b ${BRANCH_NAME} upstream/develop

...have a play around...

Publish and push your local branch to your forked repo:

$ git push -u origin ${BRANCH_NAME}

From there, you can make Pull Requests from your pushed branches to any feature/* branches on our Memex repo via your forked repository on GitHub.

You can keep your forked repo's branches in-sync with our upstream changes. We recommend doing this before making or requesting any reviews on PRs.

Replay all your commits on top of our current develop branch:

$ git pull --rebase upstream develop

Overwrite your personal forked repo's branch with your synced local branch state:

$ git push --force

Keep in-the-loop and chat with us on the Memex repo's Issues and PRs pages so we can work together on different parts of the code.

Now you are ready to hack! 😃 We recommend reading through the Code Overview to get an idea of how the extension works and also looking @ Submitting Changes before making any pull requests.

Managing dependencies

We currently use yarn and its lockfile to manage all of our main dependencies that are available on either NPM or GitHub. Please use yarn add $PACKAGE_NAME to add any new deps and properly update the lockfile (please don't manually attempt to update the lockfile). Likewise, yarn remove $PACKAGE_NAME handles removal.

Please be aware that managing deps via the npm CLI tool will not update the lockfile.

For deps that aren't available on NPM or GitHub, these will require some more complicated form of management. We will treat these on a case-by-case basis and discuss with the team.

Documenting

We try to encourage documententing module exports using JSDoc with TypeScript, and also higher level overviews of different "feature modules" (directories containing multiple modules related to a particular feature) in README markdown modules.

If you have made changes to any exports in existing modules, please update the corresponding docs if needed.

If you are creating a new feature module, please add a brief overview in the way of a README in the corresponding directory.

Naming conventions

Over the course of the Memex project's life, different naming conventions for different files/modules have crept in. You'll see JS and TS modules often differing in the naming conventions they are named with depending on the type of code they contain.

For all future contributions, we request that you please name any new JS/TS/CSS modules using kebab-case.

We also ask that you postfix -container to any module name where the main export is a React container component. There may be certain cases where this isn't as obvious or appropriate though. Hopefully we can resolve that in the review stage.

Styling

We are using prettier. This will automatically format all the styling for the code every time a commit is made, so you can focus on coding and not on code styling.

Code Overview

A brief overview of Web Extensions

A web extension consists of a number of different scripts:

  • background.js always runs, in an 'empty invisible tab', listening for messages and events.
  • content_script.js is loaded into every web page that is visited. It is invisible from that web page's own scripts, and can talk to the background script.
  • User Interfaces, The UI's are set up and declared in the src/manifest.json file. At the moment these consist of the popup and options scripts.

Certain modules in the source code will end up in each of these scripts after being output from the build process, depending on the import trails throughout different modules. Each script is completely distinct from any other script. The main thing this means is you should not import from a module in the one script from any module in another script, or you end up duplicating the all the modules specific to one script in multiple other scripts in the build output. This is the main reason for our util/webextensionRPC module:

We can setup remote functions (we sometimes call them "endpoints") with the makeRemotelyCallable export, then be able to remotely call them from other scripts with that remoteFunction export. This wraps around the WebExt runtime.sendMessage and related messaging APIs which afford interscript communication. Generally we setup endpoints via makeRemotelyCallable from the BG script and call them from UI scripts via remoteFunction, although there's nothing stopping you from doing it the other way. It just generally makes a more sense to contain a lot of the business logic in the BG script (it acts somewhat like our "backend server", being the only place the DB is ever interacted with), and keep the UI scripts concerned solely with managing UI state and rendering views.

Besides these parts: browser-polyfill.js provides support for the WebExtension/BrowserExt API in order to make the same code run in different browsers (and to structure the callback mess).

This API is available in Chrome/Chromium by default (under window.chrome) but is meant to be developed and standardized as it's own thing.

For more info please see Anatomy of a WebExtension

Application Structure

To keep things modular, the source code in src/ is split into "feature modules"/directories which are grouped by functionality. They can almost be thought of as separate libraries and some features may end up being factored out into their own repos later on.

Feature Structure

For new BG script features, these should go into the background/ dirs of top-level (src/) feature modules.

An existing example top-level feature module is backup/. You'll see background/ and content_script/ dirs in there containing BG script and content script logic related to that feature, respectively. The backup/background/index module acts as the BG script entrypoint for the backup feature, exporting a main class, BackupBackgroundModule, which accepts a few different args upon instantiation. Most important being storageManager; this allows access to the DB. The BackupBackgroundModule class has a lot of different methods to cover different feature behaviours, but all DB manipulation itself is done in another class in the backup/background/storage module.

Looking into that, you'll see it extends the abstract FeatureStorage class, which affords consistent set up to interact with the storageManager DB instance and set up new Storex (our DB) data collections. Finally, all BG script features should be instantiated in the BG script entrypoint module.

Note that many features do not yet use this structure. We intend to port over legacy code eventually, but all new BG feature code should be written in this way.

Feature list

Note this may not be up-to-date and links may get broken as things change. Feel free to submit a PR to fix anything you notice is broken, missing, or just doesn't make sense.

src/blacklist/: blacklist

This allows a user to stop and/or delete the index/logging of specific domains or pages they visit.

src/common-ui/components: ui elements

This contains common user interface elements such as the loading indicator.

src/dev/: development tools

Tools to help during development. They are not used in production builds.

src/imports/: browser history import

This allows users to import their whole browser history, however, due to slow speeds and multiple setbacks it has been deprecated and may only be kept on as a dev tool to import test docs.

src/options: the settings page

This shows the settings page of the extension which includes (for the time being)

  • Blacklist
  • Acknowledgements
  • Privacy
  • Help Me Please

src/overview/: overview

The overview is a user interface that opens in its own tab. It shows a 'facebook newsfeed' like page that lists random pages you have visited. It also provides tools for more advanced search options and a complete search overview with screenshots. It is built with React and Redux.

See The Docs for more details.

src/page-analysis: page analysis

This extracts and stores information about the page in a given tab, such as:

  • The plain text of the page, mainly for the full-text search index.
  • Metadata, such as its author, publication date, etc...
  • A screenshot for visual recognition.

See The Docs for more details.

src/popup: extension popup

The popup is a mini UI that pops up when you click on the worldbrain. It contains

  • search
  • pause
  • settings
  • feedback

See The Docs for more details.

src/search: document search

This provides a Full-Text-Search through all the web pages a user visits.

See The Docs for more details.

src/util/: utilities

Contains small generic things, stuff that is not project-specific. Things that could perhaps be packaged and published as an NPM package some day. See The Docs for more details.

src/background.ts: WebExtension background script

This provides a wrapper for all the process run in the background which sits in an empty tab listening for events.

See A brief overview of web extensions for more details.

src/content_script.js: A Part of Web Extensions

This just wraps the content scripts of activity-logger and page analysis for use as a Web Extension.

src/omnibar.js: Search-Bar Controls

Allows a user to type w + space_bar to search through worldbrain without leaving the search bar.

See Mozilla Omnibox Docs for more details

src/pouchdb.js: Our Persistent Browser Database

This initializes the user database using PouchDB

Dependencies

  • react - A Javascript component-based 'framework' (it's actually a library) used for the User Interface
  • react-redux - A global state handler that syncs with react to create a nice workflow.
  • babel
  • webpack