Skip to content

Latest commit

 

History

History
289 lines (182 loc) · 12 KB

GETTING-STARTED.md

File metadata and controls

289 lines (182 loc) · 12 KB
Raw

Contributing

Installation

This assumes a basic knowledge of git, npm and usage of the command line.

** Please ensure you are running on an environment using the latest LTS version of Node.**

First steps:

Clone this repo:

$ git clone https://github.com/WorldBrain/Memex

Install yarn:

We recommend intalling the latest stable version of Yarn by following the instructions specific to your operating system on the Yarn installation page.

Run yarn to install dependencies

This could take a while....

$ yarn

Now run yarn watch to compile incremental builds

$ yarn watch

You can also enable optional OS build notifications in watch mode

$ yarn watch:notif

Subsequent development builds are sped up via use of caching. If any odd side-effects are encountered between builds, you can clear the cache to ensure your next build is completely fresh.

$ yarn cache:clean

Running The Extension

As of now it should work in most modern browsers except Safari (we mainly use Chrome for testing so if any inconsistencies are found in Firefox, Opera or any other browser please create an issue and submit a fix)

Note: It is highly recommended to create a new browser profile for dev purposes especially if you currently use the extension and would like to develop without interfering with its daily use

Creating a New Browser Profile

Chrome:

  1. In the top right corner of your browser click the user icon (above the menu). Alternatively go into settings
  2. Now click Manage People, in settings: Under People click Manage other people
  3. Now click Add People in the lower left
  4. Choose a name such as Worldbrain Test and any icon.
  5. All Finished! 🎉

Firefox:

  1. In firefox url type in about:profiles
  2. Click Create a new Profile
  3. Hit continue
  4. Now enter a name such as Worldbrain Test and hit continue
  5. You should see the new Profile and can now click Launch profile in new browser
  6. All Finished! 🎉 For more info see: creating multiple profiles

Running + Debugging

Chrome:

  1. Open a New Tab
  2. Type chrome://extensions into the address bar
  3. At this point, it is recommended to bookmark Extensions for ease of use in development
  4. Check developer mode box
  5. Click Load unpacked extension...
  6. Now navigate to the folder where you cloned the repo and there should be a new folder named extension (this was created by [yarn watch]) go into this folder then click select this folder
  7. Everything should be all loaded! 😃
  8. To view developer tools go to the Extension Page and under inspect views: click background page

Note: This is only for debugging the background scripts. Content-Script works within the reg dev tools of any given tab and the Options, Overview and Popup UI dev tools can be accesed by right click Inspect on the given element.

Firefox:

  1. Enter about:debugging into the address bar
  2. Check the Enable add-on debugging box
  3. Click Load Temporary Add-on
  4. Now navigate to the folder where you cloned the repo and there should be a new folder named extension (this was created by [yarn watch]) go into this folder select the manifest.json file and then click open
  5. Everything should be all loaded! 😃
  6. To view the developer tools simply click Debug under the Worldbrain Extension in about:debugging

Creating your own branches to work on

We recommend creating a forked repo on your GitHub account, which you can create branches to push your commits to, and easily keep up-to-date with our upstream changes.

Clone your forked repo:

$ git clone https://github.com/${GH_USERNAME}/Memex

Add & fetch our repo as a remote named upstream:

$ git remote add -f upstream https://github.com/WorldBrain/Memex

Create and move to a new local branch based on our current develop branch:

$ git checkout -b ${BRANCH_NAME} upstream/develop

...have a play around...

Publish and push your local branch to your forked repo:

$ git push -u origin ${BRANCH_NAME}

From there, you can make Pull Requests from your pushed branches to any feature/* branches on our Memex repo via your forked repository on GitHub.

You can keep your forked repo's branches in-sync with our upstream changes. We recommend doing this before making or requesting any reviews on PRs.

Replay all your commits on top of our current develop branch:

$ git pull --rebase upstream develop

Overwrite your personal forked repo's branch with your synced local branch state:

$ git push --force

Keep in-the-loop and chat with us on the Memex repo's Issues and PRs pages so we can work together on different parts of the code.

Now you are ready to hack! 😃 We recommend reading through the Code Overview to get an idea of how the extension works and also looking @ Submitting Changes before making any pull requests.

Documenting

We try to encourage documententing module exports using JSDoc with TypeScript, and also higher level overviews of different "feature modules" (directories containing multiple modules related to a particular feature) in README markdown modules.

If you have made changes to any exports in existing modules, please update the corresponding docs if needed.

If you are creating a new feature module, please add a brief overview in the way of a README in the corresponding directory.

Styling

We are using prettier. This will automatically format all the styling for the code every time a commit is made, so you can focus on coding and not on code styling.

Code Overview

A brief overview of Web Extensions

A web extension consists of three main parts:

  • background.js always runs, in an 'empty invisible tab', listening for messages and events.

  • content_script.js is loaded into every web page that is visited. It is invisible from that web page's own scripts, and can talk to the background script.

  • User Interfaces, The UI's are set up and declared in the extension/manifest.json file and at the moment consist of four elements the Popup, Overview, Options and Omnibar.

The parts communicate in three ways:

  • Messaging through browser.runtime.sendMessage, usually done implicitly by using a remote procedure call (util/webextensionRPC.js).
  • Bidirectional messaging using browser.runtime.connect. We use this to communicate between Overview UI script and background script, and also the deprecated imports UI (via Options UI script) and background script. See Runtime for more info
  • Through the in-browser PouchDB database, they get to see the same data and can react to changes made by other parts.

Besides these parts: browser-polyfill.js provides support for the WebExtension/BrowserExt API in order to make the same code run in different browsers (and to structure the callback mess).

This API is available in Chrome/Chromium by default (under window.chrome) but is meant to be developed and standardized as it's own thing.

For more info please see Anatomy of a WebExtension

Application Structure

To keep things modular, the source code in src/ is split into "feature modules"/directories which are grouped by functionality. They can almost be thought of as separate libraries and some features may end up being factored out into their own repos later on.

src/blacklist/: blacklist

This allows a user to stop and/or delete the index/logging of specific domains or pages they visit.

src/common-ui/components: ui elements

This contains common user interface elements such as the loading indicator.

src/dev/: development tools

Tools to help during development. They are not used in production builds.

src/imports/: browser history import

This allows users to import their whole browser history, however, due to slow speeds and multiple setbacks it has been deprecated and may only be kept on as a dev tool to import test docs.

src/options: the settings page

This shows the settings page of the extension which includes (for the time being)

  • Blacklist
  • Acknowledgements
  • Privacy
  • Help Me Please

src/overview/: overview

The overview is a user interface that opens in its own tab. It shows a 'facebook newsfeed' like page that lists random pages you have visited. It also provides tools for more advanced search options and a complete search overview with screenshots. It is built with React and Redux.

See The Docs for more details.

src/page-analysis: page analysis

This extracts and stores information about the page in a given tab, such as:

  • The plain text of the page, mainly for the full-text search index.
  • Metadata, such as its author, publication date, etc...
  • A screenshot for visual recognition.

See The Docs for more details.

src/popup: extension popup

The popup is a mini UI that pops up when you click on the worldbrain. It contains

  • search
  • pause
  • settings
  • feedback

See The Docs for more details.

src/search: document search

This provides a Full-Text-Search through all the web pages a user visits.

See The Docs for more details.

src/util/: utilities

Contains small generic things, stuff that is not project-specific. Things that could perhaps be packaged and published as an NPM package some day. See The Docs for more details.

src/background.ts: WebExtension background script

This provides a wrapper for all the process run in the background which sits in an empty tab listening for events.

See A brief overview of web extensions for more details.

src/content_script.js: A Part of Web Extensions

This just wraps the content scripts of activity-logger and page analysis for use as a Web Extension.

src/omnibar.js: Search-Bar Controls

Allows a user to type w + space_bar to search through worldbrain without leaving the search bar.

See Mozilla Omnibox Docs for more details

src/pouchdb.js: Our Persistent Browser Database

This initializes the user database using PouchDB

Dependencies

  • react - A Javascript component-based 'framework' (it's actually a library) used for the User Interface
  • react-redux - A global state handler that syncs with react to create a nice workflow.
  • babel
  • webpack