SOURCES.org

This file is an overview of available sources along with descriptions and some guidance.

See instructions below for additional information.

Extra dependencies

Some sources require additional dependencies, which you can install if needed.

• pip3 install --user promnesia[optional]

  dependencies that bring some bells & whistles: logzero, python-magic

• pip3 install --user promnesia[HPI]

  dependencies for HPI: HPI

• pip3 install --user promnesia[html]

  dependencies for sources.html: beautifulsoup4, lxml

• pip3 install --user promnesia[markdown]

  dependencies for sources.markdown: mistletoe

• pip3 install --user promnesia[org]

  dependencies for sources.org: orgparse

Alternatively, you can just install all of them in bulk: pip3 install --user promnesia[all].

Sources

These are included with the current Promnesia distribution:

• promnesia.sources.auto
  • discovers files recursively
  • guesses the format (orgmode/markdown/json/etc) by the extension/MIME type
  • can index most of plaintext files, including source code!
  • autodetects Obsidian vault and adds `obsidian://` app protocol support promnesia.sources.obsidian
  • autodetects Logseq graph and adds `logseq://` app protocol support promnesia.sources.logseq
• promnesia.sources.browser

  Uses HPI for visits from web browsers.

• promnesia.sources.fbmessenger

  Uses HPI for the messages data.

• promnesia.sources.github

  Uses HPI github module

• promnesia.sources.hackernews

  Uses HPI dogsheep module to import HackerNews items.

• promnesia.sources.hypothesis

  Uses HPI hypothesis module

• promnesia.sources.instapaper

  Uses HPI instapaper module

• promnesia.sources.pocket

  Uses HPI for Pocket highlights & bookmarks

• promnesia.sources.reddit

  Uses HPI reddit module

• promnesia.sources.roamresearch

  Uses HPI for Roam Research data

• promnesia.sources.rss

  Uses HPI for RSS data.

• promnesia.sources.shellcmd

  Greps out URLs from an arbitrary shell command results.

• promnesia.sources.signal

  Collects visits from Signal Desktop’s encrypted SQLIite db(s).

• promnesia.sources.smscalls

  Uses HPI smscalls module

• promnesia.sources.stackexchange

  Uses HPI for Stackexchange data.

• promnesia.sources.takeout

  Uses HPI google.takeout module

• promnesia.sources.telegram

  Uses telegram_backup database for messages data

• promnesia.sources.twitter

  Uses HPI for Twitter data.

• promnesia.sources.vcs

  Clones & indexes Git repositories (via sources.auto)

• promnesia.sources.viber

  Collects visits from Viber desktop app (e.g. `~/.ViberPC/XYZ123/viber.db`)

• promnesia.sources.website

  Clones a website with wget and indexes via sources.auto

• promnesia.sources.zulip

  Uses HPI for Zulip data.

Instructions

browser

The browser source here uses HPI, which uses browserexport to save/parse browser history. Since browsers remove old history after your history database has grown to a certain size, this saves the raw database files to a directory you specify. browserexport supports many browsers, including any Chromium/Firefox based browsers (e.g. Brave, Vivaldi, Waterfox). For more instructions on backing up databases, see here, but as a quickstart, after setting up HPI, run:

hpi module install my.browser.export

Then, to save your browser history, run a command like:

browserexport save -b firefox --to ~/data/browser_history
browserexport save -b chrome --to ~/data/browser_history
browserexport save -b safari --to ~/data/browser_history

This will save your browser history to a directory, which you can then point HPI at, in your HPI config:

class browser:
    class export:
        # path[s]/glob to your backed up browser history sqlite files
        export_path: Paths = "~/data/browsing/*.sqlite"

    class active_browser:
        # paths to sqlite database files which you use actively
        # to read from. For example:
        # from browserexport.browsers.all import Firefox
        # export_path = Firefox.locate_database()
        export_path: Paths

Finally, you can add the browser source to promnesia, in your config.py:

from promnesia.sources import browser

SOURCES = [
    browser,
    ...
]

Extending

Adding your own data sources is very easy! One day I’ll add some proper documentation, but the easiest at the moment is to use existing simple modules as a reference, for example:

• pocket, 25 lines of code
• fbmessenger, 30 lines of code

After that, you can simply import your custom source in config.py, and use it, same way as any builtin source.