Add MKDocs documentation site for Browsertrix Crawler 1.0.0

.github/workflows/docs-publish.yaml

@@ -0,0 +1,21 @@
+name: docs-publish
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'docs/**'
+
+permissions:
+  contents: write
+
+jobs:
+  deploy_docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: 3.x
+      - run: pip install mkdocs-material
+      - run: cd docs/ && mkdocs gh-deploy --force

.husky/pre-commit

@@ -1,4 +1,3 @@
 #!/usr/bin/env sh
 . "$(dirname -- "$0")/_/husky.sh"
-
 yarn lint:fix

docs/docs/CNAME

@@ -0,0 +1 @@
+crawler-docs.webrecorder.net

docs/docs/develop/index.md

@@ -0,0 +1,39 @@
+# Development
+
+## Usage with Docker Compose
+
+Many examples in User Guide demonstrate running Browsertrix Crawler with `docker run`.
+
+Docker Compose is recommended for building the image and for simple configurations. A simple Docker Compose configuration file is included in the Git repository.
+
+For example, to build the latest image, simply run:
+
+```sh
+docker-compose build
+```
+
+Docker Compose also simplifies some config options, such as mounting the volume for the crawls.
+
+For example, the following command starts a crawl with 2 workers and generates the CDX.
+
+```sh
+docker-compose run crawler crawl --url https://webrecorder.net/ --generateCDX --collection wr-net --workers 2
+```
+
+In this example, the crawl data is written to `./crawls/collections/wr-net` by default.
+
+While the crawl is running, the status of the crawl prints the progress to the JSON-L log output. This can be disabled by using the `--logging` option and not including `stats`.
+
+## Multi-Platform Build / Support for Apple Silicon (M1/M2)
+
+Browsertrix Crawler uses a browser image which supports amd64 and arm64.
+
+This means Browsertrix Crawler can be built natively on Apple Silicon systems using the default settings. Simply running `docker-compose build` on an Apple Silicon should build a native version that should work for development.
+
+## Modifying Browser Image
+
+It is also possible to build Browsertrix Crawler with a different browser image. Currently, browser images using Brave Browser and Chrome/Chromium (depending on host system chip architecture) are supported via [browsertrix-browser-base](https://github.com/webrecorder/browsertrix-browser-base), however, only Brave Browser is receiving regular version updates.
+
+The browser base image used is specified and can be changed at the top of the Dockerfile in the Browsertrix Crawler repo.
+
+Custom browser images can be used by forking [browsertrix-browser-base](https://github.com/webrecorder/browsertrix-browser-base), locally building or publishing an image, and then modifying the Dockerfile in this repo to build from that image.

docs/docs/index.md

@@ -0,0 +1,41 @@
+---
+hide:
+  - navigation
+  - toc
+---
+
+# Home
+
+Welcome to the Browsertrix Crawler official documentation.
+
+Browsertrix Crawler is a simplified browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses [Puppeteer](https://github.com/puppeteer/puppeteer) to control one or more [Brave Browser](https://brave.com/) browser windows in parallel. Data is captured through the [Chrome Devtools Protocol (CDP)](https://chromedevtools.github.io/devtools-protocol/) in the browser.
+
+
+!!! note
+
+    This documentation applies to Browsertrix Crawler versions 1.0.0 and above. Documentation for earlier versions of the crawler is available in the [Browsertrix Crawler Github repository](https://github.com/webrecorder/browsertrix-crawler)'s README file in older commits.
+
+
+## Features
+
+Thus far, Browsertrix Crawler supports:
+
+- Single-container, browser based crawling with a headless/headful browser running pages in multiple windows.
+- Support for custom browser behaviors, using [Browsertrix Behaviors](https://github.com/webrecorder/browsertrix-behaviors) including autoscroll, video autoplay and site-specific behaviors.
+- YAML-based configuration, passed via file or via stdin.
+- Seed lists and per-seed scoping rules.
+- URL blocking rules to block capture of specific URLs (including by iframe URL and/or by iframe contents).
+- Screencasting: Ability to watch crawling in real-time.
+- Screenshotting: Ability to take thumbnails, full page screenshots, and/or screenshots of the initial page view.
+- Optimized (non-browser) capture of non-HTML resources.
+- Extensible Puppeteer driver script for customizing behavior per crawl or page.
+- Ability to create and reuse browser profiles interactively or via automated user/password login using an embedded browser.
+- Multi-platform support -- prebuilt Docker images available for Intel/AMD and Apple Silicon (M1/M2) CPUs.
+
+## Documentation
+
+Our docs are still under construction. If you find something missing, chances are we haven't gotten around to writing that part yet. If you find typos or something isn't clear or seems incorrect, please open an [issue](https://github.com/webrecorder/browsertrix-crawler/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc) and we'll try to make sure that your questions get answered here in the future!
+
+## Code
+
+Browsertrix Crawler is free and open source software, with all code available in the [main repository on Github](https://github.com/webrecorder/browsertrix-crawler).

docs/docs/js/insertversion.js

@@ -0,0 +1,34 @@
+const KEY = "/.__source";
+let retries = 0;
+
+function loadVersion() {
+  const value = self.sessionStorage.getItem(KEY);
+  if (value) {
+    parseVersion(value);
+  } else if (retries++ < 10) {
+    setTimeout(loadVersion, 500);
+  }
+}
+
+function parseVersion(string) {
+  const version = JSON.parse(string).version;
+  if (!version) {
+    return;
+  }
+
+  const elems = document.querySelectorAll("insert-version");
+  for (const elem of elems) {
+    try {
+      const code = elem.parentElement.nextElementSibling.querySelector("code");
+      code.childNodes.forEach((node) => {
+        if (node.nodeType === Node.TEXT_NODE) {
+          node.nodeValue = node.nodeValue.replaceAll("VERSION", version);
+        }
+      });
+    } catch (e) {}
+  }
+}
+
+if (window.location.pathname.startsWith("/deploy/local")) {
+  window.addEventListener("load", () => loadVersion());
+}