Add MKDocs documentation site for Browsertrix Crawler 1.0.0

[tw4l]

Fixes #493

This PR updates the documentation for Browsertrix Crawler 1.0.0 and moves it from the project README to an MKDocs site.

TODO:

• We need to decide on a domain for Browsertrix Crawler docs (currently set to https://crawler-docs.webrecorder.net - if changed, will need to be updated in README and CNAME). crawler.docs.browsertrix.com for now
• Incorporate regex changes from [Docs] Better regex examples #436 after testing (DONE)

To try the docs locally (after installing mkdocs-material via pip in a virtual env or similar):

cd docs && mkdocs serve

Many thanks to @Shrinks99 for help setting this up!

[Shrinks99]

RE domain: Eventually as part of a large site refresh I'd like the docs to be all under our domain so that search engines crawl the whole thing and understand it as one large website. This will also simplify the navigation of all tools (and their documentation) developed under the Webrecorder brand.

If not under a subdomain (maybe the best option for now, simplifies things) something like webrecorder.net/docs/browsertrix-crawler ?? We'd also have webrecorder.net/docs/archive-webpage for example.

EDIT: See chat for further discussion. It's lookin like subdomains!

[Shrinks99]

Nice job with the categories! Site looks great. :D

The cli-options page is probably fine for now, though I'd want to break these out to other pages — presumably people come to the docs because that text didn't help them!

Have noted below, but pretty sure insertversion.js should be removed. I've made a whooooole bunch of copy edits as suggestions.

Once these fixes are in, good to go from me!

[tw4l]

Thanks @Shrinks99 ! Suggestions applied and much appreciated :)

The cli-options page is probably fine for now, though I'd want to break these out to other pages — presumably people come to the docs because that text didn't help them!

Yeah, there's definitely opportunity to improve and expand the docs moving forward, though I think it is sometimes useful as a reference (I've consulted this list in the existing README a few times because it's easier to read in a place like this than in a console).

[Shrinks99]

Pushed two more small fixes. Looks great! Happy to see this information moved out of the monolithic readme file!

[tw4l]

It looks like we now have two CNAME files, one in the root directory for the crawler and another in docs/docs. I assume we only want the latter? Not changing at this point because I'm not sure what URL we settled on for the docs.