Searchable Clojure examples programatically compiled from lots of scraping. GetClojure extracts valid s-expressions from a set of logfiles, runs them in a sandbox, and captures the value and output of every s-expression. The result is then made searchable.
Go to GetClojure and start searching.
| namespace | purpose |
|---|---|
| getclojure.config | Where any globally-necessary project configuration and functions live |
| getclojure.elastic | Houses configuration, query, and seeding from files for elasticsearch |
| getclojure.extract | Contains logic related to parsing clojure IRC logfiles and spitting out files for later consumption |
| getclojure.format | Logic for formatting and syntax highlighting s-expressions |
| getclojure.layout | Static HTML generation via hiccup |
| getclojure.routes | Routes for the application |
| getclojure.seed | Runs the full ETL pipeline from logs to seeding elasticsearch |
| getclojure.server | The entrypoint for the server |
| getclojure.sexp | Logic related to running s-expressions: sandboxing via SCI |
| getclojure.util | Everyone has a junk drawer. This is ours. |
In order to run locally in development mode you'll need to do the following:
- OpenJDK 17. No guarantees it works with older JVMs.
pip install pygments: There are a variety of options for keeping your python envs separate. virtualenv, pyenv, etc. Use whatever works for you. Note that you'll need python 3.X or later.docker-compose up -d: Runs elasticsearch.- Download the logs and extract them into the
resources/logsdirectory. - Set the appropriate env vars in your
.envrc. If you don't use direnv, you'll need to exportAPP_ENV=developmentandINDEX_NAME=getclojure. - Run
lein seed 25to get the s-expressions found in the first 25 logfiles, orlein seed :allto run the entire pipeline across all local logfiles. - Start the server:
lein ring server-headless - Visit localhost:8080 and search.
Contributions welcome. This project uses lein test-refresh for rerunning
tests. Run lein test-refresh :all to run all tests, including the integration
tests. If you have questions about contributing, please reach out to me on
Clojurians Slack (@devn) or Libera IRC (devn).
- To borkdude for providing so much great open source software to the Clojure community.
- To Chris Houser for giving me a treasure trove of logs to harvest.
YourKit is kindly supporting open source projects with its full-featured Java Profiler.
YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products:
Copyright © 2022 Devin Walters
Distributed under the Eclipse Public License, the same as Clojure.