A hexagonal circle packing implementation for canvasing/sampling large geographical areas (specifically optimized for Google/Instagram/Twitter APIs)
Here's the blog post that explains the project and process behind it.
$ git clone git@github.com:shayanjm/hexpacker.git
$ lein repl
$ lein uberjar
$ java -jar hexpacker-0.1.1-standalone.jar [args]
There are 3 main components for dealing with distributed computing with hexpacker: Generator, Worker, and Transformer.
-
hexpacker.gen/main
generates the lat/long pairs for a given coverage area, chunked into batches of 1000 pairs. -
hexpacker.work/main
reads in one such batch and executes (currently google) queries using the lat/long pairs and returns a set of the clean data. Requires an environmental variableWORKER
to be set to an integer value which denotes the ID of the worker processing the data (depending on how many machines you want to distribute to). This is set up inconfig.clj
-
hexpacker.transform/main
reads in the accumulated result sets, transforms them, and writes them as csv.
$ lein uberjar
$ java -cp hexpacker-0.1.1-standalone.jar clojure.main -m hexpacker.gen | WORKER=1 java -cp hexpacker-0.1.1-standalone.jar clojure.main -m hexpacker.work 15 | java -cp hexpacker-0.1.1-standalone.jar clojure.main -m hexpacker.transform
Since this repo is an implementation (as opposed to a library), you will have to load/include the appropriate namespaces manually.
-
hexpacker.stitch
contains all of the code pertaining to circle packing/coordinate generation. Incanter is also included in this namespace in case you want to run some tests via plotting. -
hexpacker.mercator
contains the mercator projection definitions (keep in mind that there are some artifacts in the file for niceness when plotting with incanter) -
hexpacker.haversine
contains the haversine (and reverse-haversine) implementations. -
hexpacker.core
is the actual 'implementation' for data gathering. You can use it as an example to build out your own cool projects.
Realistically, you should only need to load the hexpacker.stitch/pack-geo-circle
function if you just want to generate circle packings.