QueryAnalysis

Getting Started

Prerequisites

You need to have Maven, OpenJDK 8 and Python 2 installed.

Installing

$ mvn clean package

Running the main Java log analyser

# Processes the example SPARQL log files into exampleMonthsFolder/exampleMonth/processedLogData
$ mvn exec:java@QueryAnalysis -Dexec.args="-w exampleMonthsFolder/exampleMonth -logging"

# There are more (useful) CLI parameter available, you can list them with:
$ mvn exec:java@QueryAnalysis -Dexec.args="--help"

Important: In order to not flush the command line with error messages all uncatched Runtime Exceptions are being written to the log files, residing in the logs/ folder, so please have a look at those regularly. The logs are not generated by default, so you should enable them using the -l option.

Running the QueryAnalysis-Script

The QueryAnalysis-Script handles both steps: Extraction using hive and processing using the java application. Extraction using hive only works on the server, but is ignored if the month exists in the months folder. To run the QueryAnalysis-Script locally, you need to provide the local months folder.

$ python QueryAnalysis exampleMonth -m ../exampleMonthsFolder

# You can also specify multiple months in the same directory by separating them using commas
$ python QueryAnalysis exampleMonth,otherMonth -m ../exampleMonthsFolder

License

The code in this repository is released under the Apache 2.0 license. External libraries used may have their own licensing terms.