AutoLog

Collect & Analyze application log sequence from the perspective of program analysis

The repository consists of multiple folders. The folder /src includes source code. The folder /data contains the generated demo data from AutoLog. The folder /scripts contain some useful scripts. The folder /third-party includes two open-source program analysis tools we adpated in this project. The folder /build contains some builded artifacts.

Generated Datasets

The demo datasets are shown in the folder /data.

Sample:

{"9E4C996176B68B9A5F521819A453BB91": {"templates": [......, 
"Resolved path is  <*> ",
 "ACCESS CHECK:  this , doCheckOwner= <*> , ancestorAccess= <*> , parentAccess= <*> , access= <*> , subAccess= <*> , ignoreEmptyDir= <*> , resolveLink= <*> ",
"UnresolvedPathException  path:  <*>  preceding:  <*> count:  <*>  link:  <*>  target:  <*>  remainder:  <*> ",
"Could not get full path. Corresponding file might have deleted already.",
"Could not get full path. Corresponding file might have deleted already."
], "label": 1}

Sample raw data:

INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.FSDirectory: Caching file names occuring more than  <*>  times 
INFO org.apache.hadoop.hdfs.server.namenode.CacheManager: Using minimum value <*> for <*>
WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo: The threshold value should not be greater than , threshold:  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo: dfs.namenode.safemode.threshold-pct =  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo: dfs.namenode.safemode.min.datanodes =  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem$SafeModeInfo: dfs.namenode.safemode.extension     =  <*> 
INFO org.apache.hadoop.hdfs.server.namenode.NNConf: ACLs enabled?  <*> 

Execution

You can play with this analysis framework with multiple ways.

• LogStatGen.jar: fetch and restore log statements from analyzed artifacts for preliminary labelling.
• LogEPGen.jar: acquire log-related execution paths.
• javacg.jar: generate call graph from java project.
• log_methods_generator.py: a simple way to generate pruned call graphs.
• log_sequence_generator.py: log path walking.

Instruction for getting project logging statements, you can conduct statement-level labelling for further anomaly detection.

java -jar LogStatGen.jar -j input_jar.jar -o log_methods.txt -m 'log'

Instruction for running the callgraph generators

java -jar javacg-0.1-SNAPSHOT-static.jar lib1.jar ... > cg.txt

Instruction for pruning log-related call graphs

python generate_log_methods.py --cg cg.txt --output log_methods.csv --matcher 'log'

Instruction for getting logEP for log-related methods, , you can conduct path-level labelling for further anomaly detection.

java -jar LogEPGen.jar -j input_jar.jar -l log_methods.txt -o log_file.json

Instruction for the generated log sequences (without labelling):

python log_sequence_generator.py --call-graph-file cg.txt --log-file log_file.json --label-file log_methods.csv --output-path gen_log.txt --length 1000000