RunValidator

RunValidator is a super simple command line tool mainly for checking a retrieval runs proper TREC syntax.

A run-file is a tabular representation of retrieval results. It shouldn't contain the table headings must be seperated by Tab and Space. Each line consists of these fields:

Field:	query-ID	iterator	document-id	rank	score	run-ID
Description:	ID or number of the query	Reserved field, should be Q0or 0	Id of the ranked document	Rank of the Document	Score of the document considering the query	ID for this run

Usage

Usage: RunValidator [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  split     Split a run-file by topic name into separate files.
  validate  Check the syntax of a ranking file.

Validation

RunValidator validates a whole retrieval run or k randomly chosen sample lines from a run. At the moment this tool checks:

1. The delimiter (Tab and Space are valid.),
2. if a line consists of 6 fields,
3. for a proper run tag,
4. if a run tag is consistent throughout the run,
5. if the iterator field is Q0 or 0,
6. for a correct ranking order.

Installation:

pip install git+https://github.com/stella-project/syntax_checker_CLI.git