Contribution Review Criteria

When contributing to VisionEval, the following set of questions will be asked of the contribution. These review criteria help ensure contributions to VisionEval satisfy the Goals and Objectives of the system. Questions are categorized by review type. The automated test system also provides some helpful information where identified.

To submit a contribution for review, issue a pull request with a comprehensive comment introducing your contribution and including pointers to the various review question components. The entire submittal should ideally be self contained so the documentation of the methods should be in the R package as well.

Review Question	Automated Test System Can Help	Review Team - Software	Review Team - Documentation	Review Team - Methods
Does it contain all the elements that are required by the VisionEval system specifications?		x	x	x
Does it include sufficient test coverage and test data?	x	x
Does it pass the ‘testModule’ test which validates that it will run correctly in the model system?	x
Does it include regression tests to enable checking that consistent results will be returned when updates are made to the framework and/or R programming environment?	x	x
Does it only interact with the computing environment by returning a properly structured list to the framework (i.e. it does not modify the global environment, does not read or write files, and only calls framework functions that are allowed)?	x	x
Does it includes all source files and data? If a contributed module does not include all source data, either because of file size or confidentiality, it should include a minimal example data file for testing and so it is clear what data structure is needed to run the module. It should also include clear instructions on how to fetch the large public data and/or a clear explanation of why non-included data is confidential and contact information for data owners.	x	x	x
Is it licensed with the VisionEval license that allows the code to be freely distributed and modified and includes attribution so that the ‘provenance’ of the code can be tracked?	x
Is it based on geographic definitions that are consistent with the model system definitions?			x	x
If the module allows the estimation of regional parameters, does it provide default data, does it have clear documentation of what the estimation data needs to be and how it is to be formatted, and does it include proper data specifications to ensure that the user’s input data are correct?		x	x	x
Does the module compute quickly enough and provide documentation justifying this claim?		x	x
Is the module documentation complete? Does it include documentation of model estimation, algorithms, and instructions for using?			x	x
Does the module only call R code and packages that work on all operating systems? If the code includes any non-R code (e.g. FORTRAN, C++) will that code compile on all operating systems?	x	x
Does it do good science and provide documentation justifying this claim? Is it consistent with good practice in strategic modeling? How might it overlap with existing modules? Why is it better, and/or different?			x	x