Contribution Review Criteria

When contributing to VisionEval, the set of questions below will be asked of the contribution. These review criteria help ensure contributions to VisionEval satisfy the Goals and Objectives of the system. Make sure to review the Developer Orientation before making a submittal. Questions are categorized by review type. The automated test system also provides some helpful information where identified.

Process

To submit a contribution for review, issue a pull request with a comment introducing your contribution. The comment should include a brief overview, responses to the questions, and pointers to related information. The entire submittal should ideally be self contained so the documentation of the methods should be in the R package as well. The Repository Manager and Review Team Chair will handle the review request and will respond to the contributor. Its a good idea to setup a pre-submittal meeting to discuss questions and better understand expectations. Check out the example submittal and review to see what the process looks like.

Review Criteria

Review Question	Test System Help	Review Team - Software	Review Team - Documentation	Review Team - Methods
1. Does it contain all the elements that are required by the VisionEval system specifications?		x	x	x
2. Why is it better, and/or different than existing modules? Does it do good science and provide documentation justifying this claim? Is it consistent with good practice in strategic modeling? How might it overlap with existing modules? How does it fit within the VE ecosystem of modules?			x	x
3. Is the module documentation complete? Does it include documentation of model estimation, algorithms, and instructions for using?			x	x
4. If the module allows the estimation of regional parameters, does it provide default data, does it have clear documentation of what the estimation data needs to be and how it is to be formatted, and does it include proper data specifications to ensure that the user’s input data are correct?		x	x	x
5. Is it based on geographic definitions that are consistent with the model system definitions?			x	x
6. Does the module compute quickly enough and provide documentation justifying this claim?		x	x
7. Does it includes all source files and data? If a contributed module does not include all source data, it should include a minimal example data file for testing and so it is clear what data structure is needed to run the module. It should also include clear instructions on how to fetch the data and/or a clear explanation of why non-included data is confidential and contact information for data owners.	x	x	x
8. Does the module only call R code and packages that work on all operating systems? If the code includes any non-R code (e.g. FORTRAN, C++) will that code compile on all operating systems?	x	x
9. Is it licensed with the VisionEval license that allows the code to be freely distributed and modified and includes attribution so that the ‘provenance’ of the code can be tracked? Does it include an official release of ownership from the funding agency?	x
10. Does it only interact with the computing environment by returning a properly structured list to the framework (i.e. it does not modify the global environment, does not read or write files, and only calls framework functions that are allowed)?	x	x
11. Does it include regression tests to enable checking that consistent results will be returned when updates are made to the framework and/or R programming environment?	x	x
12. Does it include sufficient test coverage and test data? Does it pass the ‘testModule’ test which validates that it will run correctly in the model system? If possible, does it test against the existing RVMPO example?	x	x
13. Any other comments?		x	x	x

Feedback

The Review Team will provide feedback for each review criteria and will required submittal category as follows:

Status	Software	Documentation	Methods
Accept
Accept but recommend revisions
Do not accept
Abstain