The goal of the publication process is to make available modeling tools that can be used or built on by the broader community. By this, we mean a complete tool that can be downloaded and run for a new location. A peer review process will be implemented for submitted projects. This process will complement, not replace, existing academic journals. The journals will be left to adjudicate the scientific content of research, while this organization will focus its review on whether the project meets basic standards of openness and usability. We will take steps to make the publication certification a prestigious achievement--one that can be bragged about alongside a journal article.
Process
The organization will maintain an editorial board, which will operate much like the editorial board of a journal. That board will maintain a much broader network of peer reviewers, with skills in different types of software systems and different types of models.
The basic standard for publication is listed in the review criteria. This will be judged by one or more peer reviewers who will be asked to download and test the model themselves. To respect their time, the goal is that this would take about the amount of time a paper review for a high-quality journal might take (likely not more than one day).
Review Criteria
The reviewers will be asked to score the project in the following areas.
Is the project relevant to travel modeling, and is it useful?
This is simply to affirm that we seek to advance travel modeling specifically, and not all of science anywhere.
Can someone, other than the original developer, download and use the model?
Are all of the software, data, parameters and related materials included, such that the model can be run by a third party? If proprietary software is used, is that software generally available, and is the exact version documented? Is both the operation and mechanics of the model clearly documented? Generally, can someone who is not the author follow what is going on.
Does the model should work for more than one location, and for more than one point in time?
In travel modeling, our goal is that the model be useful for other places, not just in one very specific circumstance. Further, out interest is in models that work for do not simply replicate existing conditions, but produce meaningful results over multiple years.
The authors will be expected to provide the inputs to set it up to run in multiple locations for multiple points in time, and the reviewer will simply verify. To facilitate this process, the organization will provide inputs in a standard format for selected cities and selected years, which authors may choose to use if they wish.
Are performance measures reported for more than one location, and for more than one point in time?
To enable the external evaluation of the model, performance measures should be reported for more than one location and for more than one point in time. We do not require that the performance measures be good--just that they be reported. It is perfectly acceptable for a model to produce poor results as long as the process by which those results were obtained is documented and transparent.
Scoring System
The scoring system will be adapted from the RAND Europe quality assurance system. The scoring options will be:
- 6 - Exceptionally good: novel, innovative, adds to and develops current best practice.
- 5 - Excellent work at the cutting edge of current practice.
- 4 - Appropriate work that reaches the standards required for publication.
- 3 - Needs minor refinement or correction to meet criteria.
- 2 - Needs significant revision to conform to criteria.
- 1 - Fundamentally flawed.
Projects will be expected to achieve a score of 4 or better in each of the key review criteria to be published. There will be options for revisions to projects that do not meet the standards on the first submittal.
What the Review Does Not Cover
As should be clear from the criteria, the focus here is on the openness and share-ability of the project, rather than on the underlying scientific merit. We defer to the traditional academic journals and scientific review process to judge the latter, and thus we complement, rather than compete with those journals. Submitted projects should note whether the associated research has been published in a peer reviewed journal or otherwise undergone peer review. In the case of existing methods (developed by someone else), those methods should be referenced. This information will help our users in evaluating whether the merits of the work have been scrutinized. Likewise, it is desirable for any associated papers to reference the published project. This gives readers the opportunity to actually use the project, and increases the overall exposure of the endeavour.
The criteria are intended to be broad enough that many different kinds of travel modeling research can conform. For example, we suggest that projects report performance measures, but do not specify what those performance measures are, because they may be different for different types of projects. The goal is that published projects meet a basic standard, not that they be flawlessly engineered software projects. There is room for further advancement through project incubation.
Publication
When a project is published, the organization will host it and make it available to our users. Users can access it with a degree of confidence that everything is available such that they can download it and use it themselves, with a moderate degree of effort. Authors will retain ownership and commit rights to the project such that they can continue to develop it as they see fit. The version that was peer reviewed will be tagged such that that exact version can be referenced for future use.