Tracking Metrics for Success

[PhilipBale]

Metrics for Success

Hi All,

The purpose of this issue is to determine metrics for tracking and measuring code reuse and open source code policies. These metrics are meant to, over time, track whether or not adoption of these policies are successful. The recommendations are broken down into the two policies: Code Reuse & Open Source Code.

Code Reuse Policy Metrics

1. Repo Reuse Metadata – Metadata in repos.json (such as array of repository links) that designate whether a project has components that originate from other federal government repositories.

   Exempted Repositories – For repositories that are excluded from the source code policy, would it make sense to include repo reuse metadata?

   Example JSON

   gov_code_usage: [{
   	project_name: required,
   	repo_url: optional,
   	project_url: optional
   }]
   

2. Package Metadata – For code available in package managers, create a metadata schema that defines the name of the package and the package managers it is available on. Doing so will allow for simpler dependency reconciliation in recommendation remove the repo wiki if not in use #3. It may also allow us to pull usage data from the package managers themselves.

   Example JSON

   packages_info: [{
   	package_name: required,
   	package_manager: required
   }]
   
   

3. Dependency Checking – A tool built for (1st level?) dependency discovery & checking in regards to package managers such as

   1. bower.json
   2. packages.json
   3. ruby gemfiles
   4. and many more

With dependency data, package metadata, and reuse metadata, we could compile Code.gov dependency trees to visually and numerically represent code reuse. This would also be helpful in regards to displaying "Built With" info on Code.gov.

Open Source Code Policy Metrics

1. GitHub/Repo Statistics – Statistics that can be pulled from any given repository include:
   1. Number of stars/followers/forks
   2. Commit count & frequency
   3. Pull Request approval/denial rates & time frames
   4. Addition/deletion count & frequency
   5. Number of contributors
   6. Contributor activity (including repeat and one-time contributors)
   7. Issue open/close rate

These statistics are most important for open source tracking, but they would also be beneficial in determining the overall activity, liveliness, and importance of a repository. All metrics being tracked on a daily or weekly basis would be useful for time-based analysis. Anything shorter (such as hourly) would likely be too frequent. It would make sense to start with GitHub and then proceed to GitLab, BitBucket, etc. as GitHub is the largest provider. https://developer.github.com/v3/repos/statistics/.

2. Code.gov Statistics – Track traffic to individual Code.gov project pages by time segments. We can compare project page metrics to increases/decreases in repo metrics. Similarly, if a community/discussion feature is built out for Code.gov, we could track contribution and traffic metrics.
3. Google Trends – Use google trends to analyze public interest in the project name

Please post suggestions/additions!

[agiron123]

I'm not sure if this is out of scope for the project, but it may be a good idea to also look at the software licenses for projects that code.gov uses, and check whether or not code.gov is in compliance with those software licenses.

[DanielJDufour]

Great suggestions! We'll definitely consider them going forward. Thanks!