Define and track metrics for success for CrateKube

[kallenboone]

Problem to solve

Without measurable metrics for success, it will be impossible to determine the health and success of CrateKube engineering. We need metrics for success to be defined and tracked in order to measure success and iterate on process based on the metrics data.

Proposal

The metrics for success will be outlined in a document in the CrateKube GitHub repo and a dashboard will be created to track the defined metrics. Once the data has been gathered, it will be analyzed to refine the metrics, set targets, and inform our discussions around process. As we continue to gather data and have discussions, the metrics will be adjusted and iterated on to better capture our health and success.

Intended users

This work will be used by all CrateKube community members.

Further details

Currently proposed metrics for success include the following:

• Throughput: the number of MRs/PRs accepted into master in a week or month
• MR/PR Rate: the number of MRs/PRs accepted into master divided by the number of contributors
• Number of reviews submitted with comments
• Number of interactions within the community

This list will likely expand and change as we gather more data and the project grows.

Permissions and Security

There are no security implications in publishing this documentation. All of this information can be made public, as none of this information should be secret.

Documentation

The metrics for success, including the reasoning behind each, will need to be outlined in a document on GitHub.

How can we measure success?

Success will be measured by how successful our assessment of success can be measured for the engineering efforts of the project.

When is this feature complete?

• The metrics for success are outlined in a document in GitHub
• Data for the defined metrics for success is collected and displayed in a public dashboard
• An issue exists to create targets for each metric add them to the dashboard

Links / references

None

[pdgrenon]

I think we are mixing concerns with this issue. I think the intent is to measure "team success", but this issue talks about measuring "project success." Which one are we aiming to define and measure in this issue?

[kallenboone]

Updated to focus on the engineering team effort.

[code4dc]

• Explore existing metrics dashboards
• Figure out what metrics are easy to track and which ones need to have some complex logic
• Port existing metrics documentation to cratekube/cratekube
• Set up POCs with some of the possible dashboard solutions and ask team for preferences on which one to incorporate

[code4dc]

Using GrimoireLab (Kibana and ES) the PRs/Contributor metric is impossible to display. We can show the number of PRs and Contirbutors in a specific window but not their ratio. Because....reasons: elastic/kibana#4707

[code4dc]

Deployed Metrics dashboard to eushvart-dev in Crate. Just click on the kibiter port# to open the dashboard.
https://console.crate.farm/env/1a2841938/apps/stacks/1st20907

[code4dc]

The work performed has been deemed sufficient and effort will be shifted to higher priority tasks.