What nonfunctional attributes should we measure for every release?

[mpstefan]

As our product becomes more established, it will become more and more important to measure certain non-functional attributes of our product for changes from release to release. This could include things such as performance or longevity to ensure that we are not, for instance, making our product significantly slower following a release.

So here are a few questions for the community:

1. What non-functional attributes should we measure as part of our release process? Why?
2. How should we measure these attributes?
3. What should be the priority order we should start measuring these attributes?
4. (More for maintainers) What requirements do we need to meet in order to start measuring these attributes?

Following this discussion, I'll create epics to track the attributes we feel are the most urgent to measure.

Thanks all!

[pleshakov]

A few things that come to mind:

(1) Zero downtime upgrades

Upgrade to new version doesn't interrupt traffic flow through the data plane.

(2) Reconfiguration performance

Ensure the control plane is able to process X number of resources in Y time and apply the configuration to the data plane

(4) Data plane performance

Ensure data plane can handle X number of requests per second with Y number response time.

(5) Scale testing

• Scaling the number of resources (HTTPRoutes, Gateways, Secrets, Pods )
• Ensure NKG can process those resources and handle the traffic
• Ensure a new version doesn't increase resource usage compared with the previous one.

(6) Reload count

Since we need to reload NGINX to apply config updates, come up with some configuration and change it over time. Ensure that next releases doesn't increase the number of observed reloads.

(7) Longevity

Ensure NKG doesn't leak resources after X number of days under normal conditions

[mpstefan]

This is a great list - are you proposing the order you presented should be the priority for us tackling them?

[pleshakov]

thanks. no particular order. except for reload count. ultimately, less number of reloads, the better, so better to prioritize it.

[sjberman]

In terms of (2) and (4) (I don't see a 3), I think we just start by getting a baseline number of where we're at. In the future we can look to improve this performance if we feel it's necessary, but at least we just keep track of this number as we do each release, to see how we naturally improve or get worse. (6) may be handled similarly. Just get numbers right now.

(5) and (7) are essentially writing tests and run them to see if we break or leak memory.

[kate-osborn]

This list looks good to me. I would add:
(8) Graceful restart
Ensure the control plane can recover and reconfigure itself after a restart event

[vepatel]

I'm guessing for 1.0 this will be all about taking a baseline measurements for all of these attributes as we don't have a anything to compare against yet?

[mpstefan]

Moving forward with this in epic #616