SVM Latency numbers differ significantly on Harvest 1.6 vs Harvest 2.0

[jmg011]

We have some users using both harvest 1.6 dashboards and Harvest 2.0 dashboards. Some of the metrics I have set for Harvest 2.0 shows high latency at SVM level compared to harvest 1.6 metrics.

For example:

In Harvest 1.6 query, the SVM level latency is using vol_summary avg_latency -> This is 0.1 ms for an svm in my test case. This matches with the "statistics vserver latency show" command on the filer

_[netapp].[perf].[$Group].[$Cluster].[svm].[$SVM].[vol_summary].[avg_latency]_

In Harvest 2.0 query, for SVM level latency, I'm using below query. This query is reporting 25 ms latency for the same SVM. If I remove CIFS latency, it still reports almost 24 ms latency for my svm test case during the same time. This is not matching with the "statistics vserver latency show" or "statistics volume latency show" commands on the filer.

avg(svm_nfs_latency{svm=~"$SVM"}) + avg(svm_cifs_latency{svm=~"$SVM"})

I'm wondering why these two numbers differ so significantly?

In Harvest 2.0, when I calculate latency of all volumes on svm and sum it, it is still higher like 17ms for this test case SVM.

sum by (svm)(volume_avg_latency{svm="testcase_svm"})/1000

Are there different objects being used for both harvest 1.6 vs harvest 2.0 SVM latency calculations?

[cgrinds]

hi @jmg011, if memory serves, 1.6 was using the perf zapi volume:node for vol_summary. Harvest 2 also collected volume:node via the volume_node.yaml template. Can you check that metric in Harvest 2 and see if it matches what you expect?

Looks like that template use node_vol for the metric prefix.

name:                     VolumeNode
query:                    volume:node
object:                   node_vol

[jmg011]

@cgrinds yes in harvest 1.6 it used vol_summary per svm, and in harvest 2.0, we are using svm_latency_* metrics to for svm level metrics. Node level latency metrics won't help get the svm level latency.

I'm wondering what commands does harvest use to calculate SVM level latency for harvest 1.6 vs harvest 2.0? Harvest 1.6 latency is matching with system level SVM metrics but it does not match with Harvest 2.0 svm_latency_* metrics.

[cgrinds]

@jmg011 We'll take a look. Sounds like 1.6 may be doing a roll-up/aggregation that 2.0 isn't doing.

[cgrinds]

hi @jmg011, we've got a better idea of what's going on here and few changes we're going to make.

The reason you were seeing different values between 1.6 was 2.0 is because they're using different aggregations. Harvest 1.6 was using the volume object + a plugin that aggregated the counters by svm. The plugin stored those aggregated metrics on the svm.vol_summary graphite leaf and that's what the SVM dashboard used.

Harvest 2 is using the volume object also, but without aggregation. The Harvest 2 dashboard does some simple aggregations of the volume metrics with average and sum.

The ONTAP cli command statistics vserver show does a roll-up of volume metrics at the svm level similar to what Harvest 1.6 is doing. That's why you saw similar values.

We're going to add a new template to collect volume metrics aggregated by svm, similar to statistics vserver show and update the SVM dashboard to use these metrics.

The template will look like this is if you want to try it.

File: conf/zapiperf/cdot/9.8.0/volume_svm.yaml

name:                     VolumeSvm
query:                    volume:vserver
object:                   svm_vol

instance_key:             uuid

counters:
  - instance_name         => svm
  - read_data
  - write_data
  - read_ops
  - write_ops
  - other_ops
  - total_ops
  - read_latency
  - write_latency
  - other_latency
  - avg_latency

export_options:
  instance_keys:
    - svm

Corresponding addition to default.yaml or custom.yaml

  VolumeSvm:              volume_svm.yaml

[jmg011]

@cgrinds - Awesome. Thank you so much for debugging it fast. I will try the new template and will provide you feedback.

[cgrinds]

hi @jmg011 any feedback on this? If this looks good to you, we'd like to target these changes for the next release and update the dashboards to use svm_vol.

[jmg011]

@cgrinds I had a crazy loaded sprint. Let me test this on Monday/Tuesday and get back.

[jmg011]

@cgrinds I tested this template for svm_vol metrics. I ran the custom template with 60s data schedule and on the filer I ran statistics vserver show for a specific vserver with 60s interval and 50 iterations to compare the data. It is pretty close.

Observation: This new template has avg, read, write and other latency for svm, but it does not have the breakdown of other latencies like we have for svm_latency metrics. We will have to keep the original svm metrics as well. I use some of those metrics for the alerting.

[cgrinds]

thanks for trying out the new template. The latest nightly build includes the template and an updated SVM dashboard that uses the new template if you want to try it out.

Both of these changes will be included in the next release thanks to you.

This new template has avg, read, write and other latency for svm, but it does not have the breakdown of other latencies like we have for svm_latency metrics. We will have to keep the original svm metrics as well. I use some of those metrics for the alerting.

We plan on adding the new template and updated SVM dashboard but will keep the existing templates. Which template are you pulling the original SVM metrics from? Is it one of Harvest's out-of-the-box templates or something you added? Either way, our change is adding a new template, not modifying any of the existing ones.

[jmg011]

@cgrinds For SVM performance, I'm using the default template, but for other SVM information I have created custom metric template. All of the dashboards I created are with many custom metrics details from filers and some from the internal platform. So I'm not using any default dashboards as is. It is much more customized. For every upgrades, I manage all those templates separately and have to do validation for the new templates to ensure nothing breaks with the upgrades w.r.t dashboards, alerting etc.. I do have some feedback in regards to the upgrade related recommendation, I will address it in a separate discussion thread as it has more detailed information.