Possible overlap of the metrics?

[jmg011]

We have two metrics that we are using for alerting on node & cluster status.

node_status
cluster_subsystem_new_status

For 7mode node use-case, both of these metrics have degraded status, firing two alerts for the same node.

I can inhibit one of the alerts when the other one is firing, but before I do that I want to understand if there could be any scenarios where node can be in degraded state but it does not show up in cluster_subsystem_new_status metric or vice versa?

[cgrinds]

hi @jmg011

Do you mean node_new_status instead of node_status?

From what I've been able to find in the documentation, it looks like it's better to use node_new_status because it checks more systems than the subsystem command.

Specifically system health config show checks the health of node-connect and system while system health subsystem show only checks the health of the node level monitors.

system health config show

node_new_status is created from the diagnosis-config-get-iter ZAPI in template conf/zapi/7mode/8.6.0/status_7.yaml and corresponds to the ONTAP CLI command system health config show

fas2240-2> system health config show

Node          Monitor                Subsystem       Health
------------- ---------------------- --------------- ------------------
fas2240-2     node-connect           SAS-connect, CIFS-NDO ok
fas2240-2     system                 SAS-connect, Environment, Memory, Service-Processor, Motherboard, IO ok
2 entries were displayed.

The documentation for this CLI command says

Description
The system health config show command displays the configuration and
status of each health monitor in the cluster. The command shows a health status for
each health monitor. The health status is an aggregation of the subsystem health for
each subsystem that the health monitor monitors. For example, if a health monitor
monitors two subsystems and the health status of one subsystem is "ok" and the other
is "degraded", the health status for the health monitor is "degraded".

system health subsystem show

while cluster_subsystem_new_status is created from the the diagnosis-subsystem-config-get-iter ZAPI in template conf/zapi/7mode/8.6.0/subsystem.yaml and corresponds to the ONTAP CLI command system health subsystem show.

The documentation for this CLI command says

Description
The system health subsystem show command displays the health status of
each subsystem for which health monitoring is available. This command aggregates
subsystem health status from each node in the cluster. A subsystem's health status
changes to "degraded" when a health monitor raises an alert. You can use the system
health alert show command to display information about generated alerts

fas2240-2> system health subsystem show

Subsystem         Health
----------------- ------------------
SAS-connect       ok
CIFS-NDO          ok
2 entries were displayed.

[jmg011]

@cgrinds Thanks for the documentation. We don't use node_new_status. I have customized the metrics to standards so anything that is new , I have renamed with node_status or cluster_status etc.

I have also made all the respective changes in the template and alerts as well so that there are standards.

You have addressed my question for this discussion. Thank you!