How to get Health info that's shown in the SysManager

[wally007]

Hello,

we are using default installation of configs supplied with harvest (latest 22.0.2 version) but we've noticed that even though we have
one emergency warning in SysManager

All Health metrics *__new_status in harvest were "0".

In our case, the error was due to the terminated LDAP connection.

Is is possible possible with Harvest to export metric that there is an Alert like in SysManager ?
And if so, can you please include it in the default configs ?

There is no need for details - but it would be nice if "cluster healthy" in Harvest meant "cluster healthy" in SysManager Dashboard.

We would like to be alerted (and therefore export the metric) whenever there's an "alert" in SystemManager (doesnt matter the source)

Thank you.

EDIT: I'd expect metrics cluster_new_status{datacenter="EQX",cluster="XXX",status="ok"} 1 or node_new_status{datacenter="EQX",cluster="XXX",node="XXX-01"} 1 to be used for this purpose. Is that expectation not correct ?

[cgrinds]

hi @wally007 you're right - we took a look at this and have root caused the issue and have some ideas on how to improve

Root cause

Harvest is using the diagnosis-status-get ZAPI to report on cluster health. That ZAPI returns the overall system health (which in this case returns A-OK), while System Manager is showing an event delivered via ONTAP's event monitoring service (EMS).

I was able to reproduce this on a local system and see the same thing. diagnosis-status-get returns healthy, as does diagnosis-subsystem-config-get-iter while support/ems/events does not.

bin/rest -p u2 show data --api  support/ems/events | gron | rg -o 'message.severity.*$' | sort | uniq -c
fetching href=[api/support/ems/events?return_records=true&fields=*]
   1 message.severity = "alert";
 381 message.severity = "debug";
  28 message.severity = "emergency";
1552 message.severity = "error";
1827 message.severity = "informational";
 307 message.severity = "notice";

Next Steps

Harvest should include an event monitoring service collector that gives you a way to register interest in events returned from the /support/ems/events REST endpoint. Something like a Harvest EMS template with a set of ems filters, rules, etc. That would let you respond to ems events of interest. I'll create a feature request to capture that.

[cgrinds]

Issue #892

[wally007]

Thank you very much. If possible, please let us know ( or document in the created GitHub issue - which I subscribed) if the new health metric will be introduced for EMS events or the existing cluster_new_status will include it.

[cgrinds]

hi @wally007 we're going to keep cluster_new_status as-is since it's what ONTAP returns directly and we'll take up adding an EMS collector in a later release