Add extra telemetry to monitor failures

[the-nando]

Fixes: #661

[lgfa29]

We have a standard set of metrics already of these kind of plugin invocations:
https://developer.hashicorp.com/nomad/tools/autoscaling/telemetry#scaling-metrics

But it's only for a few actions, so it would good to expand to others as well.

Although having to remember to emit these metrics all the time is kind of annoying and easy to forget, so I think we can do something better.

Instead of leaving it to the caller to remember to handle this, the way I'm thinking about is to have the plugin manager return an instance of the requested plugin wrapped in a struct that handles the metric.

Here is a quick prototype of how this would look like for a target plugin: 4340235

This way, emitting the metrics is completely transparent to callers and we can ensure the metrics are always being handle.

What do you think of this approach?

(cc @Juanadelacuesta and @jrasell as you've been digging into the autoscaler recently)

[jrasell]

@lgfa29 that approach makes sense to me and acts like a middleware/wrapper which should make plugin developer life easier.

[lgfa29]

Thanks for the extra input 😄

@the-nando do you think you would be able to implement something like this?

[the-nando]

Thanks @lgfa29 for the input, much better and cleaner than my hacky attempt at wiring metrics from within the plugin.
I'll take a stab at it in a week or so.

[lgfa29]

Thank you! Feel free to reach out if you need any help 🙂