statsd/dogstatsd support for tracking consul template events

[ross]

It would be really handy to have built in support for sending statsd/dogstatsd metrics on notable consul-template events. The most obvious of which being restarts. For statsd that'd probably be something like consul-template.<identifier>.restart, for dostatsd consul-template.restart#template:identifier. They may not currently be something obvious to use for identifier currently, perhaps the "flag" that enables stats would include that.

This originally occurred to us while trying to debug/track down hashicorp/consul#2430 where it would have been really useful to be able to track which processes were being restarted when at a glance.

Originally brought up on the list where @slackpad mentioned "I can think of some other situations where watching restarts could be useful, such as with quiescence timers." General/immediate solution suggested there was to use a wrapper around the restart command that emits the stats.

[ross]

As a (hopefully temporary solution) I'm using something similar to the following as a wrapper around the restart command. For datadog the #{template} would be moved to a tag rather than part of the name.

#!/usr/bin/env ruby

require 'socket'

if ARGV.size < 2
  $stderr.puts "usage:\n  #{$0} <template-name> <cmd...>\n"
  exit 1
end
template = ARGV.shift

begin
  u = UDPSocket.new
  u.send("consul_template.#{template}.restart:1|c", 0, "127.0.0.1", 8125)
rescue
  $stderr.puts 'Failed to send stats'
end

exec(*ARGV)

[pvandervelde]

@sethvargo Do you have any idea about the priority of getting consul-template to produce metrics. We would love to monitor consul-template. We're specifically interested in whether it is up and if it has a vault token or not.

[findkim]

@ross thank you for submitting a detailed issue and for your patience. I'd like to inform you and @pvandervelde that metrics is now targeted for the 0.26.0 release.

Thank you for the events listed above -- restarts, quiescence timers, and vault token. I am currently collecting areas of Consul Template that would be useful to report metrics on, and will consider these suggestions.

[ross]

👋 @findkim & @pvandervelde. I realize it's a bigger shift than the stats work that was added in #1378, but identifying the templates only by hash in the metrics doesn't really help much in cases where you have many templates running on a host. We have 8+ in some situations and having the hash which is only consistent for a while and can't easily be tracked back to the template won't let us replace our wrapper.

The same goes really for the info in logging from consul-template which is very difficult to use/make sense of since it only refers to things by hash as well.

Don't have a solid thought on what the best solution would be. I guess either using the full file-path & name, filename, or adding some sort of configuration key for template "name." I'd probably lean towards the template name if I were picking.

[findkim]

Hi @ross, appreciate the follow-up on this ticket!

I agree with you that hashes obfuscate the mapping of logs and metrics back to a template. The hashes are generated based on the content of the template, and so if the template changes so would the hash, and results in the issue you mentioned about only being consistent for a while.

In attempt to keep the PR #1378 narrowly scoped to one enhancement, it does seem to double down on the difficulty you outlined.

Thanks for the template identifier suggestions. Some combination of full file-path / filename are reasonable options for templates that are files. Since templates can also be configured raw in template.contents, I am also leaning towards an optional unique "name" for templates. I opened a new issue to track this enhancement #1381 and we can continue the conversation there.

[eikenb]

Moving to go-metrics. See #1395.