Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

One command for multiple templates when using wait{} feature #1799

Open
Dragonn opened this issue Aug 29, 2023 · 0 comments
Open

One command for multiple templates when using wait{} feature #1799

Dragonn opened this issue Aug 29, 2023 · 0 comments

Comments

@Dragonn
Copy link

Dragonn commented Aug 29, 2023

Consul Template version

Latest available in APT repository

consul-template v0.33.0 (15b89c9)

Configuration

  • configuration file config.hcl
consul {
  address = "consul.service.consul:8500"

  auth {
    enabled = false
  }

  token = "<censored>"
}

# not using Vault
vault {
  renew_token = false
}

wait {
  min = "5s"
  max = "15s"
}

template {
  source = "/etc/haproxy/consul/domains.ctmpl"
  destination = "/etc/haproxy/domains.map"
  exec {
    command = "echo /bin/systemctl reload haproxy"
  }
}

template {
  source = "/etc/haproxy/consul/haproxy.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  exec {
    command = "echo /bin/systemctl reload haproxy"
  }
}
  • template domains.ctmpl
{{ $x := key "kubernetes/proxy/test" | parseJSON }}
{{ $x.foo }}
  • template haproxy.ctmpl
{{ $x := key "kubernetes/proxy/test" | parseJSON }}
{{ $x.bar }}
  • key in Consul KV storage at /kubernetes/proxy/test
{
  "foo": "foo-val",
  "bar": "bar-val"
}

Command

consul-template --config config.hcl

Debug output

$ consul-template --config config.hcl
2023-08-29T15:19:56.016+0200 [INFO] consul-template v0.33.0 (15b89c9)
2023-08-29T15:19:56.016+0200 [INFO] (runner) creating new runner (dry: false, once: false)
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) final config: {"Consul":{"Address":"consul.service.consul:8500","Namespace":"","Auth":{"Enabled":false,"Username":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"d56a77c5-5c9b-2304-f3c2-d06d1432acbb","TokenFile":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":[],"Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"trace","FileLog":{"LogFilePath":"","LogRotateBytes":0,"LogRotateDuration":86400000000000,"LogRotateMaxFiles":0},"MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"domains.map","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":["echo /bin/systemctl reload haproxy"],"Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"consul/domains.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""},{"Backup":false,"Command":[],"CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"haproxy.cfg","ErrMissingKey":false,"ErrFatal":true,"Exec":{"Command":["echo /bin/systemctl reload haproxy"],"Enabled":true,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"User":null,"Uid":null,"Group":null,"Gid":null,"Source":"consul/haproxy.ctmpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":"","MapToEnvironmentVariable":""}],"TemplateErrFatal":null,"Vault":{"Address":"https://vault.edrive.cz:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false,"ClientUserAgent":null,"DefaultLeaseDuration":300000000000,"LeaseRenewalThreshold":0.9,"K8SAuthRoleName":"","K8SServiceAccountTokenPath":"/run/secrets/kubernetes.io/serviceaccount/token","K8SServiceAccountToken":"","K8SServiceMountPath":"kubernetes"},"Nomad":{"Address":"","Enabled":false,"Namespace":"","SSL":{"CaCert":"","CaCertBytes":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"AuthUsername":"","AuthPassword":"","Transport":{"CustomDialer":null,"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":5000000000,"MaxIdleConns":0,"MaxIdleConnsPerHost":100,"TLSHandshakeTimeout":10000000000},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true}},"Wait":{"Enabled":true,"Min":5000000000,"Max":15000000000},"Once":false,"ParseOnly":false,"BlockQueryWaitTime":60000000000,"ErrOnFailedLookup":false}
2023-08-29T15:19:56.016+0200 [INFO] (runner) creating watcher
2023-08-29T15:19:56.016+0200 [INFO] (runner) starting
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) running initial templates
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) initiating run
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) missing data for 1 dependencies
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) missing dependency: kv.block(kubernetes/proxy/test)
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) add used dependency kv.block(kubernetes/proxy/test) to missing since isLeader but do not have a watcher
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) was not watching 1 dependencies
2023-08-29T15:19:56.016+0200 [DEBUG] (watcher) adding kv.block(kubernetes/proxy/test)
2023-08-29T15:19:56.016+0200 [TRACE] (watcher) kv.block(kubernetes/proxy/test) starting
2023-08-29T15:19:56.016+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:19:56.016+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) starting fetch
2023-08-29T15:19:56.017+0200 [TRACE] kv.block(kubernetes/proxy/test): GET /v1/kv/kubernetes/proxy/test?stale=true&wait=1m0s
2023-08-29T15:19:56.017+0200 [DEBUG] (runner) missing data for 1 dependencies
2023-08-29T15:19:56.017+0200 [DEBUG] (runner) missing dependency: kv.block(kubernetes/proxy/test)
2023-08-29T15:19:56.017+0200 [DEBUG] (runner) missing data for 1 dependencies
2023-08-29T15:19:56.017+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:19:56.017+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:19:56.148+0200 [TRACE] kv.block(kubernetes/proxy/test): returned "{\n  \"foo\": \"foo-val\",\n  \"bar\": \"bar-val\"\n}"
2023-08-29T15:19:56.148+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) marking successful data response
2023-08-29T15:19:56.148+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) successful contact, resetting retries
2023-08-29T15:19:56.148+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) received data
2023-08-29T15:19:56.148+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) starting fetch
2023-08-29T15:19:56.148+0200 [TRACE] kv.block(kubernetes/proxy/test): GET /v1/kv/kubernetes/proxy/test?index=21107074&stale=true&wait=1m0s
2023-08-29T15:19:56.149+0200 [DEBUG] (runner) receiving dependency kv.block(kubernetes/proxy/test)
2023-08-29T15:19:56.149+0200 [DEBUG] (runner) initiating run
2023-08-29T15:19:56.149+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:19:56.149+0200 [DEBUG] (runner) rendering "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:19:56.156+0200 [INFO] (runner) rendered "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:19:56.156+0200 [DEBUG] (runner) appending command ["echo /bin/systemctl reload haproxy"] from "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:19:56.156+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:19:56.156+0200 [DEBUG] (runner) rendering "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:19:56.157+0200 [INFO] (runner) rendered "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) skipping command ["echo /bin/systemctl reload haproxy"] from "consul/haproxy.ctmpl" => "haproxy.cfg" (already appended from "consul/domains.ctmpl" => "domains.map")
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:19:56.157+0200 [INFO] (runner) executing command "[\"echo /bin/systemctl reload haproxy\"]" from "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:19:56.157+0200 [INFO] (child) spawning: sh -c echo /bin/systemctl reload haproxy
/bin/systemctl reload haproxy
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) enabling global quiescence for "853cb92a01067d68224ae836337b9e3c"
2023-08-29T15:19:56.157+0200 [DEBUG] (runner) enabling global quiescence for "f8ace4e53b32abdf7df9b5c7b7da2b80"
2023-08-29T15:20:10.282+0200 [TRACE] kv.block(kubernetes/proxy/test): returned "{\n  \"foo\": \"foo-val2\",\n  \"bar\": \"bar-val2\"\n}"
2023-08-29T15:20:10.282+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) marking successful data response
2023-08-29T15:20:10.282+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) successful contact, resetting retries
2023-08-29T15:20:10.282+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) received data
2023-08-29T15:20:10.282+0200 [TRACE] (view) kv.block(kubernetes/proxy/test) starting fetch
2023-08-29T15:20:10.282+0200 [TRACE] kv.block(kubernetes/proxy/test): GET /v1/kv/kubernetes/proxy/test?index=21107080&stale=true&wait=1m0s
2023-08-29T15:20:10.282+0200 [DEBUG] (runner) receiving dependency kv.block(kubernetes/proxy/test)
2023-08-29T15:20:10.282+0200 [DEBUG] (runner) initiating run
2023-08-29T15:20:10.282+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:20:10.283+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:20:10.283+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:20:10.283+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:20:10.283+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:20:10.283+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:20:15.284+0200 [DEBUG] (runner) received template "f8ace4e53b32abdf7df9b5c7b7da2b80" from quiescence
2023-08-29T15:20:15.284+0200 [DEBUG] (runner) initiating run
2023-08-29T15:20:15.284+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:20:15.284+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:20:15.285+0200 [DEBUG] (runner) rendering "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:20:15.291+0200 [INFO] (runner) rendered "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:20:15.291+0200 [DEBUG] (runner) appending command ["echo /bin/systemctl reload haproxy"] from "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:20:15.291+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:20:15.291+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:20:15.291+0200 [INFO] (runner) executing command "[\"echo /bin/systemctl reload haproxy\"]" from "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:20:15.291+0200 [INFO] (child) spawning: sh -c echo /bin/systemctl reload haproxy
/bin/systemctl reload haproxy
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) enabling global quiescence for "f8ace4e53b32abdf7df9b5c7b7da2b80"
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) received template "853cb92a01067d68224ae836337b9e3c" from quiescence
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) initiating run
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:20:15.292+0200 [DEBUG] (runner) rendering "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:20:15.293+0200 [INFO] (runner) rendered "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) appending command ["echo /bin/systemctl reload haproxy"] from "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:20:15.293+0200 [INFO] (runner) executing command "[\"echo /bin/systemctl reload haproxy\"]" from "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:20:15.293+0200 [INFO] (child) spawning: sh -c echo /bin/systemctl reload haproxy
/bin/systemctl reload haproxy
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:20:15.293+0200 [DEBUG] (runner) enabling global quiescence for "853cb92a01067d68224ae836337b9e3c"
2023-08-29T15:20:20.294+0200 [DEBUG] (runner) received template "f8ace4e53b32abdf7df9b5c7b7da2b80" from quiescence
2023-08-29T15:20:20.294+0200 [DEBUG] (runner) initiating run
2023-08-29T15:20:20.294+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:20:20.294+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) rendering "consul/haproxy.ctmpl" => "haproxy.cfg"
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:20:20.295+0200 [DEBUG] (runner) enabling global quiescence for "f8ace4e53b32abdf7df9b5c7b7da2b80"
2023-08-29T15:20:25.299+0200 [DEBUG] (runner) received template "853cb92a01067d68224ae836337b9e3c" from quiescence
2023-08-29T15:20:25.299+0200 [DEBUG] (runner) initiating run
2023-08-29T15:20:25.299+0200 [DEBUG] (runner) checking template 853cb92a01067d68224ae836337b9e3c
2023-08-29T15:20:25.299+0200 [DEBUG] (runner) rendering "consul/domains.ctmpl" => "domains.map"
2023-08-29T15:20:25.299+0200 [DEBUG] (runner) checking template f8ace4e53b32abdf7df9b5c7b7da2b80
2023-08-29T15:20:25.300+0200 [DEBUG] (runner) diffing and updating dependencies
2023-08-29T15:20:25.300+0200 [DEBUG] (runner) kv.block(kubernetes/proxy/test) is still needed
2023-08-29T15:20:25.300+0200 [DEBUG] (runner) watching 1 dependencies
2023-08-29T15:20:25.300+0200 [DEBUG] (runner) all templates rendered
2023-08-29T15:20:25.300+0200 [DEBUG] (runner) enabling global quiescence for "853cb92a01067d68224ae836337b9e3c"
^C2023-08-29T15:20:26.444+0200 [DEBUG] (cli) receiving signal "interrupt"
Cleaning up...
2023-08-29T15:20:26.444+0200 [INFO] (runner) stopping
2023-08-29T15:20:26.444+0200 [DEBUG] (runner) stopping watcher
2023-08-29T15:20:26.444+0200 [DEBUG] (watcher) stopping all views
2023-08-29T15:20:26.444+0200 [TRACE] (watcher) stopping kv.block(kubernetes/proxy/test)

Expected behavior

Only one command should be executed even though two templates are generated.

Actual behavior

On consul-template startup, both templates are generated and commands are deduplicated (got skipping command in log) and run reload command just once. On change in KV store, templates are regenerated (which is OK) and reload commands are executed twice (once for each template)

Btw this feature works as expected when

  • rendering simplest possible template, e.g. whole key {{ key "kubernetes/proxy/test }}
  • not using wait feature in configuration file

Steps to reproduce

  1. create value anywhere in Consul KV storage
  2. create two non-trivial templates
  3. use wait section in configuration file
  4. run consul-template process
  5. change value in Consul KV

References
@Dragonn Dragonn changed the title One command for multiple templates when using wait One command for multiple templates when using wait{} feature Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Dragonn