Kv.put() method with Consul v1.10.1 results in KV entries that are visible via CLI/API, but not UI

[e1nh4nd3r]

Using the Diplomat gem as part of an internal Jenkins pipeline to take a .json file as a source of truth and insert the whole JSON payload into the KV. Our use-case here is using a GitOps workflow to ensure mcrouter configurations laid down by consul-template are only managed by this GitOps workflow (Jenkins Pipeline).

It successfully inserts into the intended Consul KV, but is only visible via the CLI/API and not via the UI. I am unsure if this is a Diplomat gem problem or if this is something wrong or incorrectly configured with our Consul clusters. The test I'd conducted below was done against a Development cluster only and in a test KV path to avoid trampling configurations in production. Any thoughts or ideas as to why this is happening?

Script invoked by the Jenkins pipeline:

# consul-uploader.rb
#
# Script to upload Consul KV data in JSON format for use by mcrouter (memcache router)
# See README.md for further details.

require 'json'
require 'yaml'
require 'diplomat'

def datacenters
    YAML.load_file('./consul.yaml')['consul_datacenters']
end

def post_to_consul
    # kv_path_prefix MUST end with a slash in order to provide a complete Consul KV path
    kv_path = "MCROUTER/"

    datacenters.each do |datacenter|
        # Set a variable to use with Diplomat.configure below to connect to the right cluster
        consul_cluster = URI("http://consul.service.#{datacenter}.consul")

        # Read in the JSON config file as a Ruby hash (config_file), then convert it to a string in the
        # expected format (kv_data)
        puts "Configuration file: ./datacenters/#{datacenter}.json"
        config_file = JSON.parse(File.read("./datacenters/#{datacenter}.json"))
        kv_data = JSON::pretty_generate(config_file, :indent => "\t")

        # Invoke Diplomat.config to talk to the correct cluster during each iteration
        Diplomat.configure do |config|
            config.url = consul_cluster
        end

        # Use Diplomat::Kv.put() alongside Diplomat::Kv.get() to confirm it has been updated.
        puts "Sending values to Consul #{consul_cluster} at #{kv_path}..."
        Diplomat::Kv.put("#{kv_path}", "#{kv_data}")
        puts "Retrieving new values from Consul #{consul_cluster} at #{kv_path} to confirm..."
        output = Diplomat::Kv.get("#{kv_path}")
        puts output
    end
end

post_to_consul

Results:

(USER)@ATH050296 consul-athena_mcrouter_json (development) $ ruby consul-uploader.rb
http://consul.service.(DATACENTER).consul
Configuration file: ./datacenters/(FILE).json
Sending values to Consul http://consul.service.(DATACENTER).consul at (USER)_TEST/MCROUTER/...
Retrieving new values from Consul at (USER)_TEST/MCROUTER/ to confirm...
{
        "pools": {
                "A": {
                        "servers": [
                                "(REDACTED):11211",
                                "(REDACTED):11211"
                        ]
                }
        },
        "route": {
                "type": "PrefixSelectorRoute",
                "policies": {
                },
                "wildcard": {
                        "type": "PoolRoute",
                        "pool": "A"
                }
        }
}

(USER)@ATH050296 ~ $ CONSUL_HTTP_ADDR='http://consul.service.(DATACENTER).consul' consul kv get (USER)_TEST/MCROUTER/
{
	"pools": {
		"A": {
			"servers": [
				"(REDACTED):11211",
				"(REDACTED):11211"
			]
		}
	},
	"route": {
		"type": "PrefixSelectorRoute",
		"policies": {
		},
		"wildcard": {
			"type": "PoolRoute",
			"pool": "A"
		}
	}
}

[e1nh4nd3r]

Tested just now by switching over to the HTTP API for Consul and using the httparty gem and I noticed something interesting. The values are posted now but for some reason Consul seems to think it's not a folder and is instead a KV pair?

Very interesting. I wonder if this has something to do with how I've implemented the Diplomat gem?

[e1nh4nd3r]

I've since created a branch in the aforementioned repo that feeds the Jenkins pipeline that does this to use the faraday gem and talk directly to the HTTP API endpoint rather than whatever the Diplomat gem is currently doing. In local testing it works fine, so I'll probably move that way for now.

[e1nh4nd3r]

Hey y'all, any updates here?

[pierresouchay]

if you look at your UI: https://user-images.githubusercontent.com/629551/208970185-64790602-675b-4641-8cdf-827ab87b3f5c.png

you'll see a login button... if this is in the API, this is done -> either you are not using the Consul UI correctly (not using the login button to see keys or the UI not having the right token to see keys), either it is a bug in Consul UI

Alternatively, try using consul kv get to see results or try pushing first an alternative (simpler k/v)... look at a basic hello world key.... does this works?

(look at doc on hashi learn on how using K/V with simple examples)

Regards