igconfig

This package can be used to load configuration values from a configuration file, environment variables, Consul, Vault and/or command-line parameters.

Install

go get github.com/worldline-go/igconfig

Example

cfg tag value are case insensitive and weakly dash/underscore so Network-Name, network_name, NeTWoK-NaMe and NeTWoKNaMe are same in both tag and configs.

NOTE if secret tag not found it will check cfg tag after that it will check variable's name.

type Config struct {
	NetworkName string `cfg:"networkName" env:"NETWORK_NAME"`
	// application specific vault
	DBSchema     string `cfg:"dbSchema"     env:"SCHEMA"       default:"transaction"`
	DBDataSource string `cfg:"dbDataSource" env:"DBDATASOURCE" log:"false"`
	DBType       string `cfg:"dbType"       env:"DBTYPE"       default:"pgx"`

	CustomConfig map[string]interface{} `cfg:"customConfig"`
}

// ---

var cfg Config

if err := igconfig.LoadConfig("myappname", &cfg); err != nil {
    log.Fatal().Err(err).Msg("unable to load configuration settings.")
}

Also check example:
Examples section
Example usage _example/readFromAll/main.go

Description

There is only a single exported function:

func LoadConfig(appName string, config interface{}) error

or if specific loaders needed:

func LoadWithLoaders(appName string, configStruct interface{}, loaders ...loader.Loader) error

There are also context accepted functions LoadConfigWithContext, LoadWithLoadersWithContext.

• appName is name of application. It is used in Consul and Vault to find proper path for variables also file name for file loader.
• config must be a pointer to struct. Otherwise, the function will fail with an error.
• loaders is list of Loaders to use.

Config struct

All exported fields of this structure will be checked and filled based on their tags or field names.

The field type is taken into consideration when processing parameters.

If the given value cannot be converted to the field's type, an error will be returned. Failing to provide proper value for a type is human-made error, which means that something is not right.

Fields can be given a tag with identifier "default", where the value of the tag will then be used as the default value for that field. If the default value cannot be converted to the field's type, an error will be returned.

Config struct can have inner structs as a fields, but not all Loaders might support them. For example Consul and Vault support inner structs, while Env and Flags don't.

Tags

Config structs can have special tags for fine-grained field name configuration.

There are no required tags, but setting them can improve readability and understanding.

cfg

cfg tag is fallback tag when no Loader-specific tag can be found. As such defining only this tag can be enough for most situations.

env

env tag specifies a name of environmental variable to get value from.

cmd

cmd tag is used to set flag names for fields.

secret

secret tag specifies name of field in Vault that should be used to fill the field.
If not exist it use as struct's field name.

default

default is special tag.

Unlike other tags it does not point to a place from which value should be taken, but instead it itself holds value.

default:"data" will mean that value of string field that has this tag will be data.

This tag is optional

Loaders

Loaders are actual specification on how fields should be filled.

igconfig provides a simple interface for creating new loaders.

Below is a sorted list of currently provided loaders that are included by default(if not stated otherwise)

Change order of loaders and configurations:

// this is the default loaders; change configurations and order or eliminate some loaders
loaders := []loader.Loader{
	&loader.Default{},
	&loader.Consul{},
	&loader.Vault{},
	&loader.File{},
	&loader.Env{},
	&loader.Flags{},
}

// read configurations with custom loaders
if err := igconfig.LoadWithLoaders("test", &conf, loaders...); err != nil {
    log.Fatal().Err(err).Msg("unable to load configuration settings.")
}

Default

This loader uses default tag to get value for fields.

Consul

Loads configuration from Consul and uses map decoder with cfg tag to decode data from Consul to a struct.

If not give CONSUL_HTTP_ADDR as environment variable, this config will skip!

For connection to Consul server you need to set some of environment variables.

Envrionment variable	Meaning
CONSUL_HTTP_ADDR	Ex: consul:8500, sets the HTTP address
CONSUL_HTTP_TOKEN_FILE	sets the HTTP token file
CONSUL_HTTP_TOKEN	sets the HTTP token
CONSUL_HTTP_AUTH	Ex: username:password, sets the HTTP authentication header
CONSUL_HTTP_SSL	Ex: true, sets whether or not to use HTTPS
CONSUL_TLS_SERVER_NAME	sets the server name to use as the SNI host when connecting via TLS
CONSUL_CACERT	sets the CA file to use for talking to Consul over TLS
CONSUL_CAPATH	sets the path to a directory of CA certs to use for talking to Consul over TLS
CONSUL_CLIENT_CERT	sets the client cert file to use for talking to Consul over TLS
CONSUL_CLIENT_KEY	sets the client key file to use for talking to Consul over TLS.
CONSUL_HTTP_SSL_VERIFY	Ex: false, sets whether or not to disable certificate checking
CONSUL_NAMESPACE	sets the HTTP Namespace to be used by default. This can still be overridden
CONSUL_CONFIG_PATH_PREFIX	sets the path prefix to be used by default.

While it is possible to change decoder from YAML to JSON for example it is not recommended if there are no objective reasons to do so. YAML is superior to JSON in terms of readability while providing as much ability to write configurations.

For better configurability configuration struct might include cfg tag for fields to specify a proper name to bind from Consul, if this tag is skipper - lowercase field name will be used to bind.

For example:

type Config struct {
    Field1 int `cfg:"field"`
    Str struct {
        Inner string
    }
}

will match this YAML

field: 50
str:
  inner: "test string"

Vault

Loads configuration from Vault and uses MapDecoder to decode data from Vault to a struct.

First Vault loads in finops/data/generic path and after that process application's configuration in finops/data/<appname> path.

generic path can have inner path, vault loader combine them.

To use additional path to load with appname set AdditionalPaths value in the loader.
Default value is [{Map: "", Name: "generic"}], Map is a wrapper for read value in key-value format, generic doesn't have map value so it will apply what read and append directly in our config.

To read more than one path, just append your path to the loader.VaultSecretAdditionalPaths slice.

Use Map value to wrap readed data with a key and Name is a path of configuration.

loader.VaultSecretAdditionalPaths = append(
    loader.VaultSecretAdditionalPaths,
    loader.AdditionalPath{Map: "loadtest", Name: "loadtest"},
)

If not given any of VAULT_ADDR, VAULT_AGENT_ADDR or CONSUL_HTTP_ADDR as environment variable, this config will skip!

If CONSUL_HTTP_ADDR exists, it uses Consul to get vault address.

Envrionment Variable	Meaning
CONSUL_HTTP_ADDR	get VAULT_ADDR from this consul server with vault service tag name.
VAULT_CONSUL_ADDR_DISABLE	disable to get VAULT_ADDR from this consul server.
VAULT_ADDR	the address of the Vault server. This should be a complete URL such as "http://vault.example.com". If need a custom SSL cert or enable insecure mode, you need to specify a custom HttpClient.
VAULT_AGENT_ADDR	the address of the local Vault agent. This should be a complete URL such as "http://vault.example.com".
VAULT_MAX_RETRIES	controls the maximum number of times to retry when a 5xx error occurs. Set to 0 to disable retrying. Defaults to 2 (for a total of three tries).
VAULT_RATE_LIMIT	EX: rateFloat:brustInt
VAULT_CLIENT_TIMEOUT	seconds
VAULT_SRV_LOOKUP	enables the client to lookup the host through DNS SRV lookup
VAULT_CACERT	TLS
VAULT_CAPATH	TLS
VAULT_CLIENT_CERT	TLS
VAULT_CAPATH	TLS
VAULT_CLIENT_CERT	TLS
VAULT_CLIENT_KEY	TLS
VAULT_TLS_SERVER_NAME	TLS
VAULT_SKIP_VERIFY	TLS
VAULT_APPROLE_BASE_PATH	set login path to be used by default.
VAULT_SECRET_BASE_PATH	set secret base path to be used by default.

For authentication, you should set VAULT_ROLE_ID and VAULT_ROLE_SECRET environment variables.

File

TOML, YAML and JSON files supported, and file path should be located on CONFIG_FILE env variable.
If that environment variable not found, file loader check working directory and /etc path with this formation <appName>.[toml|yml|yaml|json] (if there is more than appName with different suffixes, order is toml > yml > yaml > json).
The appName used as the file name is not the full name, only the part after the last slash. So if your app name is transactions/consumers/internal/apm/, the loader will try to load a file with the name apm.

The key is checked against the exported field names of the config struct and the field tag identified by cfg.
If a key is matched, the corresponding field in the struct will be filled with the value from the configuration file.

NOTE: if cfg tag not exists, it is still read values in file and match struct's field name!
Don't want to read a value just delete it in your config file or add cfg:"-".

FileLoader editable, you can add your own decoder or new file format or order of file suffixes.

Environment variables

For all exported fields from the config struct the name and the field tag identified by "env" will be checked if a corresponding environment variable is present. The tag may contain a list of names separated by comma's. The comparison is upper-case, even if tag specifies lower- or mixed-case. Lower-case environment variables are ignored.

Once a match is found the value from the corresponding environment variable is placed in the struct field, and no further comparisons will be done for that field.

Set value in inner struct:

type Config struct {
    Inner Inner
}

type Inner struct {
	GetENV       string `env:"TEST_ENV"` // env:"test_EnV" same as TEST_ENV
	// GetENV       string `cfg:"TEST_ENV"` // cfg tag also usable
}

To set GetENV value use INNER_TEST_ENV environment value. Or you can change INNER name with env tag.

type Config struct {
    Inner Inner `env:"IN"`
}

Now value use IN_TEST_ENV

NOTE: if env tag not exists, it will check cfg tag and if both not exists, it will check struct's field name as uppercase.

Flags (command-line parameters)

For all exported fields from the config struct the tag of the field identified by "cmd" will be checked if a corresponding command-line parameter is present. The tag may contain a list of names separated by comma's. The comparison is always done in a case-sensitive manner. Once a match is found the value from the corresponding command-line parameter is placed in the struct field, and no further comparisons are done for that field. For boolean struct fields the command-line parameter is checked as a flag. For all other field types the command-line parameter should have a compatible value. Parameters can be supplied on the command-line as described in the standard Go package "flag".

Log with context

Set a new zerolog logger and attach to the context, igconfig will use that context's logger.

// set new schemas for log
logConfig := log.With().Str("component", "config").Logger()
// replace context.Background() with own context
logCtx := logConfig.WithContext(context.Background())

// call igconfig with context
if err := igconfig.LoadConfigWithContext(logCtx, "test", &conf); err != nil {
    log.Ctx(logCtx).Fatal().Err(err).Msg("unable to load configuration settings.")
}

Print configuration

secret tag is disabled to print but if you want to print it add aditional option to secret called loggable or log.

Info string     `cfg:"info" secret:"info,log"` // Set secret's log option

Or you can set general log to manage it all tags in releated field.

User string     `cfg:"user" secret:"user" log:"true"` // Set general log

conf := config.AppConfig{} // set up config value somehow
// log is zerolog/log package
log.Info().
    EmbedObject(Printer{Value: conf}).
    Msg("loaded config")

Examples

Example usage of File

(
export CONFIG_FILE=_example/readFromAll/dataFile/test.yml
go run _example/readFromAll/main.go
)

Example usage of Vault server

Set Vault or Consul server address with releated environment variables.

Run a development vault

docker run -it --rm --cap-add=IPC_LOCK --name=dev-vault -p 8200:8200 vault

After that connect this vault with vault CLI app.

# export address for http
export VAULT_ADDR="http://127.0.0.1:8200"
# login with root token (appears in docker output)
vault login <token>
# unseal it
vault operator unseal <unsealkey>
# create kv secret engine
vault secrets enable -path=finops -version=2 kv
# create policy to read
{
cat <<EOF
path "finops/*" {
  capabilities = ["read", "list"]
}
path "finops/data/generic/super-secret" {
  capabilities = ["deny"]
}
EOF
} | vault policy write finops-read -

# create a approle with policy and enable connection without secret_id
vault auth enable approle
vault write auth/approle/role/my-role bind_secret_id=false secret_id_bound_cidrs="127.0.0.0/8,172.17.0.0/16" policies="default","finops-read"
# learn role-id
ROLE_ID=$(vault read -field=role_id auth/approle/role/my-role/role-id)

# fill some data
vault kv put finops/generic/keycloack @_example/readFromAll/dataVault/generic_keycloack.json
vault kv put finops/generic/super-secret @_example/readFromAll/dataVault/generic_supersecret.json
vault kv put finops/test @_example/readFromAll/dataVault/test.json
vault kv put finops/loadtest @_example/readFromAll/dataVault/loadtest.json

After that add our data in your finops kv section. Under usually should be a generic section and you should add keycloack and migration in there. also add your application name data in finops.

(
export VAULT_ADDR="http://localhost:8200"
export VAULT_ROLE_ID=${ROLE_ID}
# export CONSUL_HTTP_ADDR="consul:8500"
export MIGRATIONS_TEST_ENV="testing_testing_1234"
go run _example/readFromAll/main.go
)

Example usage of Consul server

Start consul agent with dev mode

docker run -it --rm --name=dev-consul --net=host consul:1.10.4

Go to localhost:8500 webui and add key values but for our tool folder should be finops.

It could be yaml or json format or you can handle by codec.Decoder interface.

Test it

export CONSUL_HTTP_ADDR="localhost:8500"
go run _example/readFromAll/main.go

Example usage of Consul dynamic listen

To listen a key, our function is easily usable.
While listening a key, you can restart consul server or close or not even started yet or delete key, it is totally safe.

Check example to get information:

export CONSUL_HTTP_ADDR="localhost:8500"
go run _example/dynamicConsul/main.go

Development

Package Test

Unit tests

go test --race -cover ./...

Code coverage report (Browser)

mkdir _out
go test -cover -coverprofile cover.out -outputdir ./_out/ ./...
# Auto open html result
go tool cover -html=./_out/cover.out
# Export HTML
# go tool cover -html=./_out/cover.out -o ./_out/coverage.html

JSON|YAML

Use yq tool for translate json to yaml or yaml to json.

# yaml to json
cat _example/readFromAll/dataFile/test.yml | yq

# json to yaml
cat _example/readFromAll/dataVault/generic_keycloack.json | yq -y