docker stack deploy in 1.13 doesn't load .env file as docker-compose up does

[twang2218]

To test docker stack deploy --compose-file function, I load one of my sample docker-compose.yml:

version: '3'
services:
    nginx:
        image: "${DOCKER_USER}/lnmp-nginx:v1.2"
        build:
            context: .
            dockerfile: Dockerfile.nginx
        ports:
            - "80:80"
        networks:
            - frontend
        depends_on:
            - php
    php:
        image: "${DOCKER_USER}/lnmp-php:v1.2"
        build:
            context: .
            dockerfile: Dockerfile.php
        networks:
            - frontend
            - backend
        environment:
            MYSQL_PASSWORD: Passw0rd
        depends_on:
            - mysql
    mysql:
        image: mysql:5.7
        volumes:
            - mysql-data:/var/lib/mysql
        environment:
            TZ: 'Asia/Shanghai'
            MYSQL_ROOT_PASSWORD: Passw0rd
        command: ['mysqld', '--character-set-server=utf8']
        networks:
            - backend
volumes:
    mysql-data:

networks:
    frontend:
    backend:

In the image section of service nginx and php, I used ${DOCKER_USER} to get the docker id from environment variables. And if I use docker-compose up, it will load .env file as default envvar files, which content is:

DOCKER_USER=twang2218

However, if I use docker stack to deploy this docker-compose.yml, I will got following errors:

$ docker stack deploy --compose-file docker-compose.yml lnmp
Ignoring unsupported options: build

Creating network lnmp_frontend
Creating network lnmp_backend
Creating network lnmp_default
Creating service lnmp_php
Error response from daemon: rpc error: code = 3 desc = ContainerSpec: "/lnmp-php:v1.2" is not a valid repository/tag

As you can see, as docker stack deploy command didn't load .env file, the ${DOCKER_USER} was replaced by empty string, which cause image name become invalid.

If .env file was loaded, the final image name should be twang2218/lnmp-php:v1.2.

The environment substitution is actually working, if I run the command this way:

$ DOCKER_USER=twang2218 docker stack deploy --compose-file docker-compose.yml lnmp
Ignoring unsupported options: build

Creating network lnmp_frontend
Creating network lnmp_backend
Creating network lnmp_default
Creating service lnmp_mysql
Creating service lnmp_nginx
Creating service lnmp_php

And we can verify it's working by docker service inspect command:

$ docker service inspect lnmp_php | grep Image
                    "Image": "twang2218/lnmp-php:v1.2@sha256:4f1aef1350aeef3f757f6b6da8f2e1a79ff849f61382320e4b668bfe2b0d1c5a",

The image name is twang2218/lnmp-php:v1.2, which is correct.

I tested this feature on Digtial Ocean droplet, which installed docker 1.13.0-rc2 via docker-machine.

Here is the version:

$ docker version
Client:
 Version:      1.13.0-rc2
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   1f9b3ef
 Built:        Wed Nov 23 06:32:39 2016
 OS/Arch:      linux/amd64

Server:
 Version:             1.13.0-rc2
 API version:         1.25
 Minimum API version: 1.12
 Go version:          go1.7.3
 Git commit:          1f9b3ef
 Built:               Wed Nov 23 06:32:39 2016
 OS/Arch:             linux/amd64
 Experimental:        false

Here is the docker info:

root@d1:~/docker-lnmp# docker info
Containers: 7
 Running: 1
 Paused: 0
 Stopped: 6
Images: 4
Server Version: 1.13.0-rc2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 43
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: vyf3mgcj3uonrnh5xxquasp38
 Is Manager: true
 ClusterID: jb8rxvd6ptrn3psfkiixxed7r
 Managers: 1
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 138.197.195.206
 Manager Addresses:
  138.197.195.206:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-51-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 488.5 MiB
Name: d1
ID: E6UB:PHX6:I2KY:Q35T:PCCI:MFDQ:ZMMN:2X7K:DEOZ:PAP7:4BUC:FP6X
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=digitalocean
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[dnephin]

This is by design. The .env support is a feature of Compose, not of the file format.

We can discuss adding this for a future release, but I don't know if it's really the best option.

[twang2218]

The .env support is quite useful in Compose, we used it in many of our compose files. It separates dynamic parts and static parts of docker-compose.yml file. With load .env by default, we can just provides different .env file for different environments, and keeping the docker-compose.yml and related scripts static.

It's not possible to use env_file or environment in docker-compose.yml to achieve the same envvars substitution result. .env is the most easy way to do it.

Otherwise, we have to prefix export in each line of .env file and manually source .env every time before loading the docker-compose.yml, and the extra steps sometimes are prone to mistake.

[vovimayhem]

I've just noticed this.
I assumed/expected it to work in the same way it does for Compose, but is not the case for docker deploy.

In my particular case, I'm was expecting to use the .env file to store sensitive data (passwords, API keys, etc) used in the services I'll be creating in the stack:

version: '3'

volumes:
  data:
    driver: local

networks:
  backend:
    driver: overlay

services:
  rabbitmq:
    image: rabbitmq:${EXCHANGE_RABBITMQ_TAG}
    volumes: [ "data:/var/lib/rabbitmq" ]
    logging: { driver: gelf, options: { gelf-address: "udp://0.0.0.0:12201" } }
    networks: [ "backend" ]
    ports: [ "15672:15672", "5672:5672" ]
    environment:
      RABBITMQ_DEFAULT_USER: ${EXCHANGE_RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${EXCHANGE_RABBITMQ_PASS}
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.queue-host == true

While this Compose file will be checked in on Git, the .env file would be ignored.

I've followed the whole stuff/history of the *.dab vs compose files, and I feel you guys are trying to avoid something - or proposing a better solution - but I lost track of the whole discussion...

[akhildangore]

Hello All,

Docker version: v1.13.0-rc4

I am getting error : Ignoring unsupported options: build, Does it mean it will not create build from Dockerfile ?

And also getting same error for network_mode: Ignoring unsupported options: network_mode.

Below is command:
DOCKER_IMAGE=akhil123 docker stack deploy -c docker-compose.yml foo

Many thanks in advance.

[thaJeztah]

@akhildangore please don't comment on issues with questions that are not directly related. The docker stack deploy feature, by design does not perform builds. Reason for this is that a build is performed on the host that the command is run from, so the image will only be available on that node. When deploying that image in a Swarm, the service cannot be started on other nodes. To deploy services, make sure the image you're deploying is pushed to a registry (or for testing; make sure the image is available on every node in the swarm)

[vovimayhem]

Are there any cases/discussion in favor of/against supporting this behavior on docker deploy?

[thaJeztah]

@vovimayhem no decision was made yet on .env file, but you may be interested in #30144, which adds support for secrets to the compose file

[vovimayhem]

I've just found that discussion. Excellent!

[whoan]

I am using a bash function as a workaround.

You may adapt it to your needs until the secrets feature be released:

dsd() {
    stack=${1:-${PWD##*/}} # by default, the name of the cointaining folder
    compose_file=${2:-docker-compose.yml}

    if [ ! -f $compose_file ]; then
        echo "Misses compose file: $compose_file" >&2
        return 1
    fi

    # execute as a subcommand in order to avoid the variables remain set
    (
        # export variables excluding comments
        [ -f .env ] && export $(sed '/^#/d' .env)

        # Use dsd your_stack your_compose_file to override the defaults
        docker stack deploy --compose-file $compose_file $stack
    )
}

[thaJeztah]

For those subscribing to this issue; secrets support for docker-compose files will be included in the 1.13.1 release, which should be not too far in the future

[vovimayhem]

@whoan I'm using this as a workaround:

env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml [STACK_NAME]

That way, variables don't get stuck on the terminal window

[andyfinch]

As per many I hit this issue and found this. A solution that has worked for me is as follows:
Create .env (or any other file name) with your variables:

MYVAR1=foo
MYVAR2=bar

Create startmystack.sh script containing the stack command required:

source .env
export MYVAR1 MYVAR2
docker stack deploy --compose-file my-docker-composes.yml mystack

Variable picked up using source from .env, exported and then are available for substitution in the compose file for Swarm.

[x-yuri]

@andyfinch Alternatively:

#!/bin/sh -eu
set -a
. .env
docker stack deploy -c docker-compose.yml STACK

or:

$ set -a && . .env && set +a
$ docker stack deploy -c docker-compose.yml STACK

set -a adds the export attribute to every variable that gets created. set +a restores the default behavior.

[moracabanas]

This is my aproach until I found this thread:

deploy.sh:

#!/bin/sh
export $(cat .env) > /dev/null 2>&1; 
docker stack deploy -c docker-compose.yml ${1:-STACK_NAME}

. deploy.sh <stack_name>

(stack_name optional if previously defined in .env)

[ivanduka]

There is a workaround if you don't want to remember custom commands to do variable substitution in docker stack deploy.
Add the following function to ~/.bashrc to the very end of the file and open bash again to enact those changes:

loadenvs() {
    local envfile="${1:-.env}"
    set -a && . $envfile && set +a
}

The script does the following:

1. Declare a function loadenvs
2. Declare a local variable envfile and assign the first argument to the function to it; if there is no argument, use '.env' instead
3. Temporarily set all declared functions as exported; run the envfile with the current shell; revert default exportability back to defaults

Then the usage is very simple:

1. If the env file is ".env" then run loadenvs && docker stack deploy...
2. If the env file name is different, specify it as an argument: loadenvs .custom_env && docker stack deploy...

Please note that if you are using sudo to run docker commands, you need to use -E option to preserve environmental variables in sudo session like that: loadenvs && sudo -E docker stack deploy...

[marlonklc]

I don't think the CLI is ever going to add .env support. The docker stack deploy command was originally built with Distributed Application Bundle files in mind (hence the --bundle-file option). DAB files were generated by Docker Compose with the docker-compose bundle command. .env and other Compose features would be handled by docker-compose, before being passed to docker stack deploy. Direct support for Compose files in docker stack deploy was always more of a compromise.

The original promise of DAB files now lives on in Docker App, which also does not process .env.

I have my developers use docker-compose config to pre-process Docker Compose projects before passing them to docker stack deploy. You can do this in one line with:

docker stack deploy -c <(docker-compose config) stack-name-here

That way, all Docker Compose features including .env processing are fully applied.

thank you so much, thats works perfect !

[quentinsf]

It's worth noting, BTW, that the new, non-hyphenated docker compose, while admirable in many ways, doesn't quite give the same output as docker-compose when run with the config command. (I noticed that variable substitution was slightly different, and the 'version' line was missing.)

So, even though I've switched to docker compose for most other things, I'm sticking with the hyphen for this particular case.

[x-yuri]

docker-compose relies on python-dotenv for interpreting .env files. And the latter... well, at least it doesn't parse as sh would.

On a side note, docker compose? I don't see such command in Docker 20.10.6. What is admirable about it?

[quentinsf]

On a side note, docker compose? I don't see such command in Docker 20.10.6. What is admirable about it?

In the docs, it's still listed as a 'technical preview' but some of the talks at DockerCon gave the impression it was now more mainstream than that. They were (confusingly) calling it Compose v2, but that's nothing to do with the version numbers in the docker-compose.yml file.

The basic idea was to rewrite docker-compose in Go and include it in the main CLI, partly for speed but partly so it always had access to the latest facilities in the engine/CLI.

There are still some differences, but I've been using it all day for the last couple of days on my Mac with no issues other than those I mentioned above.

More info in this talk, if it's accessible:
https://docker.events.cube365.net/dockercon-live/2021/content/Videos/WGpFvvKmhhwvY52va

Quentin

[sorenwacker]

And with that the env-file works? Unfortunately, the page does not really explain how to use it though. I still get

• error decoding 'Volumes[3]': invalid spec: :/appmedia/: empty section between colons

Indicating that it is either not working or not installed correctly.

Replacing the variables all over the compose file that were before niceley and uniquely handled by the .env file seems a bit inconvenient and it creates incompatibilities.

[Janevski]

BEWARE of the <(docker-compose -f CUSTOM-COMPOSE-FILENAME.yml config HACK!!! The output of docker-compose config and docker-compose -f myfile.yml config is NOT necessarily the same! The latter sometimes wraps env variables in extra quotes, so the output will be wrong!

Example:

environment:
  SERVER_URL: '"xxx"'

instead of

environment:
  SERVER_URL: xxx

Is this issue tracked somewhere already?

The translation hack is useless and error prone.
Yes it would work for a simple 'hello world' configuration, but not for a real docker swarm with replication etc, since it's meant for the compose, not the swarm.

[sincerite]

In docker compose V2 this approach won't work causing unsupported Compose file version: 1.0 error. (see docker/compose#8950)
This happens because now config command trimms version field.

It seems to me that the creators did it on purpose to even take away these hacks we had 🥇 😸

[Seabird22]

For people who would like to supply a custom .env file like .env.prod you can use the following: (Combining @kinghuang and @vovimayhem answers)

env $(cat .env.prod | grep ^[A-Z] | xargs) docker-compose config | docker stack deploy -c - stackname

To do this with a local registry, don't forget to initialize a local registry with a given port (f.e.5000) and tag your services in the docker-compose.yml file with:

image: 127.0.0.1:5000/<parent_folder>_<service_name>

Then build with docker-compose and push them to the registry as explained here: https://docs.docker.com/engine/swarm/stack-deploy/

[aaraney]

I wrote a compose-spec (compose v2) compliant docker-cli plugin replacement of docker stack deploy called deployx using compose-go that also does environment variable interpolation and handles .env files. My hope is this kind of functionality can be added to docker stack deploy, but in the mean time, I hope this plugin solves this issue for some people!

[webertim]

If anyone is looking for a powershell script to replace the environment variables in a docker-compose.yaml file, here your go:

# Define the path to env file as parameter as well as the path to docker-compose.yaml
param(
    [string]$envFilePath = ".env",
    [string]$dockerComposeFilePath = "docker-compose.yaml"
) 

# Check if the .env file exists
if (Test-Path $envFilePath) {
    # Read the contents of the .env file
    $envVariables = Get-Content $envFilePath

    # Loop through each line in the .env file
    foreach ($line in $envVariables) {
        # Split the line into variable name and value
        $parts = $line -split "="

        [System.Environment]::SetEnvironmentVariable($parts[0], ($parts[1..($parts.Length - 1)] -join "="), [System.EnvironmentVariableTarget]::Process)
    }
} else {
    Write-Host "The .env file does not exist."
}

# Input string containing references to environment variables
$inputString = Get-Content $dockerComposeFilePath -Raw

# Regular expression pattern to match environment variable references
$pattern = '\${([a-zA-Z_][a-zA-Z0-9_]*)}'

# Replace environment variable references with their values
$outputString = [regex]::Replace($inputString, $pattern, {
    param($match)

    $envValue = [System.Environment]::GetEnvironmentVariable($match.Groups[1].Value)
    if ($envValue -ne $null) {
        return $envValue
    } else {
        return $match.Value
    }
})

# Output the modified string
Write-Output $outputString

You can even pass the name of the .env and docker-compose.yaml file (with -envFilePath and -dockerComposeFilePath)

[msk010]

This is my approach
eval $(< .env) docker stack deploy --compose-file docker-compose.yml stack-name

[x-yuri]

@msk010 Doesn't work for me:

$ cat .env
AA_A=1
AA_B=2

$ eval $(< .env) set | grep AA

[JohnSundarraj]

Here is what we do, it works very well on Docker Swarm cluster.

1. Put all your environment variables in a .env file like below.

$ cat STG.env
export ENVIRONMENT=stg
export DOMAIN=stg.example.com
export NGINX_PORT=8080

2. Run below commands to create Docker Swarm stack.

$ source STG.env
$ envsubst < stack.yml | sudo docker stack deploy -c - stg

Hope it helps.

[rustamwin]

Here is another workaround:

docker compose -f stack-deploy.yml config | grep -v '^name' | docker stack deploy -c -