Replace per-project docker-compose.yml with a single file plus environment variables

[orangejulius]

If you look at the individual projects within this repository, they all contain 3 primary files:

• pelias.json
• .env
• docker-compose.yml

The first two are pretty project specific, but the third, docker-compose.yml is, for the most part, identical between projects.

The primary way we as pelias developers use the docker-compose.yml file, is to specify different docker image tags when setting up a build for testing.

With that in mind, what if we replaced the docker-compose.yml in each project with a single file that used environment variables to fill in anything we frequently wanted to modify, like the exact docker image tags?

A section in docker-compose.yml that looks like this currently:

  api:
    image: pelias/api:master
    container_name: pelias_api
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4000" ]
    ports: [ "4000:4000" ]
    volumes:
      - "./pelias.json:/code/pelias.json"

Could be changed to look like this, as a simple example:

  api:
    image: pelias/api:${PELIAS_API_TAG}
    container_name: pelias_api
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=${PELIAS_API_PORT}" ]
    ports: [ "${PELIAS_API_PORT}:${PELIAS_API_PORT}" ]
    volumes:
      - "./pelias.json:/code/pelias.json"

The $PELIAS_API_* variables could have defaults in .env, but would also be easy to change.

This would reduce duplication between the projects (we already have issues with minor differences creeping in, since we have so many projects now, and also make it easier (hopefully) to customize.

@missinglink, @Joxit, thoughts?

[missinglink]

Sounds like an interesting idea.

Seems that it wouldn't allow us to completely remove the docker-compose.yml file, since it's required but it would make it easier to diff for changes and make edits to it.

We could possibly configure the pelias command to use a standard copy located in the root of the project when one wasn't available in cwd, we'd just need to ensure that relative paths to things like ./pelias.json still functioned correctly.

The most common edit I make to the yaml file is to change a tag for something I'm testing, so I think this would make that easier and less error-prone.

I'm in two minds about it, I could also see it introducing another level of variables which could end up making the code more complex and harder to configure and maintain, depends how it's implemented and how it's documented.

[missinglink]

Some care would need to be taken when defining a convention for naming of the env vars, we'd need names to be fairly obvious, unambiguous and unchanging.

[Joxit]

Hum, what if the docker-compose.yml file is in the root of the project ? or in the project folder ?

I assume that nobody is going to use this project to set up two regions on the same machine.
Each region has the same hierarchy so we can create a docker-compose.yml like this one:

services:
  api:
    image: pelias/api:${PELIAS_API_TAG}
    container_name: pelias_api
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=${PELIAS_API_PORT}" ]
    ports: [ "${PELIAS_API_PORT}:${PELIAS_API_PORT}" ]
    volumes:
      - "./projects/${PELIAS_REGION}/pelias.json:/code/pelias.json"
  placeholder:
    image: pelias/placeholder:master
    container_name: pelias_placeholder
    user: "${DOCKER_USER}"
    restart: always
    environment: [ "PORT=4100" ]
    ports: [ "${PELIAS_PLACEHOLDER_PORT}:4100" ]
    volumes:
      - "./projects/${PELIAS_REGION}/pelias.json:/code/pelias.json"
      - "./projects/${PELIAS_REGION}/${DATA_DIR}:/data"
      - "./projects/${PELIAS_REGION}/blacklist/:/data/blacklist"

And the environment PELIAS_REGION will be one of the folders in projects ?

Changing the port of services via an environment variable (like placeholder, libpostal etc) is not a good idea because the endpoint is in the config.json file. We can change the exposed port, but not the internal one (like I did in placeholder example).

[missinglink]

We are currently testing this new workflow out in the Geocode Earth build environment.

We generally launch several full-planet builds a week, this change should make doing that much easier and less error-prone. Let's see how it goes and consider rolling it out for everyone once all the rough edges are ironed out.

[niels-heinemann]

Making the exposed ports configurable would be must-have enhancement for machines with a big docker orchestra and multiple elasticsearch containers for instance.