fly-execute

Running local tasks while developing your CI pipeline in Concourse

The problem

When developing relatively complex pipelines for Concourse, tasks definitions and scripts are typically defined in individual files (e.g. YML and .sh), separate from the main pipeline definition YML file.

While this is a great structure for a CI pipeline in production, it creates a pain point for the developer of that same pipeline: for every code tweak made to a task's YML or script file, a git commit/push is required in order for the Concourse server to pick up the task's latest files from git.

The solution

Run and debug local copies of task files until they are fully functional and ready to be pushed to the git repository.

Concourse's fly execute command allows for the execution of local individual tasks along with local versions of artifacts (e.g. script files).

For example, for the pipeline files listed further below, while I'm still developing or debugging the task defined by sayhi.yml and its script sayhi.sh, I could use the following command to run the local task from within the my-repo directory:

fly execute -c ./tasks/sayhi.yml --input my-repo=.

Concourse will execute the task and, due to the provided --input option, it will use my current directory (.) as the required my-repo input for the task, making all files from that local directory available to the running container.

Pipeline files dir tree	Task definition file:  sayhi.yml	Task script:  sayhi.sh
my-repo |-- main |   +-- pipeline.yml |-- scripts |    |-- sayhi.sh |    +-- sayyo.sh  +-- tasks      |-- sayhi.yml      +-- sayyo.yml	--- platform: linux image_resource:   type: docker-image   source:     repository: ubuntu     tag: "latest" inputs: - name: my-repo run:   path: my-repo /scripts/sayhi.sh	#!/bin/bash set -xe echo "Hi there!"   find . env

Reusing input from builds on the Concourse server

Not only local directories can be used as inputs by the fly execute command. A resource input from an existing pipeline/job on the Concourse server can also be used as input for your local task run.

For example, suppose that my sayhi.yml task definition had a second required input for another git repository:

  inputs:
     - name: my-repo
     - name: other-git-repo

Instead of using a local copy of that repo for my local task run, I could add the --inputs-from parameter to my previous command:

    fly execute -c ./tasks/sayhi.yml --input my-repo=. --inputs-from myPipelineName/myJobName

While running the local task, Concourse will still use the local directory for the my-repo input as before, but it will also retrieve the inputs from the defined job and make them available for the local task run.     

Handling params entries in local task runs

If your task definition requires parameters from a params section in the YML, they can be provided to the local task run by using environment variables.

For example, let's say that sayhi.yml requires two parameters, USERNAME and CITY, to be used by sayhi.sh.

To make that work in a local task run, the following section should be added to the task YML file:

params:
  USERNAME: 
  CITY: 

Then, in your command line, define the corresponding environment variables with the values that you need for the local task run.

For example:     export USERNAME=John && export CITY=Raleigh

The fly execute command will search for environment variables that match the entries from the params section while executing a task and make them available for the task execution scripts.

In a scenario like that, my sayhi.sh script could be updated with the following command:

      echo "Hi $USERNAME, greetings from $CITY."

The fly execute command also provides options to exclude local files from inputs and also to capture the output of a local task run.

More details can be found on the fly execute documentation page.

Back to Concourse Pipelines Pro Tips