How to dynamically switch included config files?

[gabrielsadakatw]

Is there a way to use a config variable to define which config file to include per run?

For example:

config_variables:
  target-db-config-file:
    description: Batect config file for target db
    default: examples/db/batect-target-db.yml
include:
  - <{target-db-config-file}

That way when executing a task we can pass in the path to the file to be included e.g. ./batect --config-var target-db-config-file=examples/db/batect-target-db.yml run.

Trying that resulted in the following error:

(at include[0] on line 23, column 5): Included file '<{source-db-config-file}' (resolved to .../...{source-db-config-file}') does not exist.

[charleskorn]

Hey @gabrielsadakatw, this isn't something that's currently supported.

However, there might be another solution... could you explain a little more about your situation? What are you trying to do?

Not sure if this is relevant for your situation: it is possible to switch between different sets of config variable values with the --config-vars-file option.

[chadlwilson]

Gabriel is asking in the context of https://github.com/ThoughtWorks-SEA/recce/blob/1560d8ee2f5f5e4ea73a44b51651d39abddb5066/batect.yml I believe.

What is (loosely) being attempted there, is a way to start and run against different sets of database containers depending on some kind of config var or env var type of flag - currently using includes (an example of one of the includes). The intent was the be able to define the containers with the same logical name (e.g source-db, target-db) and unify the inputs (say usernames, ports to use/expose) but then toggle between them somehow without creating n batect tasks for every single combination.

The alternative I contemplated to something like "conditional includes" (but also don't think is possible right now), would be to make the dependencies of a container or task dynamic in some way, also driven from config.

If looking for a non docker-compose solution, the alternative would perhaps be to orchestrate the whole thing with something like testcontainers with some kotlin scripting, rather than trying to use batect for this; possibly I was trying to push it further than it was intended to go 😉

[charleskorn]

You might be able to achieve what you're after by inverting the includes...

If there's only one database involved, something like this might work: instead of having a root batect.yml that conditionally includes different config files for different databases, you could have a series of config files (eg. database-1.yml, database-2.yml), each of which defines the database container and includes a common shared.yml which defines the rest of the application.

You could then switch between the different databases with the --config-file CLI option, for example: ./batect --config-file database-1.yml the-task.

However, if you want to be able to combine different pairs of databases, this won't quite work unless you create a config file for each pair of source and destination databases. For example, you could create a database-1-to-2.yml config file, which includes shared.yml, database-1-source.yml and database-2-target.yml. This might be manageable for a small number of combinations, but for a large number I imagine this would be unmanagable.

Hope that helps!

[chadlwilson]

Oh wow, I didn't know there was a --config-file option. Actually I realise now after chatting to @gabrielsadakatw and yourself that I know little about the CLI flags other than --clean-cache and --config-var so I failed to explore that adequately to find other easy ways to alter the behaviour. Inverting the includes dependencies as you suggest may indeed work, or at least be "better" than currently.

Let us have a think about that and give it a go at some point. In the context of the way recce "scenarios" currently work, I think it's probably OK to have a config file for each "pair" of example source/target databases based on where the tool currently is, since typically the scripting/SQL to set up some example data on each DB varies.

Much appreciate the suggestion!