File-based configuration and composable build plans

[coffee-cup]

This RFC proposes an architectural change to Nixpacks that will allow

• File base configuration for full control of the build plan
• Support manually specifying the provider and skipping detection
• Support running multiple providers on a single app
• Consistent way of configuring Nixpacks across env vars, the CLI, and in providers
• Move re-usable behaviour into providers (e.g. Procfile support)
• Future: Support external static and dynamic provders

But before we get into the details, lets see some examples of nixpacks.toml files and how it could work.

Simple configuration of build

# Override the command for build and start
phases.build.cmds = ["yarn run client:build"]
start.cmd = "yarn run client:start"

More configuration with extending

[phase.setup]
# Append to the existing nix packages installed
nixPkgs = ["...", "cowsay"]

[phase.test]
dependsOn = ["install"]
cmds = ["yarn test"]

[phase.build]
# Ensure that the test phase runs before build 
dependsOn = ["...", "test"]

[start]
cmd = "cowsay starting && yarn start"

Run multiple providers

# Run both the node and rust providers and merge their build plans
providers = ["node", "rust"]

# Override configuration for Rusts build phase
[phase."rust:build"]
cmds = ["cargo build --release"]

[phase."node:build"]
dependsOn = ["rust:build"]

[start]
cmd = "yarn run start"

Creating an new phase

[phase.test]
dependsOn = ["install"]
nixPkgs = ["yarn test"]

[phase.build]
# Ensure that the build phase runs after the test phase
dependsOn = ["...", "test"]

screenshot-2022-09-06-19.22.00.mp4

Note: A lot of this PR is already implemented while I was experimenting with different configuration formats. It can easily change based on feedback on this proposal, but you can try it out with this branch

File Based Configuration

The only way to configure Nixpacks currently is through environment variables and CLI flags (which doesn't work on Railway). Like many other build tools, Nixpacks needs a way to be configured through a file so that it can be checked into version control and the changes tracked over time. I propose we support both a nixpacks.toml and nixpacks.json file. The contents of these configuration files is simply a serialized BuildPlan, the exact same plan that the providers generate based on the contents of the app. This allows every part of the build to be customized by the user and will ensure full customization moving forward since we don't have to maintain a separate configuration object.

Everything is a plan

At the end of the day, everything that can configure a Nixpacks build becomes a plan. This includes things like CLI flags and environment variables, as well as the result of running a provider. These plans are then merged together resulting in a final plan that contains everything needed to create an image from the user's source code.

Merging priority

Build plans are merged together with the following priority

• Provider-generated plans (lowest)
• File-based plan
• Environment variable plan
• CLI args plan

A non-null property in a plan will override the property in a lower priority plan. If a property is null in a high-priority plan but non-null in a lower-priority plan, the non-null value will be used. For example,

# File-based plan (nixpacks.toml)
phase.build.cmds = ["yarn run build"]
start.cmd = "yarn start"

# Environment variable plan
start.cmd = "./start.sh"

# Merged plan
phase.build.cmds = ["yarn run build"]
start.cmd = "./start.sh"

Property extending

Extending array properties is done with the "..." or "@auto" values. This allows you to easily reference the array values of the plan being merged into. For example

# Provider plan
phase.setup.nixPkgs = ["nodejs", "yarn"]
phase.build.cmds = ["yarn run build"]

# File-based plan (nixpacks.toml)
phase.setup.nixPkgs = ["...", "postgresql"]
phase.build.cmds = ["yarn run client:build"]

# Merged (frozen) plan
phase.build.cmds = ["nodejs", "yarn", "postgresql"] # Extended
phase.build.cmds = ["yarn run client:build"] # Not extended

This is powerful since the plan from the providers is always the lowest priority, which means you can always extend the automatic configuration, or override it completely.

Manually setting the provider

Providers can be manually specified by setting them in the nixpacks.toml file. A null value for the providers indicates that a single provider that passes the detect phase should be run (same as current behaviour). This is useful since you do not have to rely on detection to use a specific language if your app is setup in an unconventional way.

# Force running the node provider regardless if it passes detection
providers = ["node"]

Multiple Providers

This also opens the doors to specifying that multiple providers should run on the app.

# Run phases for both node and rust providers
providers = ["node", "rust"]

This is useful if your app uses multiple languages and you want an easy way to install everything (e.g. Rust server that uses Node for the frontend). To avoid naming conflicts with provider build phases, the phases should be prefixed with the provider that generated them (e.g. "build" becomes "node:build").

Note: After some initial experimentation with multiple providers, a few providers will need to be updated to handle when certain language files are missing. There will also need to be some optimizations when installing Nix/Apt packages. But these problems are all easily fixable.

Reusable Providers

Since build plans can be composed together, all re-usable behaviour can be packaged into a provider. This includes things like Procfile support and Postgres/MySQL installation.

Provider Configuration

Many of the providers have some sort of configuration that the user can provide. This includes things like versions of tools to install or directories to serve. These will be able to be specified in the nixpacks.toml file under nested under config.

providers = ["node", "staticfile"]

[config.node]
version = "16"

[config.staticfile]
root = "dist"

These will be deserialized into structs defined by each provider and passed into get_build_phase(app, env, config) .

Nixpacks Freeze

There may be a case where you want to eject from the automatic Railway providers. You can accomplish this by setting providers = [] which means that no Railway providers will run and only the plan from the nixpacks.toml file (or environment variables) will be used. There will also exist a nixpacks freeze command that will generate a build plan off your app and freeze it, which can be saved to disk in a nixpacks.toml file. When building with this file, the same Nix archive, packages, and commands will be used every time. This will allow you full control over your build with no magic.

Container building

Running Nixpacks with no providers allows you to do some interesting things since the nixpacks.toml file is basically a DSL on top of building Dockerfiles, but in a shortened and much more readable way. Consider

providers = []
baseImage = "debian:bullseye-slim"

[phases.setup]
aptPkgs = ["cowsay"]

[start]
cmd = "cowsay started"

This creates an image that doesn't use Nix or the Nixpacks providers at all. This use case will need some optimizations, but it gives a look into the future of Nixpacks and file base configuration.

External static and dynamic providers

Being able to manually specify providers to run opens the door up for external providers.

providers = ["github.com/user/provider", "node"]

Static providers

Static providers simply provide a build plan in either toml or json format. They do not have access to the user app or environment. This can be useful for things like postgres/mysql support or installing a combination certain Debian pacakges (e.g. font/emoji support, c libraries).

Supporting static providers is possible with the architecture proposed above and will open up Nixpacks to a re-usable build configuration marketplace.

Dynamic providers

Dynamic providers can analyze the user code and look at the environment to generate a build plan (like our providers). This has many security risk involved since you will be running untrusted code so needs to be handled with care. This is much more longer looking and will require interfacing with a binary or compiling code on the fly.

[josegonzalez]

I just came here to see if anything progressed from #120 (I'm adding nixpacks support to Dokku right now). This is great to see, and I'm excited for where this is headed, as I didn't want to add a bunch of knobs that might be overriden by a file-based config.

[josegonzalez]

Question: How does a user specify which nixpacks.toml file to use? Is it automatically picked up, or must it be specified on the cli?

[coffee-cup]

I think it should be automatically picked up if it exists (either nixpacks.toml or nixpacks.json). But we can also have a flag so the user can manually specify the config file to use.

[Milo123459]

Personally, I think [phase."node:build"] looks a bit clanky and should be replaced with [phase.node.build]. It seems a bit weird with the colons.

[coffee-cup]

I mostly agree and the prefix used is configurable. The quotes are just needed because of a limitation of the toml format (they are not needed in JSON). This also only affects running multiple providers on a single app, which won't be implemented right away.

However, I don't think it should be [phases.node.build] since that fundamentally changes the structure of what a phase is. Instead of being a well-defined list of configurations for the build, it becomes a loosly structure object that can contain any keys and makes serialization/deserialization (by Nixpacks itself and other tools), much harder.

[coffee-cup]

The prefix is to prevent phases defined by multiple providers from conflicting with each other (and maybe allow running them in parallel someday). If both the node and python providers generate setup, install, and build phases, they should be separated. But fundamentally the plans are the same data structure