Databricks Jsonnet Guide

With over 1000 jsonnet files and templates, Databricks is to the best of our knowledge one of the larger users of Jsonnet. This guide draws from our experience coaching and working with engineers at Databricks.

Jsonnet is a language used most commonly to describe a finite number of complex, differentiated resources. For example, we may be describing services deployed within a Kubernetes cluster, differentiated by running in development versus production. As another example, we may be describing resources within a Cloud Provider, such as an Amazon RDS or Google Cloud SQL database, deployed across differnet regions.

Because we are most commonly describing a finite and somewhat fixed set of resources, it is useful to think of jsonnet as code which is executed at commit time (in the code repository sense) to materialize specific resources such as Kubernetes Deployment JSONs or AWS CloudFormation templates. In this way, the materialized templates are production code and source code diffing tools are unit tests, which means that we can be more certain in the correctness of jsonnet templates without writing specialized tests.

Jsonnet is a relatively constrained language, but we have found that sometimes the most obvious way to build and extend jsonnet also leads to significant headache down the line for people to understand and extend code. We have found that the following guidelines work well for us on projects with high velocity -- depending on the needs of your team, your mileage might vary.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Table of Contents

1. Document History

2. Syntactic Style

   • Autoformatting
   • Variable Declaration
   • Line Length
   • Spacing and Indentation
   • Blank Lines (Vertical Whitespace)

3. Defining and Using Abstractions

   • Defining Classes
   • Defining Methods
   • Using Classes
   • File Structure
   • Documentation Style

4. Best Practices

   • The Golden Pattern
   • Parameter Objects
   • Overriding and Inserting Fields

Document History

• 2017-06-17: Initial version.

Syntactic Style

Autoformatting

Use jsonnet fmt to format files. This will fix basic style errors.

Variable Declaration

• Variables should be named in camelCase style, and should have self-evident names.

  local serverPort = 1000;
  local clientPort = 2000;
  

• Prefer local to :: syntax for private/local variables. Unlike ::, variables defined with local cannot be overridden by children, nor accessed by other files.

  {
    // CORRECT
    local myVariable = 3,
    result: myVariable + 1,
  
    // INCORRECT
    myVariable:: 3,
    result: $.myVariable + 1,
  }
  

Line Length

• Limit lines to 100 characters.
• The only exceptions are import statements and URLs (although even for those, try to keep them under 100 chars).

Spacing and Indentation

• Put one space before and after operators

  local c = a + b;
  

• Put one space after commas.

  ["a", "b", "c"] // CORRECT
  
  ["a","b","c"] // INCORRECT
  

• Put one space after colons.

  {
    // CORRECT
    foo:: "bar",
    baz: "taz",
    { hello: "world" },
  
    // INCORRECT
    foo :: "bar",
    baz:"taz",
    { hello : "world" }, 
  

• Use 2-space indentation in general.

• Only method or class parameter declarations use 4-space indentation, to visually differentiate parameters from method body.

  // CORRECT
  local multiply(
      number1,
      number2) = {
    result: number1 * number 2
  }
  

• Do NOT use vertical alignment. They draw attention to the wrong parts of the code and make the aligned code harder to change in the future.

  // Don't align vertically
  local plus     = "+";
  local minus    = "-";
  local multiply = "*";
  
  // Do the following
  local plus = "+";
  local minus = "-";
  local multiply = "*";
  

Blank Lines (Vertical Whitespace)

• A single blank line appears:
  • Within method bodies, as needed to create logical groupings of statements.
  • Optionally before the first member or after the last member of a class or method.
• Use one or two blank line(s) to separate class definitions.
• Excessive number of blank lines is discouraged.

Defining and Using Abstractions

Defining Classes

• Rather than defining a concrete JSON file, it is often useful to define a template which takes some set of parameters before being materialized into JSON. We can liken named functions which take a set of parameters and result in a fixed scheme to "classes" in object-oriented languages, and so we will use that terminology.
• When defining a class, use the following syntax:

  local newAnimal(name, age) = {
    name: name,
    age: age,
  };
  {
    newAnimal:: newAnimal,
  }
  

• Returning a dictionary with a "newXXX" method (rather than just returning the constructor directly) allows exposing constants, static methods, or related class constructors from the same file. In other words, it allows extending this class in the future without refactoring all downstream consumers.
• When defining a class with both required and optional parameters, put required parameters first. Optional parameters should have a default, or null if a sentinel value is needed.

  local newAnimal(name, age, isCat = true) = { ... }
  

• Wrap parameter declarations by putting one per line with 2 extra spaces of indentation, to differentiate from the method body. Doing this is always acceptable, even if the definition would not wrap.

  local newAnimal(
      name,
      age,
      isCat = true) = { 
    name: name,
    ...
  }
  

Defining Methods

• Method definitions follow the same syntactic style as class definitions.
• Methods defined within a class should always be defined with ::, as they fail to render with :.
• Methods which return single values (rather than a dictionary) should use parentheses () to enclose their bodies if they are multi-line, identically to how braces would be used.

  {
    multiply:: function(number1, number2): (
      number1 * number 2
    ),
  }
  

Using Classes

• Import all dependencies at the top of the file and given them names related to the imported file itself. This makes it easy to see what other files you depend on as the file grows.

  // CORRECT
  local animalTemplate = import "animal.jsonnet.TEMPLATE";
  animalTemplate.newAnimal("Finnegan", 3)
  
  // AVOID
  (import "animal.jsonnet.TEMPLATE").newAnimal("Finnegan, 3)
  

• Prefer using named parameters, one per line, when constructing classes or invoking methods, especially when they wrap beyond one line:

  // PREFERRED
  animalTemplate.newAnimal(
    name = "Finnegan",
    age = 3,
  )
  
  // ACCEPTABLE, since it does not wrap
  animalTemplate.newAnimal("Finnegan", 3)
  

File Structure

• Jsonnet files which can be materialized with no further inputs should end with the ".jsonnet" suffix.
• Jsonnet files which requires parameters to be materialized or which are libraries should end with the ".jsonnet.TEMPLATE" suffix.
• Additional suffixes can be appended to file name to indicate the type of resource a Jsonnet file is describing, for clarification purpose:

  instance.cfn.jsonnet.TEMPLATE <-- A Cloudformation template
  manager-deploy.jjb.jsonnet <-- A Jenkins Pipeline Job template
  core-vnet.tf.jsonnet.TEMPLATE <-- A Terraform template
  

• Structuring libraries and imports is not a solved problem, so use your best judgement. In general if you have one common template and many individual instantiations, a workable pattern is:

  central-database/database.jsonnet.TEMPLATE <-- common template
  central-database/dev/database.jsonnet <-- dev instantiation which imports common template
  central-database/prod/database.jsonnet <-- prod instantiation which also imports common template
  

• Jsonnet files which are used to describe Kubernetes resources, e.g. deployment, service, should be put inside the package of the service or component itself, for better component isolation:

  myservice/deploy/myservice-dev.jsonnet <-- A Kubernetes deployment for 'dev' environment
  webapp/deploy/webapp-service.jsonnet.TEMPLATE <-- A Kubernetes service template
  

Documentation Style

• Use // for comments.
• Document parameters using a description, @param, and @returns similar to JavaDoc:

  // Multicellular, eukaryotic organism of the kingdom Animalia 
  // @param name Name by which this animal may be called.
  // @param age Number of years (rounded to nearest int) animal has been alive.
  local Animal(name, age) = { ... } 
  

• Always put documentation at the top of each jsonnet file or template to indicate its purpose.

Best Practices

The Golden Pattern

In most cases, you can define a single class which outputs the entire template you're looking for, and have a single concrete jsonnet file per actual resource to create. For example, you might define a common database template and a single dev jsonnet and a single prod jsonnet which imports from the common template.

In this case, a pattern like the following is most preferred:

// Common Kubernetes Deployment template for a specific web application (webapp.jsonnet.TEMPLATE)
local newWebApp(customerName, releaseTag) = {
  ... Kubernetes Deployment definition ...
  serviceName: customerName + "-webapp",
  dockerImage: "webapp:" + releaseTag,
  ... etc ...
};

{
  newWebApp:: newWebApp,
}


// A dev webapp deployment (in dev/ericl-webapp.jsonnet)
local webAppTemplate = import "../webapp.jsonnet.TEMPLATE";
webAppTemplate.newWebApp(
  customerName = "dev-test-1",
  releaseTag = "bleeding-edge",
)


// A production webapp deployment (in prod/foocorp-webapp.jsonnet)
local webAppTemplate = import "../webapp.jsonnet.TEMPLATE";
webAppTemplate.newWebApp(
  customerName = "foocorp",
  releaseTag = "2.42-rc1",
)

For a more complete example, see the examples and blog post.

Use this pattern as far as it will get you. Avoid implementing further abstractions and avoid default parameters for as long as possible. Keeping the number of abstractions low usually makes templates easier to understand. Avoiding default parameters means you have to explicitly choose a value in every situation (reducing corretness bugs).

On the flip side, new additions may require significant mechanial work due to repition. Templates do not have to be DRY (don't repeat yourself) because they are fully materialized at commit time, so correctness issues of repetitiveness are reduced and readability is more important. Use your best judgement when deciding when to build out a new abstraction to avoid repetition.

Parameter Objects

In situations where sets of parameters are shared between multiple templates or objects, define parameter objects which extract out the common set.

local newAwsVpcParams(region, accountId, virtualNetworkId, encryptionKeyId) = {
  ...
};


local newUsersDatabase(awsVpcParams, instanceSize, highAvailability) = {
  ...
}

local newFrontendVirtualMachine(awsVpcParams, instanceName) = {
  ...
} 

Overriding and Inserting Fields

Jsonnet has excellent support for supplementing existing JSON structures with new fields, for example:

local protoCat = animalTemplate.newAnimal(name = "Finnegan", age = 3);
// ... many lines later ...
local myCat = protoCat + {
  catYears: 28,
}

This pattern is very convenient for implementing arbitrary transformers on data strutcures, but it should be used with caution because it makes it hard to reason about how code within the parent class is executing and what fields are provided on "myCat".

This becomes especially confusing if you have multiple layers of such additions, or helper functions with transform properties within the input. In the rare cases that parameters are determined at various points within the code, prefer to make helper functions with construct and mutate parameter lists, which are ultimately passed to an Animal object.

For example:

// Better, but still to be avoided
local protoCat = animalTemplate.newAnimalParams(name = "Finnegan", age = 3);
// ... many lines later ...
local myCatParams = protoCat + {
  catYears: 28,
};
local myCat = animalTemplate.newAnimal(myCatParams)

This pattern ensures that inputs and outputs are fully determined by the code within Animal, rather than split between Animal and callers.