Tile Author Guide

Kiln provides utilities and conventions for maintaining the source to build and release ".pivotal" files.

The shell script examples use kiln 0.85.0, yq 4.34.1, and curl 7.88.1.

Some parts of this page are also documented in README.md. This guide intends to be more opinionated while the README.me is more general.

Authoring Product Template Parts
Managing BOSH Release Tarballs
- Kilnfile BOSH Release Specification and Locking
  - Sources
    - BOSH.io
    - GitHub releases <- Please use this for BOSH Release tarballs
    - Build Artifactory <- Please use this for compiled BOSH Release tarballs
    - AWS S3
    - Local Files
- BOSH release compilation
Stemcell Version Management
Tile release note Generation
TanzuNet Release Publication
Importing Go Source Code

Helpful external links

Metadata Testing Example

Again, see hello-tile (non-VMware employees) or the TAS repo (VMware employees) for tile source code that kiln "just works" with.

Bootstrapping a tile repository

The following code bootstraps Product template parts from an empty directory working tile; it creates conventional defaults to start/standardize your tile authorship journey with Kiln. It does not generate source for a working tile.

# Make directories and keep them around
mkdir -p bosh_variables forms instance_groups jobs migrations properties releases runtime_configs
touch {bosh_variables,forms,instance_groups,jobs,migrations,properties,releases,runtime_configs}/.gitkeep

# Add a tile image
curl -L -o icon.png "https://github.com/crhntr/hello-tile/blob/main/icon.png?raw=true"

# Add a version file (this is the default location Kiln reads from)
echo '0.1.0' > 'version'

# Create a "base.yml" with some minimal fields
# See documentation here: `https://docs.vmware.com/en/Tile-Developer-Guide/3.0/tile-dev-guide/property-template-references.html`
cat << EOF > base.yml
name: my-tile-name
label: ""
description: ""
icon_image: $( icon )
metadata_version: "3.0.0"
minimum_version_for_upgrade: 0.0.0
product_version: $( version )
provides_product_versions:
  - name: hello
    version: $( version )
rank: 1
serial: false
releases: []
stemcell_criteria: []
job_types: []
runtime_configs: []
property_blueprints: []
form_types: []
EOF

PPE TODO

Create a kiln init command that actually makes a working tile.

Authoring Product Template Parts

A tile is a zip file with a ".pivotal" suffix. The zip file must have a YAML file in the metadata subdirectory of the tile. The file specification is documented in the Property and Template References page in the Ops Manager Tile Developer Guide.

While you can write your entire Product Template in a single file, breaking up metadata parts into different yaml files and directories makes it easier to find configuration. Kiln expects certain directories to contain Product Template (metadata) parts.

./forms should contain YAML files that have form_types
./instance_groups should contain YAML files that have job_types
./jobs should contain YAML files that have job templates and may contain a job mainfest
./migrations should contain Javascript files that are migrations (TODO add link to specification)
./properties should contain YAML files that have property-blueprints
./releases should be an empty directory (maybe containing .gitkeep) and will be where Kiln writes BOSH Release Tarballs
./runtime_configs should contain YAML files that have runtime configs

Also add the following files

base.yml should be the entrypoint for the manifest template.
version should contain the tile version
icon.png should be the tile image

While you can use other directory and filenames and pas those to Kiln, you would have a better experience if you follow the above naming conventions.

Adding product template parts to your Metadata

Product Template baking functions

Each product template part has a "name" field. When you kiln bake, the contents of base.yml go through a few templating steps. One of those steps exposes the following functions. These functions will read, interpolate, and format metadata parts for use in your tile's metadata.yml.

You can reference the part in base.yml by using the following template functions:

bosh_variable reads, interpolates, and formats a product template part from ./bosh_variables
form reads, interpolates, and formats a product template part from ./forms
property reads, interpolates, and formats a product template part from ./properties
release reads, interpolates, and formats a BOSH release data from either the ./releases directory or the Kilnfile.lock
stemcell reads, interpolates, and formats a Stemcell Criteria data from the Kilnfile.lock
version returns the contents of the ./version file
variable finds the named variables from either a --variables-file or --variable
icon returns the base64 encoded content of icon.png
instance_group reads, interpolates, and formats a product template part from ./instance_groups
job reads, interpolates, and formats a product template part from ./jobs
runtime_config reads, interpolates, and formats a product template part from ./runtime_configs

Other functions:

tile TODO function documentation
select TODO function documentation
regexReplaceAll TODO function documentation

See the crhntr/hello-tile/base.yml for some use of the above functions. The property definition in crhntr/hello-tile/properties/hello.yml in referenced in base.yml using $( property "port" ). Most other product template part functions behave similarly.

Managing BOSH Release Tarballs

kiln fetch downloads BOSH Release Tarballs from any of the following "sources" and puts them in a "./releases" directory.

Before Kiln can help you manage the BOSH Releases you put in the tile, you need to upload BOSH Release tarballs to a place accessable to Kiln.

See the sources below to decide which is right for your release.

Unlike Tile Generator. Kiln does not create releases. The Kiln way is to

have BOSH releases each have their own source control
have CI build and upload (finalized) BOSH release tarballs somewhere
have a tile source repository
- in the tile source repository specify how Kiln can get the BOSH release tarballs

While the following examples start from empty directories and mostly use S3 and BOSH.io. There are similar simple scripts for a small test tile illustrating similar usage patterns to the follwoing example. See hello-tile. Hello Tile consumes a single custom BOSH Release, hello-release, from a GitHub release. It does not upload the release to TanzuNet but adds the built tile to a GitHub Release.

EXAMPLE Using Kiln to Download BOSH Release Tarballs

This starts from an empty directory and downloads the latest BPM release from bosh.io. Note, the Kilnfile and Kilnfile.lock (unfortunately/frustratingly) must be created manually.

# Create and go into an empty directory
mkdir -p /tmp/try-kiln-fetch
cd !$
mkdir -p releases
touch releases/.gitkeep # not required but a good idea

# Hack a Kilnfile and Kilnfile lock
echo '{"release_sources": [{type: bosh.io}], "releases": [{"name": "bpm"}]}' | yq --prettyPrint . > Kilnfile
yq '{"releases": [. | {"name": "bpm", "version": .version, "sha1": .sha, "remote_source": "bosh.io", "remote_path": .remote_path}]}' <(kiln find-release-version --release=bpm) | yq --prettyPrint . > Kilnfile.lock

# Call Kiln fetch
kiln fetch

# See the fetched release
stat releases/bpm*.tgz

The files should look something like these

# Expected Kilnfile
release_sources:
  - type: "bosh.io"
releases:
  - name: bpm

# Expected Kilnfile.lock
releases:
  - name: bpm
    version: "1.2.3"
    sha1: "ad12bb4e1d2c0b94ef679670a99adaff920850d0"
    remote_source: bosh.io
    remote_path: "https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.2.3"

PPE TODO

The YQ expressions are a hack to get this to work from an empty directory. We need to improve this process. Kiln fetch was built around an existing "assets.lock"; the developer experience for starting from an empty directory is not polished.

Ordering

Kiln searches each BOSH Release tarball source in order when you call kiln find-release-version; The tarball lock printed if from the first release source that has a matching lock.

You should put your publishable release tarball source (where you cache compiled release tarballs first).
Then you should put your closed source release tarball source.
Then you should put open source places like BOSH.io or open source GitHub.

EXAMPLE Using Kiln to update BOSH Release Tarballs

Updating a release in a lock file requires two kiln commands.

Please follow the "Kiln Fetch Example" before following this one.

# (optional) Add a version constraint to the Kilnfile.
# This shows how Kiln will select a version that matches a constrint.
yq '(.releases[] | select(.name == "bpm")) |= .version = "~1.1"' Kilnfile

# Find a new BOSH Release Tarball version (on bosh.io)
export NEW_RELEASE_VERSION="$(kiln find-release-version --release=bpm | yq '.version')"
echo "Kiln found: bpm/${NEW_RELEASE_VERSION}"

# Update the Kilnfile.lock with the new version
kiln update-release --version="${NEW_RELEASE_VERSION}" --name="bpm"

The syntax for BOSH Release Tarball version constraints is Masterminds/semver. Other parts of the Cloud Foundry ecosystem use blang/semver. If you get unexpected results, this difference may be the cause. For simple version constraints they are similar enough.

kiln update-release ignores the content of Kilnfile. This can cause kiln validate to fail when a version passed to kiln update-release does not match the constraint in the Kilnfile. This behavior may/should change.

PPE TODO

This developer experience needs work IMO.

The release name flag difference is awkward. In find-release-version, the flag is --release; In kiln update-release, the flag is --name.

Maybe this should be one command and an optional flag

kiln update-release →kiln update-bosh-release bpm
kiln find-release-version → kiln update-bosh-release --dry-run bpm

Release Sources

While different credentials per release source element are currently supported. I would recommend one set of credentials per release source type.

In the Kilnfile.lock, BOSH release tarballs lock elements have a few fields.

name: The BOSH release name
version: The BOSH release version
sha1: The sha1 sum of the BOSH release tarball
remote_source: The identifier of the BOSH Release tarball source specified in the Kilnfile where the tarball is stored
remote_path: A source specific string to identify where the tarball is. This may be a URL.

Kilnfile Secrets

A Kilnfile (currently) specifies all the strings required to configure a BOSH Release tarball source. This includes secrets. While you can just add the secrets to the Kilnfile, don't. The Kilnfiles go through an initial templating step before being parsed. Please don't use this for anything but secret injection. Most kiln commands recieve a --variable or a --variables-file flag.

To use the --variable flag run something like this:

kiln fetch --variable=fruit=banana

In your Kilnfile use the fruit variable like this.

release_source:
  - some_field: $(variable "banana")
  - some_field: "$(variable "banana")"
  - some_field: '$(variable "banana")'

Explicit Release Source ID

Please set an explicit ID for each release source. Kiln has fall-back behavior to use other fields to identify a release source (like bucket for S3 or owner for GitHub...) but this fall-back behavior can be hard to follow. Just set id on all of your release sources and make mapping releases in the Kilnfile.lock to the release source in Kilnfile easier to follow.

Path Templates

While some BOSH release tarball sources use URLs as the remote_path in their release locks, others (S3 and Artifactory) rely on a path template in their configuration.

The path_template is uses text/template and is passed a value with the following type.

package cargo

// PathTemplateData is passed to template.Execute along with the parsed Release Source path template.
// this type is not the real one in the source.
type PathTemplateData struct{
    // Name is set to the BOSH Release name
    Name string
    
    // Version is set to the BOSH Release version
    Version string

    // Name is set to the Kilnfile.lock StemcellCriteria OS value
    StemcellOS string

    // Name is set to the Kilnfile.lock StemcellCriteria Version value
    StemcellVersion string
}

Here are some example path templates (the rest of the release source config has been omitted).

release_sources:
  - path_template: "bla-bla/{{.Name}}/{{.Version}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"
  - path_template: "bla-bla/{{.Name}}/{{.Version}}/{{.Name}}-{{.Version}}.tgz"
  - path_template: "bla-bla/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"

Avoid using other Go text/template actions like {{if pipeline}} and the like.

BOSH.io

Kiln can only download releases from BOSH.io and can not upload BOSH Releases to BOSH.io.

This release source has minimal configuration. Just add it to your release_sources and you can get releases from BOSH.io.

release_sources:
  - type: bosh.io
    id: community # (optional) the default ID for this type is the constant string "bosh.io"

The value of remote_path in the BOSH release tarball lock is a URL.

GitHub Releases

Kiln can only download releases from GitHub Releases and can not upload BOSH Releases to Github.

To download BOSH Release Tarballs from GitHub Releases, add the following

release_sources:
  - type: "github"
    id: crhntr # (optional) the default ID in this case is the value of org
    org: "crhntr"
    github_token: $(variable "github_token")

github_token is always required even for public repositories because we make API requests

You will need one entry per organization. Some examples are: "pivotal", "cloudfoundry", "pivotal-cf", or your personal/company GitHub username.

You will need to add the following flag to most commands:

# Optional helper
export GITHUB_TOKEN="$(gh auth status --show-token 2>&1 | grep 'Token:' | awk '{print $NF}')"

# Example Kiln variable flag
kiln fetch --variable="github_token=${GITHUB_TOKEN}"

The value of remote_path in the BOSH release tarball lock is a URL.

Build Artifactory

Kiln can fetch and upload releases to/from Build Artifactory.

The release source specification should look like this:

release_sources:
  - type: "artifactory"
    id: "official-storage"  # (optional) the default ID for this type is the constant string value "artifactory"
    repo: "some-repository"
    username: $(variable "artifactory_username")
    password: $(variable "artifactory_password")
    path_template: "some-path-template/{{.Name}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"

Notes:

repo is not a GitHub repository. It refers to an Artifactory repository.
username and password values can be non-empty arbitrary values for read-only access

Please see Path Templates. The value of remote_path in the BOSH release tarball lock is part of the path needed to construct a URL to download the release.

AWS S3

Kiln can fetch and upload releases to/from AWS S3.

release_sources:
  - type: "artifactory"
    bucket: "some-bucket"
    id: "legacy-storage"  # (optional) the default ID for this type is the value of bucket
    region: "some-region"
    access_key_id: $(variable "access_key_id")
    secret_access_key: $(variable "secret_access_key")
    path_template: "some-path-template/{{.Name}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"

Please see Path Templates. The value of remote_path in the BOSH release tarball lock is part of the path needed to make the S3 object name.

Local tarballs

kiln bake adds the BOSH release tarballs in the releases directory to the tile reguardless of if they match the Kilnfile.lock. Building a tile with arbitrary releases in the tarball is not secure; this behavior should only be used for development not for building production tiles.

Default credentials file

You can add a default credentials file to ~/.kiln/credentials.yml so you don't need to pass variables flags everywhere. Don't do this with production creds but if you have credentials you can safely write to your disk, consider using this functionality. The file can look like this

# GitHub release sources credentials. Github Release should contain BOSH tarballs
github_token: some-token

# S3 release BOSH release tarball source credentials
aws_secret_access_key: some-key
aws_access_key_id: some-id

# Artifactory BOSH release tarball release source credentials
artifactory_username: some-username
artifactory_password: some-password

BOSH Release Compilation

The following code will allow you to export/cache compiled BOSH Release Tarballs to either S3 or Artifactory after the tile has been deployed. Before kiln cache-compiled-releases exports the BOSH Release Tarballs it checks the specified source to ensure a tarball with a name containing the stemcell version does not exist.

# create a tile with the releases you want compiled
kiln bake

# Add an S3 (or Artifactory) release source to your Kilnfile
yq -i '.release_sources = [{"type": "s3", "id": "my_compiled_release_bucket", bucket": "some_bucket", "publishable": true, "access_key_id": "some_id", "secret_access_key": "some_id"}] + .release_sources' Kilnfile

# claim a bosh director and configure the environment
smith claim -p us_4_0
eval "$(smith om)"
eval "$(smith bosh)"
export OM_PRIVATE_KEY="$(cat $(echo "${BOSH_ALL_PROXY}" | awk -F= '{print $2}'))"

# deploy your Product (the commands should look something like this)
om upload-product --product=tile.pivotal
om configure-product --config=simple_config.yml
om apply-changes --product-name=my-tile-name

# Download Compiled BOSH Releases from the BOSH Director and Upload them to the S3 Bucket or Artifactory
kiln cache-compiled-releases --upload-target-id=my_compiled_release_bucket --name=hello

# Commit and push the changes to Kilnfile.lock
git add -p Kilnfile.lock
git commit -m "compile BOSH Releases with $(yq '.stemcell_criteria.os' Kilnfile.lock)/$(yq '.stemcell_criteria.version' Kilnfile.lock)"
git push origin HEAD

Stemcell Version Management

kiln find-stemcell-version and kiln update-stemcell

Find the latest stemcell releases on TanzuNet. They behave similarly to the bosh release commands above.

If I remember right, the find-stemcell-version command has a bug where the stemcell criteria version in the Kilnfile is not respected and the result of the command is always the latest version.

Tile Release Note Generation

If you use the GitHub release source for your BOSH release tarballs, you can generate release notes for your tile. The notes will contain release notes for each BOSH release.

This feature requires you to have a previous tile release that used a Kilnfile.lock to specify the BOSH releases packaged. You pass two references

the Git Reference or SHA of the commit of the tile repo source used to generate the previously published tile
Git Reference or SHA of the commit of the tile repo source used to generate the next tile

While you can override the template and the regular expression used to make these notes. They are quite hard to craft. I recommend you use the defaults.

kiln release-notes --update-docs=path-to-release-notes-file/notes.md "${PREVIOUS_RELEASE_SHA}" "${NEXT_RELEASE_SHA}"

If you omit --update-docs the notes will be written to standard out.

TanzuNet Release Publication

kiln publish does not in-fact publish a tile. It changes some of the configuration on a previously created TanzuNet release. While we use it for TAS, it is not ready/intended to be used by other tiles quite yet.

Importing Go Source Code .

Note the Kiln repository is pre-1.0.0. While we try to maintain backwards compatablility with the commands. The package API is subject to change without notice.

See the Hello Tile manifest test to see how to use this in tests. Follow the conventions you see in hello-tile, and you should be able to run kiln test. The github.com/pivotal-cf/kiln/pkg/proofing/upgrade package can help you detect changes that would require foundation provider/operator) intervention.

go get github.com/pivotal-cf/kiln/pkg/cargo

package main

import (
	"log"
	"fmt"
	
	"gopkg.in/yaml.v3"

	"github.com/pivotal-cf/kiln/pkg/cargo"
	"github.com/pivotal-cf/kiln/pkg/tile"
	"github.com/pivotal-cf/kiln/pkg/proofing"
)

func main() {
	// log release versions
	lock, err := cargo.ReadKilnfileLock("Kilnfile.lock")
	if err != nil {
		log.Fatal(err)
	}
	for _, releaseLock := range lock.Releases {
		fmt.Println(releaseLock.Name, releaseLock.Version)
	}

	// print name and rank from tile.pivotal
	manifestBytes, err := tile.ReadMetadataFromFile("tile.pivotal")
	if err != nil {
		log.Fatal(err)
	}
	var productTemplate proofing.ProductTemplate
	err = yaml.Unmarshal(manifestBytes, productTemplate)
	if err != nil {
		log.Fatal(err)
    }
	fmt.Println(productTemplate.Name, productTemplate.Rank)
}

Tile Build Packs

There is an internal VMware intiative to build tiles using TAP and buildpacks. The Kiln Buildpack can take tile source code and create a tile. For it to work, you need to have your BOSH Release Tarballs fetch-able by Kiln (and only using GitHub or BOSH.io release sources) and it is nice if your bake command does not require too many flags (see Tile Source Conventions).

The private repository kiln buildpack has the Paketo buildpack source. You can run the acceptance tests with a TILE_DIRECTORY environment variable set to your tile source to see if your tile will build with the buildpack.

mkdir -p /tmp/try-kiln-buildpack
cd !$

set -e

# Clone the Buildpack source
git clone git@github.com:pivotal/kiln-buildpack
cd kiln-buildpack

# Check the path to your tile directory
stat ${HOME}/workspace/my-tile-source/Kilnfile.lock

# WARNING this does a git clean. This is important to simulate building from source.
# If you have releases fetched already, you won't get an acurate test. 
cd stat ${HOME}/workspace/my-tile-source && git clean -ffd && cd -

# Run the acceptance test against your tile source
TILE_DIRECTORY="${HOME}/workspace/my-tile-source" go test -v --tags=acceptance .

stat /tmp/try-kiln-buildpack/kiln-buildpack/tile.pivotal

The buildpack is intended to use on TAP. It is still in early development.

Other Tips

.gitignore

The following are some useful .gitignore contents.

releases/*.tgz
*.pivotal

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TILE_AUTHOR_GUIDE.md

TILE_AUTHOR_GUIDE.md

Tile Author Guide

Table of Contents

Helpful external links

Bootstrapping a tile repository

Authoring Product Template Parts

Adding product template parts to your Metadata

Product Template baking functions

Managing BOSH Release Tarballs

EXAMPLE Using Kiln to Download BOSH Release Tarballs

Ordering

EXAMPLE Using Kiln to update BOSH Release Tarballs

Release Sources

Kilnfile Secrets

Explicit Release Source ID

Path Templates

BOSH.io

GitHub Releases

Build Artifactory

AWS S3

Local tarballs

Default credentials file

BOSH Release Compilation

Stemcell Version Management

Tile Release Note Generation

TanzuNet Release Publication

Importing Go Source Code .

Tile Build Packs

Other Tips

.gitignore

Files

TILE_AUTHOR_GUIDE.md

Latest commit

History

TILE_AUTHOR_GUIDE.md

File metadata and controls

Tile Author Guide

Table of Contents

Helpful external links

Bootstrapping a tile repository

Authoring Product Template Parts

Adding product template parts to your Metadata

Product Template baking functions

Managing BOSH Release Tarballs

EXAMPLE Using Kiln to Download BOSH Release Tarballs

Ordering

EXAMPLE Using Kiln to update BOSH Release Tarballs

Release Sources

Kilnfile Secrets

Explicit Release Source ID

Path Templates

BOSH.io

GitHub Releases

Build Artifactory

AWS S3

Local tarballs

Default credentials file

BOSH Release Compilation

Stemcell Version Management

Tile Release Note Generation

TanzuNet Release Publication

Importing Go Source Code .

Tile Build Packs

Other Tips

.gitignore