Document our BOSH release

[gcapizzi]

• Submodules dependencies in src/gopath
• Jobs:
  • how do we handle the containerd daemon?
  • what's netplugin-server?
• Anything interesting in our control scripts that we haven't mentioned yet?
• How do we implement BPM support?

Tasks

Beta Give feedback

1. Decide on BPM support
2. Document containerd daemon

[MarcPaquette]

Howdy @gcapizzi,

Would you like to set up some time to pair in our overlapping hours to go over what we should do to get this closed out?

Thanks,
@MarcPaquette

[MarcPaquette]

Hey @winkingturtle-vmw,
When you get some time I'd like to get this taken care of while we're working on our WG CI Export.
Thanks,
@MarcPaquette

[MarcPaquette]

Some work for this was done with #323

[MarcPaquette]

For Containerd's daemon:

We set up the config file here:

garden-runc-release/jobs/garden/spec

Line 22 in 861e755

config/containerd.toml.erb: config/containerd.toml

Containerd's config file is built here: https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/templates/config/containerd.toml.erb#L29C17-L51

Garden Start starts up the containerd process by calling the method start_containerd:

garden-runc-release/jobs/garden/templates/bin/garden_start.erb

Line 46 in 861e755

start_containerd

Which then kicks off the steps to invoke containerd with the previously generated config file:

garden-runc-release/jobs/garden/templates/bin/containerd_utils.erb

Lines 28 to 68 in 861e755

	start_containerd() {
	log "starting containerd"
	if pidof containerd; then
	echo "containerd already running"
	return
	fi
	# migrate the containerd state folder to /var/vcap/sys/run
	new_state_dir=/var/vcap/sys/run/containerd/state
	old_state_dir=/var/vcap/data/containerd/state
	if [ -d "$old_state_dir" ] && [ ! -d "$new_state_dir" ]; then
	mkdir -p $(dirname "$new_state_dir")
	mv "$old_state_dir" "$new_state_dir"
	fi
	containerd_config_filepath="$GARDEN_CONFIG_DIR/containerd.toml"
	exec_command="exec"
	<% if p("garden.experimental_rootless_mode") -%>
	maximus=$(/var/vcap/packages/garden-idmapper/bin/maximus)
	cp "$GARDEN_CONFIG_DIR/containerd.toml" "$GARDEN_ROOTLESS_CONFIG_DIR"
	chown "$maximus:$maximus" "$GARDEN_ROOTLESS_CONFIG_DIR/containerd.toml"
	containerd_config_filepath="$GARDEN_ROOTLESS_CONFIG_DIR/containerd.toml"
	exec_command="exec execas --uid $maximus --gid $maximus"
	<% end -%>
	log "running containerd"
	$exec_command /var/vcap/packages/containerd/bin/containerd -c "$containerd_config_filepath" \
	1>> "${LOG_DIR}/containerd.stdout.log" \
	2>> "${LOG_DIR}/containerd.stderr.log" \
	&
	echo "$!" > "$CONTAINERD_PIDFILE"
	address=$(grep containerd.sock $containerd_config_filepath | awk '{print $3}' | tr -d '"')
	while ! /var/vcap/packages/containerd/bin/ctr -a $address --connect-timeout 100ms c ls; do
	log "waiting for containerd to become available"
	sleep 0.1
	done
	}

Garden's start had previously been acomplished by the configuration of this BPM configuration file: https://github.com/cloudfoundry/garden-runc-release/blob/861e755aed81d437f84f6af709344537578dd6e9/jobs/garden/templates/config/bpm.yml.erb#L1C1-L8C22

[MarcPaquette]

BPM Support:

Implemented via a couple of flags:
https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/spec#L303-L305

https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/spec#L157-L159

Some additional Parameters that have caveats when enabling BPM:
https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/spec#L157-L159
https://github.com/cloudfoundry/garden-runc-release/blob/develop/jobs/garden/spec#L231-L233

garden-runc-release/jobs/garden/spec

Line 5 in 833f960

bin/bpm-pre-start.erb: bin/bpm-pre-start

which points to

garden-runc-release/jobs/garden/templates/config/bpm.yml.erb

Line 10 in 833f960

pre_start: /var/vcap/jobs/garden/bin/bpm-pre-start

garden-runc-release/jobs/garden/templates/bin/bpm-pre-start.erb

Line 12 in 833f960

<%- if_p('garden.additional_bpm_volumes') { |vols| if vols.include?('/var/vcap/data/rep/shared/garden') -%>

Where we can see where the additional volumes get mounted

garden-runc-release/jobs/garden/monit

Line 4 in 833f960

start program "/var/vcap/jobs/bpm/bin/bpm start garden"

Which runs (start):

garden-runc-release/jobs/garden/templates/bin/garden_start.erb

Line 85 in 833f960

<% if !p("bpm.enabled") %>

and stop

garden-runc-release/jobs/garden/templates/bin/garden_stop.erb

Line 7 in 833f960

<% if !p("bpm.enabled") %>

[MarcPaquette]

Starting to Look into Netplugin-Shim. Using this comment as a place to take notes.

https://github.com/cloudfoundry/garden-runc-release/wiki/Git-Repositories#netplugin-shim

From: cloudfoundry/cf-deployment@9068ed1
In order to run Garden in BPM containers Garden needs specific network
settings:

• Diego talks to Garden via a socket
• Garden talks to a networking server that runs outside the Garden
  container over a dedicated socket
• The networking server invokes the external networker

[MarcPaquette]

Netplugin-shim is now being removed from garden-runc-release, so the need to document it has evaporated. I'lll work on documenting everything else at this point.