Skip to content

GitLab CI in bwCloud

Jump to bottom
Michael Kuron edited this page Aug 12, 2019 · 7 revisions

bwCloud

The University of Freiburg operates an OpenStack cloud, which we would like to use for additional runners. First, go to https://bwservices.uni-freiburg.de/, log in with your University of Stuttgart account (acXXXXXX), and sign up for the bwCloud SCOPE service. You'll get an email after a day or two saying that you were granted access. Then you can go to the same page again and set a password.

Next, ask go to https://bw-support.scc.kit.edu and open a support ticket to request creation of a new group project. @mkuron already did that and you can ask him to request that your account be added to the stuttgart_espresso project.

Once you have access, log into https://portal.bw-cloud.org/, which is the OpenStack management portal. Switch to the stuttgart_espresso project.

  • Under Compute, Key Pairs, I created an SSH key pair named gitlab, whose private key is stored on elk in /etc/gitlab-runner/bwcloud-acXXXXXX.pem.
  • Under Network, Security Groups, I created a firewall ruleset named gitlab-runner, which allows access to TCP port 2376 for 129.69.120.0/24.
  • Go to Identity, Projects and copy the Project ID (cceb6ed21ffc4b32a26169873a531ba8) for the group project.

GitLab Runner registration

On elk, run sudo gitlab-runner register --executor docker+machine. Tell it to register with https://gitlab.icp.uni-stuttgart.de and add the tags docker,linux,nofirewall,ptrace. Then edit /etc/gitlab-runner/config.toml until it looks like this:

concurrent = 100
check_interval = 0

[session_server]
  session_timeout = 1800
  listen_address = "0.0.0.0:8093"
  advertise_address = "elk.icp.uni-stuttgart.de:8093"

[[runners]]
  name = "bwCloud"
  url = "https://gitlab.icp.uni-stuttgart.de/"
  token = "AAAAAAAAAAAAAAAAAA"
  executor = "docker+machine"
  limit = 48
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "ubuntu"
    privileged = false
    cap_add = ["SYS_PTRACE"]
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache:/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.machine]
    IdleCount = 1
    MaxBuilds = 4
    MachineDriver = "openstack"
    MachineName = "gitlab-ci-%s"
    MachineOptions = [
      "openstack-username=acXXXXXX@uni-stuttgart.de",
      "openstack-domain-name=Default",
      "openstack-password=YYYYYYYYYYYYYY",
      "openstack-tenant-id=cceb6ed21ffc4b32a26169873a531ba8",
      "openstack-region=Mannheim",
      "openstack-auth-url=https://idm01.bw-cloud.org:5000/v3",
      "openstack-flavor-name=m1.medium",
      "openstack-image-name=Debian 10",
      "openstack-ssh-user=debian",
      "openstack-keypair-name=gitlab",
      "openstack-user-data-file=/etc/gitlab-runner/cloud-runner.sh",
      "openstack-private-key-file=/etc/gitlab-runner/bwcloud-acXXXXXX.pem",
      "openstack-sec-groups=default,gitlab-runner"
    ]
    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 0
    OffPeakIdleTime = 0
    OffPeakPeriods = [
      "* * 0-8,19-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]

Now put the following into /etc/gitlab-runner/cloud-runner.sh:

#!/bin/bash

which cloud-init >/dev/null
if [ "$?" != "0" ]; then
        echo "Only run this script via cloud-init"
        exit 1
fi

# create the cache directory and make it writable
mkdir /cache
chmod 777 /cache

# install the binfmt handlers for QEMU
apt-get update
apt-get install -y qemu-user-static

Using docker-machine manually

On elk, you can directly use docker-machine. You need to use sudo -H to get access to the instances created by gitlab-runner. If you want to create your own instances for testing, you can do that without sudo.

To view running instances created by gitlab-runner:

sudo -H docker-machine ls

To log into one of them:

sudo -H docker-machine ssh runner-ZZZZZZZZ-gitlab-ci-NNNNNNNNNN-HHHHHHHH

To manually create an instance (the arguments are the same as those given in the MachineOptions section above, i.e. openstack-username becomes --openstack-username.):

docker create --driver openstack --openstack-keypair-name=gitlab --openstack-private-key-file=./bwcloud.pem \
  --openstack-domain-name=Default --openstack-tenant-id=cceb6ed21ffc4b32a26169873a531ba8 \
  --openstack-username=acXXXXXX@uni-stuttgart.de --openstack-password=YYYYYYYYYY \
  --openstack-auth-url https://idm01.bw-cloud.org:5000/v3/ --openstack-flavor-name=m1.medium \
  --openstack-image-name="Debian 10" --openstack-region=Mannheim --openstack-ssh-user=debian \
  --openstack-sec-groups=default,gitlab-runner my_test_vm

To log into that instance:

docker-machine ssh my_test_vm

To delete your instance:

docker-machine rm -f my_test_vm

Shortcomings

docker-machine's OpenStack support is quite incompletes. This makes the configuration file less pretty in two ways, slows down builds by requiring unnecessary reboots and prevents the use of ccache.

The provided Linux images are also not perfect. A bug in the current Debian image delays the boot process unnecessarily, while a missing kernel module in the Ubuntu image breaks QEMU support for us.

Hexadecimal project ID in config file

It is also possible to specify openstack-tenant-name, but that uses a deprecated API that bwCloud no longer supports. Ticket: https://github.com/docker/machine/issues/3774.

Personal user account in config file

An acXXXXXX account name is used in the configuration file, which might become a problem once that person leaves the university. One could create a project token under Identity, Application Credentials, but docker-machine cannot currently use that. Ticket: https://github.com/docker/machine/issues/4684.

Root disk is only 12 GB

All instances have a 12 GB root disk, which only a few builds and Docker images fit onto. One could attach 40 GB temporary disks to the instance via

openstack server create --image "Debian 10" --flavor m1.nano --block-device vdb=empty:snapshot:40:true test1

after creating a blank snapshot via

openstack volume create --size 1 empty
openstack volume snapshot create --volume empty empty

but docker-machine doesn't support block device mappings. Once it does, one can modify cloud-runner.sh to format and mount that disk.

Current workaround: set MaxBuilds = 4 so machines are discarded after a few builds.

ccache is not persistent and docker images are always pulled from our GitLab server

Possible workaround: install an NFS server and a Docker registry proxy in a permanently-running instance. Beware that the NFS server would be writable to all bwCloud users because we can't simultaneously use a private and a public network. Ticket: https://github.com/docker/machine/issues/1444.

Leftover instances

Sometimes the GitLab runner doesn't properly destroy instances. They disappear from docker-machine, but remain running. This seems to sometimes happen when GitLab runner is restarted, but also when instance creation fails.

Clone this wiki locally