gitlab-ci helper scripts for OpenEmbedded builds

This project contains gitlab-ci config snippets and associated script to help in building an OpenEmbedded-based project.

This includes:

• support for building for several platforms

• support for building images or SDK

• publication of package feeds

• publication of shared-state for reuse by developpers

• offload of build artifacts through rsync servers, as gitlab does not support large-enough artifacts

• support for platform-specific branches using a branch prefix (eg. board1/master)

• persistence of downloads and shared-state directories

• different downloads directories between major poky versions

• different shared-state directories when you decide it (typically on toolchain change)

• cascading inheritance of shared-state for use in dev branches to limit cache polution and inflation (eg. BUILD_GEN: 1.2+next+mytest)

• (optional) protection from shared-state polution by forcing non-official branches to use copy-on-write shared-state

• optional usage of buildhistory repo and PR server

• buildhistory stored in seprate branches for each platform and each source branch

• archiving of interesting task logs as build artifacts (filtering out setscene and rm_work), even when bitbake ultimately fails

• archiving of cooker log as build artifact, as the log of a full builds can exceed gitlab’s job-log size limit

assumptions on project setup

• hosted by gitlab 12.0 or higher (for multiple extends)

• this repository is included in your project repo as /ci (eg. using git submodule or subtree)

• yocto project’s Poky repository is available in your project repo as /poky

• all other layers are available under your project repo after git checkout and submodule update --init

• if specified, $BUILDHISTREPO must be writable from your gitlab-ci user

• you have set up a PR server, accessible from your runner (which assumes both are located inside of a trusted network)

• local.conf sets RM_WORK_EXCLUDE = "" for maximal reuse of shared-state

This has been only used with a submodule-based setup, other setups may work too (eg. using git subtrees or Android’s repo), examples and patches welcomed.

current limitations

• assumes a single shell runner containing all tools required to build your project

• the shared-state and download dirs are kept in the runner

• to have the CI build for a given target plaform from a given branch of your source repository where it was never CI-built before, you must first push the $BRANCHPREFIX_ARCH/$BASEBRANCHNAME first

• lots of siginfo files in the shared-state get just their timestamp modified, resulting in much more rsync work than really needed

• usage through a submodule hits a limitation of current gitlab, where we must refer to the same submodule sha1 in the gitlab-ci.yml’s `include:file statement

• using after_script does not cause the job to fail if one of the deploying commands fail

• canceling a job from Gitlab does not fully cancel it (may be linked to the double-interrupt idiom designed for interractive use)

• the deploy scripts have to hijack the whole artifacts:paths definition; to mitigate this, we create a directory named artifacts in which your own scripts can add other files you want saved as artifacts

• cleanup of copy-on-write shared-state currently has to be done manually

• buildhistory push is not robust about any change done concurently during the build (limits the single shell runner to one job at a time - but anyway there’s no job control in place today to make running several of them concurently on the same runner)

• buildhistory usage relies on external creation of new buildhistory branches, effectively limiting the build to externally-defined branches, which is a problem to run build jobs on tags

planned evolutions:

• support docker-based runners (investigate CROPS and Pyrex)

• get gitlab-ci to support submodules natively

• add a safety net to avoid sha1 discrepancy between /ci and /.gitlab-ci.yml

• check reachability of deploy servers before starting the build

• auto-management of buildhistory branches, mirroring source history

how to use

NOTE

this section is far from exhaustive, please refer to the "example use" and read oe.yml to use. Most notably, variables required by the various job templates are defined inside oe.yml.

Aside from gitlab-ci variables, the configuration can be further adjusted for the project’s need using hook scripts:

ci-mkconf-hook.sh

a bash snippet to be sourced after creation of the build tree, suitable for appending project-specific variables to conf/local.conf. It is sourced by mkconf after sourcing poky/oe-init-build-env (or a user-provided alternate env-setting script) and thus runs from within the build tree, and can use any of the local variables set in mkconf, including OE envvars.

protection against shared-state polution

A file named permissions can be created in the sstate-cache dir, to specify which repositories, branches, and tag patterns will be allowed to contribute to this particular shared-state. It is expected to contain one line per allowed repository, with fields separated by a | character.

Currently defined fields are:

<repo_url>|<branch_list>|<tag_pattern_list>

<branch_list> is interpreted as a space-separated list of branch names. <tag_pattern_list> is interpreted as a space-separated list shell glob patterns against which tags are checked.

If <branch_list> or <tag_pattern_list> is not specified or empty, all branches (resp. all tags) are allowed. If no line matches the repo to which the job belongs, or if the branch or tag does not match the declaration for this repo, the build is aborted.

If the permissions file does not exist, no restriction is enforced. If it is empty, the rules described above imply no job can contribute to this shared-state any more.

To avoid older settings from breaking, the build will also be aborted if a legacy BASEGEN_ALLOWED_BRANCHES variable is defined, to give the opportunity to migrate to permissions files, rather than just ignoring the requested safeguard (as low-tech as it was).

debugging

You may run mkconf from your own working copy to see if the generated build configuration meets your expectations. It will emulate being running by gitlab-ci from your current branch.

using a newer version

• update the scripts (eg. git add the newer version of submodule)

• update include sha1 in [.gitlab-ci.yl]

• check what changed in [NEWS.adoc] since the previous version

example use

include:
  - remote: "https://raw.githubusercontent.com/BladeGroup/gitlab-oe/161fd388350fb263a35f1ec598d1911ab3926035/oe.yml"

variables:
  GIT_SUBMODULE_STRATEGY: recursive
  BUILD_GEN: "1.2"
  POKY_BASE: "warrior"
  B: "build"
  BUILDHISTREPO: "ssh://git@git.company.net/oe/buildhistory.git"
  PRSERV: "prserv.company.net:8585"

.template_variables_board1: &variables_board1
  YOCTO_ARCH: "x86_64"
  UPDATE_ARCH: "board1"
  BRANCHPREFIX_ARCH: "board1"

.board1_exceptions:
  except:
    - /^board2\/.*$/

cache:
  paths:
  - $B/cache

stages:
  - build

.my_oe_setup:
  extends: .oe_setup
  tags:
    - oe
  when: manual
  only:
    - branches

board1-image:
  variables:
    <<: *variables_board1
  extends:
    - .my_oe_setup
    - .oe_deploy
    - .oe_buildlogs_artifacts
    - .board1_exceptions
  allow_failure: false
  stage: build
  script:
    - ./ci/run-bitbake --dir "${B}" -- -k core-image-x11 my-packagegroup
# save/deploy artifacts
    - ./ci-my-images-deploy ...

board1-sdk:
  variables:
    <<: *variables_board1
    IMGROOT: core-image-x11
  extends:
    - .my_oe_setup
    - .oe_buildlogs_artifacts
    - .board1_exceptions
  stage: afterbuild
  script:
    - ./ci/run-bitbake --dir ${B} -- ${IMGROOT} -c populate_sdk
    - tar -C $B/tmp/deploy/sdk/ -cvf - . --xform=s,^.,${YOCTO_ARCH}/${BASEBRANCHNAME}, | ssh -p ${SSH_SDK_PORT} ${SSH_SDK_SRV} tar -C ${SSH_SDK_DIR} -xf -

using the server’s shared-state

The rsync-sstate script will mirror the shared-state uploaded to a server after the build. The rsync URL for the shared-state repo can be specified in a .gitlab-oe.conf at the toplevel of your git repository, alongside .gitlab-ci.yml. It can be overriden by the --repo flag if needed.

The shared-state generation will be read from .gitlab-ci.yml, and can be overriden by the --gen flag.

If your BUILD_GEN is set to 1.2+next+mytest, you must set SSTATE_MIRRORS to make use or 1.2+next and 1.2, or the artifacts provided by those base shared-states won’t be visible and will be rebuilt. For example:

SSTATE_MIRRORS ?= "\
file://.* file://${HOME}/sstates/sstate-cache-1.2/PATH \n \
file://.* file://${HOME}/sstates/sstate-cache-1.2+next/PATH \n \
"
SSTATE_DIR = "${HOME}/sstates/sstate-cache-3+dev"

Those base sstates are fetched automatically by rsync-sstate.

example .gitlab-oe.conf config

REPO=user@host:/path/to/sstates