Use Github Actions for functional testing jobs

[EmilienM]

Problem Description

It was in late 2017 when openlab folks reached out and proposed to integrate OpenLab systems into Gophercloud CI.
This was a great idea at that time, Zuul is one of the most powerful system to drive continuous integration and using nodepool was a given for us, where we had free resources for testing.
The challenge after a few releases, where CI jobs weren't very well maintained and we could observe a lot of failures in our tests. Here is a quick overview of what we skip because it doesn't work fine in Openlab.
Very often, it involves patching OpenLab Zuul jobs which is an unmaintained (by unmaintain I mean that they were copied once and now additions are made when needed, but they don't follow OpenStack zuul jobs anymore, where a lot of features were added) fork of the OpenStack Zuul jobs and therefore hard to make it work at every release.
For example, if a contributor wants to add a new service, they'll have to patch OpenLab Zuul jobs to configure Devstack for the new service (this logic already exists in OpenStack zuul jobs, but we have to duplicate it here), wait for a review and then run the tests in Gophercloud. If a configuration is wrong or missing, they have to repeat this operation until this works, because the integration between Openlab Zuul jobs & Gophercloud isn't fast as you can see.

This process makes iterations slower than they could be if we moved Gophercloud to OpenStack, where we could directly use the Devstack jobs that are well maintained by the OpenStack community. See #2257, it's pretty clear that for now we won't move the project.

In summary, here is a list of problems that we have today:

• Jobs are slow to run (more than 2 hours for some of them)
• Jobs are inefficient (deploying a lot of services and combining tests)
• Process to add/change jobs is complex and relies on unmaintained forks (even though OpenLab maintainers are super responsive and usually merge PRs in less than a day or two).
• Impossible to test a CI change without merging code in openlab repo and then make a change to Gophercloud
• Nobody in Gophercloud (beside a few stackers) seem comfortable with OpenLab & Zuul

Proposed Change

Overview

I would like to propose a new way forward to improve our CI.

• Something simple and easy to maintain - the only required skills would be
  • Go & how to run acceptance tests
  • Gophercloud itself
  • Devstack (our framework to install OpenStack)
  • Github CI tooling based on Github Actions - using the devstack-action that I wrote
• Something that we maintain and can quickly iterate over (ie in-tree) - Github actions are in-tree so within a single PR we can change our CI config to e.g. enable a new service, add Gophercloud code to add a new service and add tests.
• Something efficient, no more jobs that tests everything, while patching one file for a specific service
  ** As a developer, if I change a file for blockstorage, I want to have a CI job that will deploy Cinder and run acceptance tests for blockstorage only. I don't care about other services.
• Something fast (a job that takes less than 30 min to finish)

Here is a prototype that implements Github Actions to run acceptance tests:

• A basic job, with defaults, testing basic acceptance tests: https://github.com/gophercloud/gophercloud/runs/4525497491
• A service-specific job, which enable Manila and only runs sharedfilesystems acceptance tests: https://github.com/gophercloud/gophercloud/runs/4488097832

Both run in ~20 minutes.
The job layout for Manila looks like this:

gophercloud/.github/workflows/functional-sharedfilesystems.yaml

Lines 1 to 71 in 7f0d0e9

	name: functional-sharedfilesystems
	on:
	pull_request:
	paths:
	- '^.*sharedfilesystems.*$'
	- '.github/workflows/functional-sharedfilesystems.yaml'
	jobs:
	functional-sharedfilesystems:
	strategy:
	fail-fast: false
	matrix:
	name: ["master"]
	openstack_version: ["master"]
	ubuntu_version: ["20.04"]
	include:
	- name: "xena"
	openstack_version: "stable/xena"
	ubuntu_version: "20.04"
	- name: "wallaby"
	openstack_version: "stable/wallaby"
	ubuntu_version: "20.04"
	- name: "victoria"
	openstack_version: "stable/victoria"
	ubuntu_version: "20.04"
	- name: "ussuri"
	openstack_version: "stable/ussuri"
	ubuntu_version: "18.04"
	- name: "train"
	openstack_version: "stable/train"
	ubuntu_version: "18.04"
	runs-on: ubuntu-${{ matrix.ubuntu_version }}
	name: Deploy OpenStack ${{ matrix.name }} with Manila and run sharedfilesystems acceptance tests
	steps:
	- name: Checkout Gophercloud
	uses: actions/checkout@v2
	- name: Deploy devstack
	uses: EmilienM/devstack-action@v0.4
	with:
	branch: ${{ matrix.openstack_version }}
	conf_overrides: |
	enable_plugin manila https://opendev.org/openstack/manila ${{ matrix.openstack_version }}
	MANILA_SERVICE_IMAGE_ENABLED=False
	SHARE_DRIVER=manila.share.drivers.lvm.LVMShareDriver
	MANILA_ENABLED_BACKENDS=chicago,denver
	MANILA_BACKEND1_CONFIG_GROUP_NAME=chicago
	MANILA_BACKEND2_CONFIG_GROUP_NAME=denver
	MANILA_SHARE_BACKEND1_NAME=CHICAGO
	MANILA_SHARE_BACKEND2_NAME=DENVER
	MANILA_OPTGROUP_chicago_driver_handles_share_servers=False
	MANILA_OPTGROUP_denver_driver_handles_share_servers=False
	SHARE_BACKING_FILE_SIZE=32000M
	MANILA_DEFAULT_SHARE_TYPE_EXTRA_SPECS='snapshot_support=True create_share_from_snapshot_support=True revert_to_snapshot_support=True mount_snapshot_support=True'
	MANILA_CONFIGURE_DEFAULT_TYPES=True
	- name: Checkout go
	uses: actions/setup-go@v2
	with:
	go-version: '^1.15'
	- name: Run Gophercloud acceptance tests
	run: ./script/acceptancetest
	env:
	DEVSTACK_PATH: ${{ github.workspace }}/devstack
	ACCEPTANCE_TESTS_FILTER: '^.*sharedfilesystems.*$'
	- name: Generate logs on failure
	run: ./script/collectlogs
	if: failure()
	- name: Upload logs artifacts on failure
	if: failure()
	uses: actions/upload-artifact@v2
	with:
	name: functional-sharedfilesystems-${{ matrix.name }}
	path: /tmp/devstack-logs/*

One layout per service (baremetal, compute, etc) and we can configure when to run them, depending on which file is touched.

Alternatives

• Stay under OpenLab and:

  • Continue to optimize the jobs in Zuul layout
  • Investigate if some of us could have merge rights into openlab-zuul project to iterate faster
  • Document how to contribute to the Zuul configs, so our community can help into improving our CI

• Move the project under Opendev and:

  • Use Opendev infra for CI
  • Rewrite Zuul layout to work with Opendev CI (should not be a big deal)
  • Take benefits from not having to maintain the Zuul playbooks
  • Investigate collaboration with other projects (e.g. Nova, etc) to run Gophercloud integration testing under these projects as well to improve quality

Developer impact

• Faster feedback from CI
• Easier to reproduce and debug when something fails
• Easier to contribute when adding/modifying components
• Have to learn about Github Actions

Quality impact

• Overall quality will improve because we'll run more tests (the ones we used to skip before)
• Tests should be more reliable and visible (per component, versus one job for networking, one for compute, etc)
• Easier to add new jobs, so quality isn't expensive to improve anymore for our contributors

Implementation

• Each CI job is named a Github Action Workflow.
• Each workflow is only run when some files are being touched.
• Each workflow will test one component (e.g. Cinder for blockstorage)
• Each workflow will follow the same layout: deploy devstack, run acceptance, collect logs when a failure occurs

Assignee(s)

@EmilienM + ?

Work items

• Add CI jobs per component (see prototype)
• Once devstack-action has reached what we need to satisfy the first item, let's release its v1
• Make the jobs working from master down to stable/train (at least)
• Remove Zuul jobs
• Document how to reproduce what does CI
• Document how to debug CI

Dependencies

• Stability of devstack
• devstack-action to reach v1

Testing

Testing happens every time a PR is made to the new CI jobs, so we get immediate feedback.
A new CI job won't be voting until it's actually working and stable.

Documentation Impact

We need to document:

• how CI works (what the jobs do, how they run, etc)
• how to reproduce CI jobs
• how to debug CI jobs

Any feedback is highly welcome.

[EmilienM]

@jtopjian I would use your feedback on this one, thanks.

[jtopjian]

I like this idea a lot and I think it's really cool that you were able to create an Action for devstack.

As a developer, if I change a file for blockstorage, I want to have a CI job that will deploy Cinder and run acceptance tests for blockstorage only. I don't care about other services.

It's been beneficial to have all services tested for the occasions when a change might affect more than one service. For example, a change to blockstorage could affect how an instance attaches a volume. Or a change to networking could affect the port configuration of an instance.

Is it possible to manually trigger different test suites with Actions? For OpenLab we would do:

recheck queens

when we wanted to run the PR against the Queens release. Can something like this be done for different OpenStack services?

I would not consider this to be a blocker since changes that affect more than one service only happen occasionally.

Both run in ~20 minutes.

This is impressive.

[EmilienM]

It's been beneficial to have all services tested for the occasions when a change might affect more than one service. For example, a change to blockstorage could affect how an instance attaches a volume. Or a change to networking could affect the port configuration of an instance.

Is it possible to manually trigger different test suites with Actions? For OpenLab we would do:

recheck queens

when we wanted to run the PR against the Queens release. Can something like this be done for different OpenStack services?

We can't, because the worklows are tight to files. And we can't ask Github to run a specific Workflow (however we can ask to re-run a workflow that failed before). I think we could/should make it so the jobs run for more files than they do in my proposal. ie bind cinder/nova together, etc and see how it goes.

[EmilienM]

First proposal: #2307

[EmilienM]

• basic job
• baremetal job
• blockstorage job
• clustering job
• compute job
• container job
• containerinfra job
• db job
• dns job
• identity job
• imageservice job
• keymanager job
• loadbalancer job
• messaging job
• networking job
• objectstorage job
• orchestration job
• placement job
• sharedfilesystems job
• workflow job