section | x-masysma-name | title | date | lang | author | keywords | x-masysma-version | x-masysma-repository | x-masysma-website | x-masysma-owned | x-masysma-copyright | |||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
32 |
masysmaci/main |
Ma_Sys.ma CI |
2020/03/28 17:56:53 |
en-US |
|
|
1.0 |
1 |
Copyright (c) 2019, 2020 Ma_Sys.ma.
For further info send an e-mail to Ma_Sys.ma@web.de.
|
The Ma_Sys.ma Continuous Integration system (short: Ma_Sys.ma CI) attempts to be a single-user, lightweight and automation-friendly system to perform a tightly defined set of tasks related to the detection of changes and invocation of build processes.
It was created out of necessity because existing systems were either very
large (e.g. Gitlab and Jenkins), unstable (e.g. Concourse) or too difficult
to automate properly. As a unique feature, Ma_Sys.ma CI does not require a
Git server, nor database of any kind. Instead, it takes repositories from below
a common root directory as its input and writes logfiles to directory
x-co-ci-logs
as its only output.
For Ma_Sys.ma purposes, Ma_Sys.ma CI serves specifically to detect changes to packages, build updated package files and synchronize them to a (private) Debian repository. The synchronization is handled by component masysmaci/pkgsync(32).
To get started with the CI quickly, download the repository to a directory
callled co-ci
and start it with docker-compose
:
$ mkdir root
$ cd root
$ git clone https://www.github.com/m7a/co-ci
$ cd co-ci
$ docker-compose up
This will first build the necessary images and afterwards start three containers equipped for building packages. Check if the CI is alive by querying its REST endpoints:
$ curl http://127.0.0.1:9030
/build
/term
Try out building a package by downloading its repository and triggering its build:
$ cd root
$ git clone https://www.github.com/m7a/bo-d5man2
$ cd bo-d5man2
$ ant trigger
Afterwards, package mdvl-d5man2
should become available through the reprepro
repository at /var/tmp/masysmacirepo
(which is the default location). In case
of failure, consult the logs in directory root/x-co-ci-logs
.
To build your own package, provide files build.xml
, debian-changelog.txt
and
hello.c
from the masysmaci/build(32) documentation
in a directory e.g. called hello
and add the following lines to build.xml
:
<!-- CI INTEGRATION -->
<target name="package_triggered_amd64" depends="package">
<property name="masysma.ci.trigger" value="newver"/>
</target>
Make it a git repository as follows:
$ cd root/hello
$ git init .
$ git add .
$ git commit -m "Initial commit. / Hello world example."
As soon as the changes are commited, the CI should pick up the changes and build
the mdvl-hello
package proposed in the
masysmaci/build(32) documentation. As only the target
for amd64
was added, it will only build the package for that specific
architecture. See Task Definition for how to integrate build processes for
other architectures.
The Ma_Sys.ma CI can be configured to adapt to your local environment by multiple different means depending on what part exactly is to be configured.
The easiest way to change configuration is by using environment variables.
Any of these variables may be supplied on the commandline for docker-compose
or a dedicated .env
file. See the docker-compose
documentation for
details.
The following environment variables are available for configuration
(default values given after =
):
MA_DEBIAN_MIRROR=http://ftp.de.debian.org/debian
: Configures the URL of a Debian mirror to use.
MA_REPOSITORY_ROOT=/var/tmp/masysmacirepo
: Configures the file system location of the reprepro repository to write
files to.
MA_GNUPG_ROOT=./dot_gnupg_sample/
: Configures a .gnupg
directory to use for the CI's container.
The data from that directory is used to sign packages added to the
reprepro repository. It is highly recommended to either change the
contents of dot_gnupg_sample
or configure a different directory here.
See section Key Files and Signatures for details.
The environment variables do not change how the system is composed. If you want to change e.g. the number of containers to use for CI builds or select different architectures to run containers for or even want to integrate with external non-Docker running environments, then it becomes necessary to dig into the details of the configuration.
This advanced configuration is made of two sides:
- The Docker-side: The selection of containers and how their images are
built is specified in
docker-compose.yml
anddockerfile_masysmaci
. Changing them is not “CI-specific” and works according to the syntax and semantics known from Docker. - The Ma_Sys.ma CI-side: To configure the CI, file
masysmaci.xml
is processed upon starting themasysmaci.pl
script. After changing the file it is thus necessary to restart the main CI container (amd64
) for the changes to take effect.
As Docker changes are known from Docker, the following only documents the
specifics of file masysmaci.xml
.
Here is a copy of the default configuration masysmaci.xml
:
<masysmaci>
<conf>
<property name="address" value="0.0.0.0"/>
<property name="port" value="9030"/>
</conf>
<runenv_ssh>
<property name="StrictHostKeyChecking" value="accept-new"/>
<property name="BatchMode" value="yes"/>
<host name="i386" phoenixroot="/home/masysmaci/root">
<property name="HostName" value="i386"/>
<property name="Port" value="2222"/>
<property name="IdentityFile" value="$MDVL_CI_PHOENIX_ROOT/co-ci/dot_ssh_server/id_ed25519_i386"/>
<property name="User" value="masysmaci"/>
</host>
<host name="armhf" phoenixroot="/home/masysmaci/root">
<property name="HostName" value="armhf"/>
<property name="Port" value="2222"/>
<property name="IdentityFile" value="$MDVL_CI_PHOENIX_ROOT/co-ci/dot_ssh_server/id_ed25519_armhf"/>
<property name="User" value="masysmaci"/>
</host>
</runenv_ssh>
</masysmaci>
Below the top-level masysmaci
element, there are two main elements for
configuration: conf
and runenv_ssh
.
Element conf
allows for general key-value associations to be stored. Here, you
can configure the following properties:
address
-- the IP address to listen on for the REST APIport
-- the port to listen on for the REST API
Element runenv_ssh
contains SSH configuration for different running
environments. The concept of a running environment is similar to a
worker of other CI systems -- it is the connection to a computer to run
commands on. As Ma_Sys.ma CI does this through SSH or local commands only, all
running enviroments are configured in terms of SSH connections.
The property
elements directly below runenv_ssh
are SSH configuration
options to set for all runenvs. In the default configuration shown above,
BatchMode
and StrictHostKeyChecking
are configured for non-interactive
SSH use and trust on first connection.
The individual running environments are configured by host
elements. For any
host, there is a name
attribute which uniquely identifies the running
environment in the Ma_Sys.ma CI. Attribute phoenixroot
configures the
directory where all repositories can be found in. This directory needs to
be available in all running environments although in theory, not all running
environments need to access all of the repositories (it is e.g. sufficient that
they can access the repositories they are trying to build packages from).
Below the host
element, properties are set to configure the IP address or
host name and port of the running environment to connect to (HostName
and
Port
). Additionally, the user to login with (User
) and the location of the
SSH private key file to use for connection need to be given (IdentityFile
).
If necessary, additional SSH options can be configured with additional
property
elements.
In the example, $MDVL_CI_PHOENIX_ROOT
is used. Note that this is the only
variable that can be substituted in the properties and it always refers to the
CI host's $MDVL_CI_PHOENIX_ROOT
and not to the running environments'!
Tasks are defined in a build.xml
which needs to be present in a repository's
root directory for it to be recognized by the Ma_Sys.ma CI.
Here are the definitions from lp-cone/build.xml
as an example:
<target name="package_triggered" depends="package">
<property name="masysma.ci.trigger" value="newver"/>
</target>
<target name="package_triggered_i386" depends="package">
<property name="masysma.ci.trigger" value="newver"/>
<property name="masysma.ci.runenv" value="ssh"/>
<property name="masysma.ci.runenv.name" value="i386"/>
</target>
<target name="package_triggered_armhf" depends="package">
<property name="masysma.ci.trigger" value="newver"/>
<property name="masysma.ci.runenv" value="ssh"/>
<property name="masysma.ci.runenv.name" value="armhf"/>
</target>
A <target>
element is recognized by the CI if it contains a property with
name masysma.ci.trigger
. The CI-specific properties are as follows:
masysma.ci.trigger
(required)
: Defines the type of trigger to use.
Currently, newver
and topleveladded
are available.
masysma.ci.runenv
(default: local
)
: Type of running environment to use.
Possible values: local
or ssh
.
masysma.runenv.name
(optional)
: The name of the running environment to use.
masysma.runenv.bg
(default: 0
)
: Specifies whether the target should run as a background process
(allows other targets to run in parallel) or in the foreground
(allows only that target to run).
masysma.ci.trigger.param
(optional)
: Specifies a parameter to pass to the trigger type.
With newver
this is ignored, with topleveladded
, this specifies a
suffix for a file to be recognized by the topleveladded
trigger.
Whenever a trigger for any of the targets is executed, the respective targets
will be invoked by the CI. In the example, three triggers for the same
condition (newver
) are defined to run on different running environments as
to build the mdvl-cone
package for the three processor architectures: amd64,
i386, armhf. Note that the masysma.ci.runenv.name
need not describe an
architecture name -- this is only the default configuration from
masysmaci.xml
.
The newver
trigger type
: The newver
trigger runs whenever a given repository is clean in the sense
that there are no files which are not commited and the package defined by
the repository's build.xml
appears to have a new version according to
debian-changelog.txt
.
The topleveladded
trigger type
: The topleveladded
trigger runs whenever a file is added in the
$MDVL_CI_PHOENIX_ROOT
directory and its suffix matches the value from
masysma.ci.trigger.param
. This mechanism is used by
masysmaci/pkgsync(32) to add newly generated
.deb
files to the repository.
Some of the Ma_Sys.ma CI's features are available through a REST API listening on port 9030 by default. The endpoints are as follows:
/term
(POST)
: Upon sending the POST request to this endpoint, the Ma_Sys.ma CI shuts
down (after awaiting the termination of running foreground subprocesses).
/build
(GET)
: Replies with a list of all repositories recognized by the CI.
This includes all repositories which contain a valid build.xml
independently of whether they use any triggers.
/build/:repository
(GET, POST)
: To use this endpoint, replace :repository
by the directory name of the
repository. Upon querying this endpoint with GET, all targets which could be
triggered are returned. Upon sending a POST request, all of these targets
are triggered.
/build/:repository/:target
(GET, POST)
: To use this endpoint, replace :repository
by the directory name of
the repository and :target
by the target name to consider.
Upon querying this endpoint with GET, the most recent build log is returned
(or 404 if none exists yet). Upon sending a POST request, the sepcific
target is triggered.
The repository contains the following structure of key material:
co-ci/
|
+-- dot_gnupg_sample/
| |
| +-- ...
|
+-- dot_ssh_armhf/
| |
| +-- authorized_keys
|
+-- dot_ssh_i386/
| |
| +-- authorized_keys
|
+-- dot_ssh_server/
|
+-- id_ed25519_armhf
|
+-- id_ed25519_i386
Directories dot_ssh_armhf
and dot_ssh_i386
contain the SSH public keys
for the SSH private keys (identities) below dot_ssh_server
. These keys are
used for the containers to communicate with each other. The containers do not
expose their respective SSH ports and hence, their SSH services are only
available in the Docker network created by docker-compose
and thus, hidden
from the outside world. As a result, there is no need to keep these keys
secret. In case of doubt, they may be re-generated by script regenerate_id.sh
.
Directory dot_gnupg_sample
contains public and private keys used for signing
Debian packages added to the reprepro repository. As the directory name already
implies, they are considered sample keys and should only be used for testing
purposes. Upon deciding to use Ma_Sys.ma CI productively, it is
highly recommended to switch to independently created and private keys.
The script to re-generate dot_gnupg_sample
(with different keys) is provided
in regenerate_dot_gnupg_sample.sh
. Note that for most cases, it is recommended
to generate the keys “manually” rather than using the script as to set a
different user name, e-mail etc.
Ma_Sys.ma CI 1.0, Copyright (c) 2019, 2020 Ma_Sys.ma.
For further info send an e-mail to Ma_Sys.ma@web.de.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.