Testing Changes and Producing Binary Packages
As described below, testing changes or producing binary packages to the ports repository assumes the presence of portshaker, poudriére, and git. For the sake of completeness, we will briefly present their configuration as well. If this is all set, just skip to the "The Basic Routine" section.
Poudriére a is a tool to test the packaging process and build binary packages in bulk for distribution. It can be installed from the Ports Collection, as ports-mgmt/poudriere. There can be also a -devel version found.
# cd /usr/ports/ports-mgmt/poudriere
# make install clean
Poudriére can work with and without zfs(8), and it can also utilize tmpfs(5) for the better performance. Setting up tmpfs(5) is highly recommended as it greatly reduces the time required for bulk builds. Poudriére can also do builds in parallel and for different architectures, so many-core amd64 systems are also certainly useful to have. Once poudriere is installed, remember to configure it to your needs, the details will not be discussed here.
In general, it is suggested to create jails for all combination of supported architectures and major operating system version. Currently, they are as follows: FreeBSD 8, 9, 10 and i386 and amd64. Note that, for each major branch, the oldest supported minor version has to be picked in order to maintain ABI compatibility for the produced packages. According to this, as the time of writing, the following jails shall be created: FreeBSD 9.3, and FreeBSD 10.1 for both i386 and amd64.
# poudriere jail -c -j 93i386 -a i386 -v 9.3-RELEASE -m ftp
# poudriere jail -c -j 93amd64 -a amd64 -v 9.3-RELEASE -m ftp
# poudriere jail -c -j 101i386 -a i386 -v 10.1-RELEASE -m ftp
# poudriere jail -c -j 101amd64 -a amd64 -v 10.1-RELEASE -m ftp
In addition to jails, there is a ports tree needed. Poudriére creates a default ports tree, but perhaps it is safer to maintain a dedicated tree. Note that this tree is going to be empty by default as portshaker will be used for population.
# poudriere ports -c -F -p haskell
Similarly to poudriere, portshaker has to be installed from the Ports Collection. It can be used for maintaining only an overlay for the ports tree, instead of the full tree. Hence portshaker takes two or more trees and merges them into a single one. This way one does not have to track the changes outside of the set of the maintained ports.
# cd /usr/ports/ports-mgmt/portshaker
# make install clean
However, there may be conflicts, when the same port exists in multiple trees. In this case, portshaker always keeps the port of the later version, ignoring the other one. This is the expected behavior, when the overlay has the latest version. But sometimes it does not, so portshaker will emit a warning that it is not going to merge the given port, as the version goes backwards. That is why it is so important to keep the overlay in sync with the Ports Collection.
The haskell ports tree of poudriere has to be configured in the configuration file of portshaker, which is $LOCALBASE/etc/portshaker.conf. Note that, due to mistakes in the portshaker port, it likes to overwrite this file with the defaults on install/reinstall/upgrade. Make sure to create a backup of this file, once it is done.
The following lines have to be present in the configuration file.
poudriere_ports_mountpoint=$LOCALBASE/poudriere/ports
mirror_base_dir="/var/cache/portshaker"
ports_trees="main haskell"
main_ports_tree="$PORTSDIR"
main_merge_from="ports"
haskell_poudriere_tree="haskell"
haskell_merge_from="ports haskell"
Obviously, both $PORTDIR and $LOCALBASE have to be replaced with their actual values. Do not attempt to use portshaker itself to update the haskell tree — it will be used in a non-standard way, see below. Instead, only a dummy script is created:
# echo > $LOCALBASE/etc/portshaker.d/haskell
# chmod +x $LOCALBASE/etc/portshaker.d/haskell
Another important component of the procedure is git. It is used because the development overlay is maintained with git, and it is quite convenient to use the same repository but on a different server. This is the server which will be simply referenced as $staging_server. This is the system that is assumed to be used for testing, it shall have poudriere and portshaker installed as well.
That is also where a directory for the staging git repository shall set up. That is what will be called $staging_repository. It is a regular directory where a bare git repository shall be created, which would be used as a target for pushing.
$ mkdir -p $staging_repository
$ cd $staging_repository
$ git init --bare
Once it is ready, the client — which is normally used for everyday development (and which may be, and actually recommended to be, different from the testing machine) — could add this repository as a remote, named stage.
$ git remote add stage ssh://$staging_server/$staging_repository
All right, that is all about the basic setup.
By the assumptions described in the section about the prerequisites, the everyday routine would be as follows.
First, the untested changes, recorded as git commits, should be transferred to the staging server. This is where the use of git and the previously set stage remote come handy.
$ git push stage
Sometimes the push operation has to be forced when it contains rebased commits. This is quite acceptable when the changes have not yet been pushed to any public places, such as GitHub.
$ git push stage --force
Before launching the build, the poudriere ports tree has to be merged. Since this is the result of merging the Ports Collection with the Haskell ports development overlay, it is recommended to update the FreeBSD ports tree to its latest version first. In case of repeated testing in a short time period, this step could be safely omitted.
Note that the steps from now on will be happening on the staging server.
# portshaker -u ports
Because the git clone may have a different history due to rebases, the whole clone has to be recreated from scratch. The yes(1) command is used to keep portshaker working with the merging as it always stops waiting for user input when it finds a conflicting port. As described above, this is fine until the overlay has the newer version of the port.
# rm -rf /var/cache/portshaker/haskell
(either) for testing:
# git clone $staging_repository /var/cache/portshaker/haskell
(or) for producing binary packages:
# git clone https://github.com/freebsd-haskell/ports /var/cache/portshaker/haskell
Finally:
# yes i | portshaker -v -m haskell
When the merging has completed, poudriere is ready to be launched. However, before we would do so, a list of ports to be tested has to be compiled. Those are the ports that are stored in the development overlay, so find(1) is used create an up-to-date list of them. The lang/ghc port is omitted on purpose, it does not have to be rebuilt if not necessary.
However, note well that all the Haskell ports is listed: this also has a purpose. Haskell ports are often interconnected, that is why all the ports have to be verified for building. A full build indeed takes some time to complete, but it is important to catch such mistakes as soon as possible. Thanks to poudriere, ports are rebuilt only when it is really required, in incremental fashion.
# find /var/cache/portshaker/haskell -depth 2 | cut -d '/' -f6- | grep -v ^\.git | grep -v "lang/ghc" | sort > freebsd-haskell.ports
Use the previously created list of ports and choose a jail for the run. Remember to add the -t flag for poudriere to allow it to test for verifying the packaging lists. Sometimes it is useful to add the -c flag to clean up all the previously built binary packages before starting the build.
# poudriere bulk -f freebsd-haskell.ports -j 101amd64 -t -p haskell
Check the resulting logs, test with other jails. In case of errors, investigate the causes, fix the problem and repeat the whole procedure.
In case of binary package building, the goal is to provide pkg(8) repositories for the users. This step is recommended to provide packages for wider testing, ideally coupled with a Call for Testing announced on the freebsd-ports mailing list. It is similar to regular testing, but checking for leftovers, that is, the -t flag can be safely omitted. This can be also taken as a final testing to see if every package builds on every possible platform.
However, this time packages for every jails shall be built — the following commands are recommended to be put in a script. This may take a while, depending on the hardware available.
# find /var/cache/portshaker/haskell -depth 2 | cut -d '/' -f6- | grep -v ^\.git | sort > freebsd-haskell.ports
# poudriere bulk -c -f freebsd-haskell.ports -j 93amd64 -p haskell
# poudriere bulk -c -f freebsd-haskell.ports -j 93i386 -p haskell
# poudriere bulk -c -f freebsd-haskell.ports -j 101amd64 -p haskell
# poudriere bulk -c -f freebsd-haskell.ports -j 101i386 -p haskell
Once the packages have been built, a webserver is needed to serve packages, for example www/nginx. Poudriére stores the generated packages under $LOCALBASE/poudriere/data/packages/$jail-$portstree. Those are the directories what should be published. For the directories, a possible layout is as follows. Here, $DOCUMENT_ROOT is the path of the document root, as configured in nginx.
location / {
root $DOCUMENT_ROOT;
autoindex on;
}
and the directories in the file system:
$DOCUMENT_ROOT/freebsd:$MAJOR:$ARCH
where $MAJOR is 9, 10 and $ARCH is x86:32 (i386) and x86:64 (amd64), respectively. Under those directories, it is recommended to have the packages created by poudriere in directories marked with dates. Then a symbolic to the latest one could be placed as Latest. An example of this layout can be found on pkg.freebsd.org.
As a result, the following pkg(8) repository configuration should just work out of the box.
Haskell: {
url: "$pkg_url/${ABI}/latest",
enabled: yes
}
where $pkg_url is the URL of the server which is publishing the binary packages. Note that the ${ABI} string should not replaced for anything. It is meant to be verbatim, that is how pkg(8) works.