Please make the packaging/build pipelines public and provide reproducible builds

[bastelfreak]

Use Case

As a service delivery partner we want to engage into the development. This isn't really possible without at least read-only access to the pipelines. We don't know which jobs exist and what they do. That makes it impossible to add support for new platforms. And as we can see on the existing requests Puppet itself is quite slow for adding support for new distributions/architectures.

Describe the Solution You Would Like

use GitHub actions for supported platforms to build packages (preferred) or at least grant partners access to the Jenkins (which was public in the past).

Describe Alternatives You've Considered

While I hate it that an open source tool uses private pipelines, you could also workaround this by providing new packages faster. However I think reproducible builds are required in the future and that also requires logs.

Additional Context

[joshcooper]

Hi @bastelfreak, so looking through your suggestions:

use GitHub actions for supported platforms to build packages

GitHub actions is a non-starter for us because we need to build packages on platforms like AIX, Solaris, and other OS'es on Power, etc architectures. While it's possible to use self-hosted runners, that's also a non-starter for security reasons.

grant partners access to the Jenkins (which was public in the past).

I think you're oversimplifying. Yes, there was a time where https://jenkins.puppetlabs.com was internet facing, but that was discontinued in 2015 for many reasons. And we also released puppet 4 (AIO packages) around that time, which greatly changed how packages are built. The point being we can't simply flip a switch and make it public again.

Going back to your desired outcome:

As a service delivery partner we want to engage into the development. This isn't really possible without at least read-only access to the pipelines.... That makes it impossible to add support for new platforms.

I think there's some middle ground here. I have a PR demonstrating how it's possible to build the puppet-runtime on Fedora 40 on docker without requiring access to our network. I'm still resolving some small issues, but in the near future it should be possible for the community to submit pull requests for new agent platforms.

you could also workaround this by providing new packages faster

We definitely were lagging in the past, but in the last 9 months we've added 14 platforms in 2 streams (which is double the work because of different ruby, openssl, etc versions). And Ubuntu 24.04 support is almost ready. So looking ahead, I would expect agent packages to be released near the GA of the OS.

[bastelfreak]

GitHub actions is a non-starter for us because we need to build packages on platforms like AIX, Solaris, and other OS'es on Power, etc architectures

I know, because of that I wrote "use GitHub actions for supported platforms". Multiple times I heard from Perforce employees that getting a new operating system into vmpooler is a bigger challenge than building the actual new packages. GitHub Actions support the majority of distributions that Perforce builds packages for. So why not use it wherever possible?

I think you're oversimplifying.

Yes I am. Because Puppet/Perfoce since 2015 doesn't provide proper information on how those internal pipelines look like, which git repositories have internal pipelines. I have to guess because it's internal and information don't exist.

in the near future it should be possible for the community to submit pull requests for new agent platforms.

And will those jobs run internally? Will we have access to the logs? Otherwise I don't see much benefit in this (under the assumption that my goal is to get official packages. It would help if my usecase is to build own packages).

[joshcooper]

Multiple times I heard from Perforce employees that getting a new operating system into vmpooler is a bigger challenge than building the actual new packages.

Your information is out of date. Adding images is fairly automated now.

So why not use it wherever possible?

It may be possible to build in docker containers in the future. It solves some problems, but creates others, such as needing to disable promotions, configure dependencies, etc in two places. And there's the issue of out-of-band security releases.

Also building packages is only half the battle. The packages still need to be tested. And because of the way puppet modifies system state, you need an actual VM, not a docker container.

And will those jobs run internally? Will we have access to the logs? Otherwise I don't see much benefit in this

Consider the fedora 40 situation. It ships with a version of gcc that fails to compile libffi 3.4.3. The issue was fixed in libffi 3.4.4, so I added a commit to bump libffi:

puppetlabs/puppet-runtime@bb9e7e0

This is a prime example of something that could have been discovered and fixed by the community without needing access to jenkins or logs. In other words, if the community can remove road blocks for new platforms, then it's fairly trivial to add official packages.

[bastelfreak]

Your information is out of date.

I think it would be awesome to have infos like that on a newsletter or the mailinglist

Consider the fedora 40 situation. It ships with a version of gcc that fails to compile libffi 3.4.3
This is a prime example of something that could have been discovered and fixed by the community

How could we discover this at the moment? I'm not aware of any documentation about puppet-runtime/vanagon that enables me to run it locally and I don't have CI logs that expose those kind of information.

[joshcooper]

I think it would be awesome to have infos like that on a newsletter or the mailinglist

We could certainly do better sending out communication about OS's we're working on/planning for the next quarter, for example, Ubuntu 24.04 is almost ready to ship nightly.

I'm not aware of any documentation about puppet-runtime/vanagon that enables me to run it locally

Yes, the docs are completely missing and need to be added before building the runtime is useful outside of Puppet. I'm looking into making that part better.