Skip to content

Puppet 8 Compatibility

Jump to bottom
Ben Ford edited this page May 16, 2023 · 13 revisions

We recently announced our plans for Puppet 8 and this document describes steps you can take to prepare. The first section describes possible compatibility issues with Ruby 3.2, while the second describes issues with Puppet 8..

Ruby 3.2 Compatibility

To verify your code is Ruby 3.2 compatible, run tests locally against Ruby 3.2:

rbenv install 3.2.0
rbenv shell 3.2.0
rm Gemfile.lock
bundle
bundle exec rake spec

And add it to your Github Action test matrix. For example, Puppet did this for Ubuntu 22.04 and Windows.

Note that if you are attempting to do this on Centos 7, the version of gcc shipped with the OS is too old (gcc 4 vs. 6). Follow the instructions at https://linuxize.com/post/how-to-install-gcc-compiler-on-centos-7/, and this will allow you to install Ruby 3.2.

Here are common "gotchas" you may run into during this process:

Explicit distinction between positional vs named args

Changes to the way keyword and positional arguments are handled in ruby 3 are extensively documented in this post: https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/

Note that this will happen at run time (not when the code is loaded). Errors will usually be "wrong number of arguments".

Example commit: https://github.com/puppetlabs/bolt/commit/acda807a1c0c00f8221ebad8c7acd446dda8ae4f

Psych compatibility

Broadly the psych library support for parsing has included several breaking changes. Some common ones:

Legacy functions used to determine psych version: https://github.com/puppetlabs/bolt/pull/3093/commits/d43a071894fcb88267cb2116e7d3486d7af5acd9

YAML#safe_load positional vs named arguments: https://github.com/puppetlabs/bolt/pull/3093/commits/e04a399c3324da506a0df4367a1b56938e89863a

{File,Dir}.exists? removed

These are now standardized to exist?.

https://www.ruby-lang.org/en/news/2022/12/25/ruby-3-2-0-released/

Example commit: https://github.com/puppetlabs/puppet/commit/a94db64910381add4428a7654d867ad023d27844

Struct::{Passwd,Group} removed

These are now standardized as Etc::Passwd and Etc::Group respectively.

Example commit: https://github.com/puppetlabs/puppet/commit/dd3df54c190cf017f62c9292ccb3cc5cb21230f1

SortedSet removed

This was extracted into a separate sorted_set gem. Consider using Set or perhaps .uniq.sort on a plain array.

Taint feature removed

The Kernel#taint and Kernel#untaint methods were removed.

https://bugs.ruby-lang.org/issues/16131

Versions of the fast-gettext gem prior to 2.1 call Kernel#untaint and therefore won't run on Ruby 3.2 This also affects modules that depend on the gem directly or transitively through gettext-setup or r10k. You'll want to update to fast-gettext 2.1 or later.

Example commits: https://github.com/puppetlabs/r10k/commit/cb0a34638f72e0a96c0f2e322d084fb9b1cead14 https://github.com/puppetlabs/gettext-setup-gem/commit/f4b292c51d2c72e76515f3b81d5ebd379a2bbda8

ERB now requires keyword arguments

Passing positional arguments to ERB.new is no longer allowed, so you'll have to change:

-ERB.new(content, 0, '-')
+ERB.new(content, trim_mode: '-')

RSpec 3.12.2 or later

Earlier versions of rspec-mocks incorrectly verified expectations when passing positional arguments to a method that accepted keyword arguments and vice-versa. Use rspec-* 3.12.2 or later when testing against Ruby 3.2. Note that the "rspec" gem only has a 3.12.0 version, but has a dependency on rspec-core, rspec-expectations, and rspec-mocks that will use the latest Z of these gems, so you should not need to specify them explicitly unless you already are.

Rake 13.0.6 or later

Earlier versions of rake called FileUtils with an options hash, but it only accept keyword arguments. This resulted in failures like:

 `touch': wrong number of arguments (given 2, expected 1)

A number of issues exist in 13.0.0 and 13.0.5, so use 13.0.6 or greater.

Example commit: https://github.com/puppetlabs/facter/commit/30061cc5bc7694ef0237d1cdf25dc1321849e1f3

OpenSSL 3 Compatibility

Puppet 8 vendors OpenSSL 3. To verify your code works as expected, we recommend testing against ubuntu-22.04 in your github action, since the OS ships with OpenSSL 3. The windows github runner for ruby 3.2 also ships with OpenSSL 3:

Dir.glob is sorted by default

Prior to ruby 3.0, Dir.glob returned filenames in non-deterministic order. In ruby 3.0 and up, the method returns sorted filenames by default. This can lead to subtle changes in behavior when upgrading from ruby 2.7. Though the resulting behavior will be deterministic moving forward.

https://bugs.ruby-lang.org/issues/8709

Module Compatibility

If you have a puppet module, then you will want to follow this process to verify it will run on Puppet 8:

  1. Update metadata.json to allow puppet 8.0, for example "version_requirement": ">= 6.0.0 < 9.0.0"
  2. Run pdk update to update the module's templates. You'll need to use template versions later than 2.6.0 to remove the dependency on puppet-module-gems.
  3. Run unit tests against puppet 8 gem, either from a local checkout or github: 
    export PUPPET_GEM_VERSION=https://github.com/puppetlabs/puppet#main
rm Gemfile.lock
bundle install
bundle exec rake spec
  4. Run acceptance tests against nightly puppet-agent packages (the latter is coming soon). Alternatively, test against the last passing puppet-agent build, if you have access to that.

Here is a list of issues your module may run into during this process and steps to remediate.

Legacy Facts

By default, puppet 8 agents will no longer send legacy facts to the server, so those facts cannot be referenced in puppet code, ERB & EPP templates nor hiera configuration. To check for legacy facts in puppet code, we recommend using the legacy_facts puppet-lint plugin.

Note the check will only detect legacy facts in puppet code. ERB/EPP templates and hiera configuration must be checked manually.

Example commit to puppet code in puppetlabs-ntp.

Example change to ERB template:

-<%= $fqdn %>
+<%= $facts.dig("networking","fqdn") %>

Example change to hiera config.yaml:

-os/%{osfamily}.yaml
+os/%{facts.os.family}.yaml

It is possible to restore puppet's 7.x behavior by setting include_legacy_facts=true in puppet.conf.

Strict Mode

Puppet now defaults strict_variables=true and strict=error. As a result, puppet code like the following will not compile. This assumes var is the name of an undefined variable:

notice($var)

Nor will code that performs coersions like:

notice("1" + 1)

In previous versions of Puppet, facts and variables were conflated in Hiera lookups. This meant that many hierarchies reference legacy facts as top-level variables, which is problematic now because referencing an undefined variable in a hiera lookup will now fail:

hierarchy:
  - name: "missing"
    path: "%{var}.yaml"

However, lookup will continue to work as expected for undefined facts, since that's what allows optional hierarchy levels to exist. For example, if the fact is missing, then the lookup will fall through to common.yaml:

hierarchy:
  - name: "overrides"
    path: "%{facts.optional}.yaml"

  - name: "common"
    path: "common.yaml"

It is possible to restore puppet's 7.x behavior, by setting strict=warning and/or strict_variables=false in puppet.conf.

Hiera 3

The hiera gem no longer ships in puppet-agent 8 packages. If you have a v3 backend that extends Hiera::Backend or a hiera v5 data provider that delegates to hiera v3, then you'll need to update the backend to hiera v5. Alteratively, you can install the hiera gem in puppetserver's JRuby and reload puppetserver:

package { 'puppetserver-hiera':
  ensure   => present,
  name     => 'hiera',
  provider => puppetserver_gem
} 
~> Service[puppetserver]

And install the hiera gem into the agent's ruby so that puppet lookup continues to work:

package { 'puppet-hiera':
  ensure   => present,
  name     => 'hiera',
  provider => puppet_gem
}

PSON

Puppet 8 will no longer vendor PSON:

PSON is a variant of JSON that puppet uses for serializing data to transmit across the network or store on disk. Whereas JSON requires that the serialized form is valid unicode (usually UTF-8), PSON is 8-bit ASCII, which allows it to represent arbitrary byte sequences in strings. PSON was forked from upstream pure JSON v1.1.9 and patches were added to allow binary data.

PSON is implemented in ruby without native extensions, so it's much slower and memory intensive than JSON or MessagePack. And PSON hasn't been updated to keep up with changes made in the upstream pure JSON library. Also if binary data is accidentally added to the catalog, such as trying to manage a Kerberos keytab file, and the catalog contains Deferred, Sensitive, etc data types, then the agent run will downgrade to PSON and fail.

In Puppet 6, we enabled "rich data" support by default between agents and servers. This is what enables Deferred functions to be evaluated on the agent. Puppet also supports a Binary data type, so there isn't any reason to use PSON anymore. For example, to distribute small binary content, you can use the binary_file function.

The PSON gem monkey patches core Ruby classes like Hash, Array, String, etc. This is why it's possible to call Hash#to_pson. Note JSON and MessagePack do the same thing with to_json and to_msgpack, respectively. This means in Puppet 8, the to_pson method will not be defined on core Ruby classes.

Puppet extensions like termini, functions, report processors, etc can check if PSON is enabled by checking the puppet setting:

if Puppet[:preferred_serialization_format] == "pson"

We extracted the PSON code into a separate repo and published the puppet-pson gem to rubygems.org. PSON can be re-enabled by installing the gem on agents and servers (similar to how Hiera 3 is being handled) and configuring the preferred_serialization_format setting.

Clone this wiki locally