"custom_facts.rb" fails to launch on some targets if Puppet has been installed on the target using the system package manager

[zbentley]

Describe the Bug

If I install the Puppet agent on a target using its system package manager, the Puppet package may not be placed in "/opt/puppetlabs", which causes bolt apply to fail.

While most of the "first-class" supported targets (e.g. debian) have puppet installation packages that work correctly, many distros (e.g. Arch) do not.

Expected Behavior

If puppet/facter/etc. are present and usable on a Linux target via an official package installed via the system package manager, bolt apply should not fail.

Steps to Reproduce

1. Get an Arch Linux environment, installed according to the defaults.
2. Install puppet per the Arch guidelines: pacman -S puppet.
3. On a separate machine, configure a Bolt project that contains the Arch environment in its inventory.
4. Do bolt apply --targets arch --log-level debug --execute 'notify{"hello world":;}'
5. Observe that the apply fails, and that log output contains a failure like this:

Starting: task apply_helpers::custom_facts on arch
Running task apply_helpers::custom_facts with '{"plugins":"Sensitive [value redacted]","_task":"apply_helpers::custom_facts"}' on ["arch"]
Running task 'apply_helpers::custom_facts' on arch
{"target":"arch","action":"task","object":"apply_helpers::custom_facts","status":"failure","value":{"_output":"","_error":{"kind":"puppetlabs.tasks/task-error","issue_code":"TASK_ERROR","msg":"The task failed with exit code 127 and no stdout, but stderr contained:\nzsh:1: /tmp/7b7dae3f-c3d9-4482-af14-a7e94a98b6cd/custom_facts.rb: bad interpreter: /opt/puppetlabs/puppet/bin/ruby: no such file or directory\n","details":{"exit_code":127}}}}
Finished: task apply_helpers::custom_facts with 1 failure in 1.73 sec

Environment

• Bolt 3.28, running from MacOS 14.
• Puppet 7.27 on ruby 3
• Arch Linux installed according to the default install guide suggestions (kernel 6.7.6 as of this writing).

Additional Context

I know that Arch isn't a first-class Puppet/Bolt supported environment, and that there are many parts of Bolt and some Bolt modules that have strong opinions about Puppet being installed into /opt/puppetlabs.

This issue is not trying to change that; Bolt itself, and Puppet distributions installed on many platforms are free to insist on /opt/puppetlabs paths for their components.

However, specifically for Ruby scripts invoked by Bolt at the "bootstrap" phase (e.g. custom_facts.rb), I think that Bolt shouldn't care where/how Puppet is installed on the target: Bolt should be able to invoke Puppet as it is installed on the target host, without insisting on /opt/puppetlabs.

Specifically, the below files look like their shebangs should be updated to be location-agnostic for Puppet (the below contains my search for the problematic shebang in files that could be invoked by Bolt on the target, rather than parts of Bolt itself).

libexec/custom_facts.rb
libexec/custom_facts.rb
libexec/query_resources.rb
modules/package/tasks/init.rb
modules/facts/tasks/ruby.rb
modules/service/tasks/init.rb
modules/puppet_agent/tasks/run.rb
modules/puppet_agent/tasks/facts_diff.rb
modules/puppet_agent/tasks/delete_local_filebucket.rb
modules/reboot/tasks/init.rb
modules/puppet_conf/tasks/init.rb

Suggested Fix

When invoking those scripts, Bolt may choose to set up PATH such that /opt/puppetlabs/bin is preferred over other locations (to increase predictability in the face of multiple Puppet installs, which seems unusual), but it should not hardcode the /opt/puppetlabs/bin shebang; doing so needlessly harms compatibility with non-first-class target distributions.

In fact, Bolt already does this in one place, libexec/bolt_catalog; that's just not sufficient to make bolt apply work in its entirety.

[amyb-asu]

This issue may also manifest as "err: sh: 1: /tmp/c70bfc00-15cc-4d00-8995-af728e5deae1/custom_facts.rb: not found" on Ubuntu. In my case I am trying to deploy to ubuntu 24.04 server, which I have installed puppet via apt.

The error message is confusing since the file does exist, but it is because just like you found with installing via pacman, apt does not install to /opt/ so there is no /opt/puppetlabs/puppet/bin/ruby.