puppetdb-ssl client cert auth used only for facts but not exported resources

[fheinle]

When I run octocatalog-diff it succeeds gathering facts from puppetdb but later fails compiling the catalog while collecting exported resources indicating errors with OpenSSL.

• What did you do?

run octocatalog-diff against any node

• What happened?

could not retrieve stuff from puppetdb, failing with e.g. openssl errors

• What did you expect to happen?

octocatalog-diff output

• How can someone reproduce the problem?

1. set up puppet using external CA (in my case FreeIPA)
2. set up puppetdb to use this CA
3. get another certificate for running octocatalog-diff, signed by FreeIPA's CA
4. set up octocatalog-diff with puppetdb-ssl-ca, puppetdb-ssl-key and puppetdb-ssl-cert set according to docs
5. run octocatalog-diff

Command used and debugging output

Command used: octocatalog-diff -n puppet.example.com --debug

Debugging output:

D, [2018-07-31T11:10:10.852241 #22274] DEBUG -- : Running octocatalog-diff 1.5.3 with ruby 2.3.1
D, [2018-07-31T11:10:10.853213 #22274] DEBUG -- : Command line arguments: ["--debug", "-n", "puppet.example.com"]
D, [2018-07-31T11:10:10.853564 #22274] DEBUG -- : Running on host portal (x86_64-linux-gnu)
D, [2018-07-31T11:10:10.853848 #22274] DEBUG -- : Compiling catalogs for puppet.example.com
D, [2018-07-31T11:10:10.854499 #22274] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for from-catalog
D, [2018-07-31T11:10:10.854846 #22274] DEBUG -- : Initialized OctocatalogDiff::Catalog::Computed for to-catalog
D, [2018-07-31T11:10:10.855092 #22274] DEBUG -- : Initialized parallel task result array: size=2
D, [2018-07-31T11:10:10.856578 #22274] DEBUG -- : Launched pid=22276 for index=0
D, [2018-07-31T11:10:10.858113 #22274] DEBUG -- : Launched pid=22278 for index=1
D, [2018-07-31T11:10:10.859047 #22276] DEBUG -- : Begin build_catalog for production
D, [2018-07-31T11:10:10.859513 #22276] DEBUG -- : Setting up Puppet catalog build for production
D, [2018-07-31T11:10:10.859701 #22276] DEBUG -- : Catalog for production will be built with OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.859876 #22276] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.860127 #22276] DEBUG -- : Start retrieving facts for puppet.example.com from OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.860333 #22276] DEBUG -- : Retrieving facts from PuppetDB
D, [2018-07-31T11:10:10.866278 #22278] DEBUG -- : Begin build_catalog for .
D, [2018-07-31T11:10:10.866789 #22278] DEBUG -- : Setting up Puppet catalog build for .
D, [2018-07-31T11:10:10.867026 #22278] DEBUG -- : Catalog for . will be built with OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.867222 #22278] DEBUG -- : Calling build for object OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.867478 #22278] DEBUG -- : Start retrieving facts for puppet.example.com from OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:10.867683 #22278] DEBUG -- : Retrieving facts from PuppetDB
D, [2018-07-31T11:10:11.011440 #22276] DEBUG -- : Success retrieving facts for puppet.example.com from OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:11.013671 #22278] DEBUG -- : Success retrieving facts for puppet.example.com from OctocatalogDiff::Catalog::Computed
D, [2018-07-31T11:10:11.016847 #22278] DEBUG -- : Symlinked /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/environments/production -> /home/florian
/dev/puppet
D, [2018-07-31T11:10:11.017517 #22278] DEBUG -- : Installed puppetdb.conf file at /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/puppetdb.conf
D, [2018-07-31T11:10:11.019272 #22278] DEBUG -- : Installed routes.yaml file at /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/routes.yaml
D, [2018-07-31T11:10:11.022572 #22278] DEBUG -- : Installed hiera.yaml from /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/environments/production/
hiera.yaml to /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/hiera.yaml
D, [2018-07-31T11:10:11.055529 #22276] DEBUG -- : ["Exit status: 0"]
D, [2018-07-31T11:10:11.055694 #22276] DEBUG -- : Success git archive /home/florian/dev/puppet:production
D, [2018-07-31T11:10:11.055731 #22276] DEBUG -- : Success git checkout /home/florian/dev/puppet:production -> /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-bootstrap-checkout-20
180731-22276-4rk774
D, [2018-07-31T11:10:11.055787 #22276] DEBUG -- : Begin install bootstrap script in target directory
D, [2018-07-31T11:10:11.056114 #22276] DEBUG -- : Success: copied /home/florian/dev/puppet/bootstrap.sh to /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-bootstrap-checkout-20180
731-22276-4rk774/bootstrap.sh
D, [2018-07-31T11:10:11.056180 #22276] DEBUG -- : Begin bootstrap with 'bootstrap.sh' in /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-bootstrap-checkout-20180731-22276-4rk774
D, [2018-07-31T11:10:11.083511 #22278] DEBUG -- : Installed fact file at /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/yaml/facts/puppet.example.com.yaml
D, [2018-07-31T11:10:11.084687 #22278] DEBUG -- : Installed SSL client certificate in /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/ssl/certs/
portal.pem
D, [2018-07-31T11:10:11.085269 #22278] DEBUG -- : Installed SSL client key in /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/ssl/private_keys/p
ortal.pem
D, [2018-07-31T11:10:11.086038 #22278] DEBUG -- : Installed CA certificate in /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/ssl/certs/ca.pem
D, [2018-07-31T11:10:12.959578 #22278] DEBUG -- : (to) Try 1 executing Puppet 4.10.12: /opt/puppetlabs/bin/puppet master --compile puppet.example.com --storeconfigs --
storeconfigs_backend=puppetdb --factpath=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/yaml/facts --facts_terminus=yaml --no-daemonize --no-ca
 --color=false --config_version="/bin/echo catalogscript" --environment=production --hiera_config=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/hi
era.yaml --environmentpath=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/environments --vardir=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-201
80731-22278-2708v6/var --logdir=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var --ssldir=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-2018073
1-22278-2708v6/var/ssl --confdir=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6
D, [2018-07-31T11:10:12.960353 #22278] DEBUG -- : ["Execute: /tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-scriptrunner20180731-22278-js3c2p/puppet.sh master --compile puppet.example.com --storeconfigs --storeconfigs_backend\\=puppetdb --factpath\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/yaml/facts --facts_te
rminus\\=yaml --no-daemonize --no-ca --color\\=false --config_version\\=\\\"/bin/echo catalogscript\\\" --environment\\=production --hiera_config\\=/tmp/ocd-ipc-20180731-
22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/hiera.yaml --environmentpath\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/environments --vardir
\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var --logdir\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var --ssldir
\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6/var/ssl --confdir\\=/tmp/ocd-ipc-20180731-22274-p7r0rx/ocd-builddir-20180731-22278-2708v6"]

# skipped a few deprecation warnings from puppet modules here

[2018-07-31T11:10:31.080994 #22278] DEBUG -- : ["STDERR: Warning: Error connecting to puppet.example.com on 8081 at route /pdb/query/v4/resources?query=%5B%22and%22
%2C%5B%22%3D%22%2C%22type%22%2C%22Sshkey%22%5D%2C%5B%22%3D%22%2C%22exported%22%2Ctrue%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22certname%22%2C%22puppet.example.com%22%5D%5D%5D, error message received was 'SSL_connect SYSCALL returned=5 errno=0 state=unknown state'. Failing over to the next PuppetDB server_url in the 'server_urls' list"]
D, [2018-07-31T11:10:31.081028 #22278] DEBUG -- : ["STDERR: Error: Could not retrieve resources from the PuppetDB at puppet:8140: Failed to execute '/pdb/query/v4/resourc
es?query=%5B%22and%22%2C%5B%22%3D%22%2C%22type%22%2C%22Sshkey%22%5D%2C%5B%22%3D%22%2C%22exported%22%2Ctrue%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22certname%22%2C%22puppet.example.com%22%5D%5D%5D' on at least 1 of the following 'server_urls': https://puppet.example.com:8081 on node puppet.example.com"]
D, [2018-07-31T11:10:31.081057 #22278] DEBUG -- : ["STDERR: Error: Could not retrieve resources from the PuppetDB at puppet:8140: Failed to execute '/pdb/query/v4/resourc
es?query=%5B%22and%22%2C%5B%22%3D%22%2C%22type%22%2C%22Sshkey%22%5D%2C%5B%22%3D%22%2C%22exported%22%2Ctrue%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22certname%22%2C%22puppet.example.com%22%5D%5D%5D' on at least 1 of the following 'server_urls': https://puppet.example.com:8081 on node puppet.example.com"]
D, [2018-07-31T11:10:31.081089 #22278] DEBUG -- : ["STDERR: Error: Failed to compile catalog for node puppet.example.com: Could not retrieve resources from the PuppetD
B at puppet:8140: Failed to execute '/pdb/query/v4/resources?query=%5B%22and%22%2C%5B%22%3D%22%2C%22type%22%2C%22Sshkey%22%5D%2C%5B%22%3D%22%2C%22exported%22%2Ctrue%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22certname%22%2C%22puppet.example.com%22%5D%5D%5D' on at least 1 of the following 'server_urls': https://puppet.example.com:8081 on node puppet.example.com"]
D, [2018-07-31T11:10:31.081124 #22278] DEBUG -- : ["Exit status: 30"]
W, [2018-07-31T11:10:31.081321 #22278]  WARN -- : Puppet command failed: STDOUT:

# STDERR repeats

D, [2018-07-31T11:10:31.081532 #22278] DEBUG -- : Catalog for . failed with OctocatalogDiff::Catalog::Computed in 20.214317574 seconds
W, [2018-07-31T11:10:31.081666 #22278]  WARN -- : Failed build_catalog for . validation: OctocatalogDiff::Errors::CatalogError Catalog failed:

# repeating deprecation warnings and error messages omitted

        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:39:in `call'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:39:in `validate'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:202:in `execute_task'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:119:in `block (2 levels) in run_tasks_parallel'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:117:in `fork'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:117:in `block in run_tasks_parallel'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:114:in `each'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:114:in `each_with_index'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:114:in `run_tasks_parallel'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/parallel.rb:94:in `run_tasks'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/catalogs.rb:92:in `build_catalog_parallelizer'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/util/catalogs.rb:29:in `catalogs'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/api/v1/catalog-diff.rb:34:in `catalog_diff'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/api/v1.rb:19:in `catalog_diff'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/lib/octocatalog-diff/cli.rb:119:in `cli'
        from /var/lib/gems/2.3.0/gems/octocatalog-diff-1.5.3/bin/octocatalog-diff:34:in `<top (required)>'
        from /usr/local/bin/octocatalog-diff:23:in `load'
        from /usr/local/bin/octocatalog-diff:23:in `<main>'

Platform and version information

• Your OS: Ubuntu 16.04 on all nodes
• Your Ruby version: ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu] from Ubuntu 16.04
• Your version of Puppet: 4.10.12 from PC1 (open source)
• Your version of octocatalog-diff: 1.5.3 installed with gem install octocatalog-diff

Do the tests pass from a clean checkout?

yes

[fheinle]

The problem appears to lie in the file names used by ssl client cert and private key used for authenticating against puppetdb. While using the content of the certs in .octocatalog-diff.cfg.rb works fine for using those credentials for gathering facts etc, running puppet master --compile later fails because it can't pick up the files. The files are stored with just the host name as their file names and not the fqdn, so puppet won't pick it up automatically. c.f.

octocatalog-diff/lib/octocatalog-diff/catalog-util/builddir.rb

Line 370 in 4860fcc

host = Socket.gethostname

Replacing this line with host = Socket.gethostbyname(Socket.gethostname).first fixed the issue by resolving the hostname to its FQDN, requires working DNS on the client, though. host = `hostname -f` works under unixoid systems, etc.

[MasterMind2k]

Is there any work-around (without applying the patch) for this?

[MasterMind2k]

Not sure if this is part of this issue, or should open a new issue.

I am using puppet's CA.

Running with puppet 6 I also encountered that puppet is missing CRL. I've managed to get it installed with the following hack (inside enc_wrapper script):

# Do you ENC thingy

# Need to manually install missing CRL file
cp /etc/puppetlabs/puppet/ssl/crl.pem $(readlink -f ../../../*builddir*)/var/ssl

Hope this helps.