Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

catalog_diff fails to evaluate catalogs with Deferred functions #105

Open
vchepkov opened this issue Sep 24, 2023 · 3 comments
Open

catalog_diff fails to evaluate catalogs with Deferred functions #105

vchepkov opened this issue Sep 24, 2023 · 3 comments

Comments

@vchepkov
Copy link

With puppetlabs/mysql module switching to Deferred function, I noticed that catalog_diff fails to evaluate the catalog. In puppetserver.log I observe the following error:

2023-09-24T14:53:46.975Z ERROR [qtp894367513-7037] [p.r.core] Internal Server Error: org.jruby.exceptions.RuntimeError: (PreformattedError) Evaluation Error: Mysql_user[root@localhost]['password_hash'] contains a Deferred value. It will be converted to the String 'Deferred({'name' => 'mysql::password', 'arguments' => [Instance of Sensitive[String]]})' (file: /etc/puppetlabs/puppetserver/code/environments/apache/modules/mysql/manifests/server/root_password.pp, line: 33)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.issue_reporter.assert_and_report(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/issue_reporter.rb:60)
	at RUBY.accept(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:520)
	at RUBY.accept(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/validation.rb:216)
	at RUBY.optionally_fail(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:44)
	at RUBY.serialization_issue(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:309)
	at RUBY.unknown_to_string_with_warning(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:206)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.unknown_to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:195)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:120)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.with(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:171)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.with_recursive_guard(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:189)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.process(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:158)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:106)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.convert(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:60)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.convert(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:22)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:127)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.catalog.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:485)
	at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2667)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.catalog.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:485)
	at uri_3a_classloader_3a_.puppetserver_minus_lib.puppet.server.compiler.compile_catalog(uri:classloader:/puppetserver-lib/puppet/server/compiler.rb:115)
	at uri_3a_classloader_3a_.puppetserver_minus_lib.puppet.server.compiler.compile(uri:classloader:/puppetserver-lib/puppet/server/compiler.rb:29)
	at uri_3a_classloader_3a_.puppetserver_minus_lib.puppet.server.master.compileCatalog(uri:classloader:/puppetserver-lib/puppet/server/master.rb:101)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.context.override(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.override(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289)
	at uri_3a_classloader_3a_.puppetserver_minus_lib.puppet.server.master.compileCatalog(uri:classloader:/puppetserver-lib/puppet/server/master.rb:100)

puppet 7.26.0
puppetserver 2021.7.5.17
@vchepkov vchepkov mentioned this issue Sep 26, 2023
Closed
@joshcooper
Copy link

The problem is catalog-diff requests a catalog in pson format so puppetserver cannot serialize the catalog in a way that preserves rich data, see

puppet-catalog_diff/lib/puppet/catalog-diff/compilecatalog.rb

Line 78 in 952b5d4

'Accept' => 'pson',

To request a "rich data" catalog you'll need to create the loaders on the client side and specify the rich data catalog mime type. If you're using the catalog v3 endpoint, then you can just use the builtin http client:

require 'puppet'

Puppet.initialize_settings

Puppet::Util::Log.newdestination(:console)
Puppet[:log_level] = :info

env = Puppet::Node::Environment.remote("production")
Puppet.push_context({
  current_environment: env,
  loaders: Puppet::Pops::Loaders.new(env, true),
  rich_data: true
})

# Probably want to load previously saved facts using `Puppet::Node::Facts.from_data_hash` instead
facts = Puppet::Node::Facts.indirection.find(Puppet[:certname])

client = Puppet.runtime[:http]
session = client.create_session
service = session.route_to(:puppet)
response, catalog = service.post_catalog(Puppet[:certname], facts: facts, environment: env.name)
pp catalog.to_data_hash

For example, I see the Regexp data type:

 "resources"=>
  [{"type"=>"Stage", "title"=>"main", "tags"=>["stage"], "exported"=>false, "kind"=>"compilable_type"},
   {"type"=>"Class", "title"=>"Settings", "tags"=>["class", "settings"], "exported"=>false, "kind"=>"unknown"},
   {"type"=>"Class", "title"=>"Main", "tags"=>["class"], "exported"=>false, "kind"=>"unknown", "parameters"=>{"name"=>"main"}},
   {"type"=>"Node", "title"=>"default", "tags"=>["node", "default", "class"], "exported"=>false, "kind"=>"unknown"},
   {"type"=>"Class", "title"=>"Abc", "tags"=>["class", "abc", "node", "default"], "exported"=>false, "kind"=>"unknown"},
   {"type"=>"Class",
    "title"=>"Abc::Bridges",
    "tags"=>["class", "abc::bridges", "abc", "bridges", "node", "default"],
    "exported"=>false,
    "kind"=>"unknown",
    "parameters"=>{"bridgenames"=>{"__ptype"=>"Regexp", "__pvalue"=>"^.*$"}}},

If you need to call the v4 catalog endpoint, then you'll need to pass the correct Accept header, which can be resolved like:

format  = Puppet::Network::FormatHandler.format_for(:rich_data_json)
headers['Accept'] = format.mime

And pass that to

puppet-catalog_diff/lib/puppet/catalog-diff/compilecatalog.rb

Line 103 in 952b5d4

Puppet.runtime[:http].post(uri, body.to_json, headers: headers, options: { ssl_context: ssl_context })

@vchepkov
Copy link
Author

For the reference, and to answer @bastelfreak question, I use the following command options

$PUPPET catalog diff $PT_server/${PT_old_environment} \
  $PT_server/${PT_new_environment} \
  --certless --show_resource_diff --no-filter_old_env \
  --render-as json --log_level warning >$TMPFILE1 2>$TMPFILE2

@vchepkov
Copy link
Author

I tried to patch at least v4 catalog here:

vchepkov@7ab194a

And It didn't help, @joshcooper , what did I miss?

2024-03-14T14:48:16.886Z INFO  [qtp1628880978-39013] [puppetserver] Puppet Compiled catalog for t440.chepkov.lan in environment modules using the v4 catalog endpoint
2024-03-14T14:48:16.897Z ERROR [qtp1628880978-39013] [p.r.core] Internal Server Error: org.jruby.exceptions.RuntimeError: (PreformattedError) Evaluation Error: Class[Nftables::Bridges]['bridgenames'] contains a Regexp value. It will be converted to the String '/^br.+/'
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.issue_reporter.assert_and_report(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/issue_reporter.rb:60)
	at RUBY.accept(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:520)
	at RUBY.accept(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/validation.rb:216)
	at RUBY.optionally_fail(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:44)
	at RUBY.serialization_issue(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:309)
	at RUBY.unknown_to_string_with_warning(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:206)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.unknown_to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:195)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:120)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.with(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:171)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1519)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:109)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.with_recursive_guard(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:189)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.process(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:158)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.to_data(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:106)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.convert(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:60)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.pops.serialization.to_data_converter.convert(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/serialization/to_data_converter.rb:22)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:127)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.catalog.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:485)
	at org.jruby.RubyArray.map(org/jruby/RubyArray.java:2667)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.resource.catalog.to_data_hash(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:485)
	at RUBY.compile_catalog(uri:classloader:/puppetserver-lib/puppet/server/compiler.rb:115)
	at RUBY.compile(uri:classloader:/puppetserver-lib/puppet/server/compiler.rb:29)
	at RUBY.compileCatalog(uri:classloader:/puppetserver-lib/puppet/server/master.rb:101)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.context.override(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62)
	at opt.puppetlabs.puppet.lib.ruby.vendor_ruby.puppet.override(/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289)
	at RUBY.compileCatalog(uri:classloader:/puppetserver-lib/puppet/server/master.rb:100)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joshcooper @vchepkov