Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

litmus does not see bolt inventory plugins #411

Open
shaun-rutherford opened this issue Apr 19, 2021 · 4 comments
Open

litmus does not see bolt inventory plugins #411

shaun-rutherford opened this issue Apr 19, 2021 · 4 comments
Labels
bug community

Comments

@shaun-rutherford
Copy link

Describe the bug
Greetings,
It appears the PDK does not support bolts various inventory plugins for use with puppet_litmus.

I made sure the bolt and puppet_litmus gem PDK uses is the latest available and tried installing the puppetlabs-terraform bolt module via PDK, still fails with the following:

pdk bundle exec bolt module add puppetlabs-terraform
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.21.1
Bolt might be installed as a gem. To use Bolt reliably and with all of its
dependencies, uninstall the 'bolt' gem and install Bolt as a package:
https://puppet.com/docs/bolt/latest/bolt_installing.html

If you meant to install Bolt as a gem and want to disable this warning,
set the BOLT_GEM environment variable. [ID: gem_install]
Project configuration file /home/user/.puppetlabs/bolt/bolt-project.yaml already includes specification with name puppetlabs-terraform. Nothing to do.

pdk bundle exec rake litmus:install_agent
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.21.1
install_agent
rake aborted!
Bolt::Plugin::PluginError::Unknown: Unknown plugin: 'terraform'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:231:in `get_hook'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:311:in `resolve_single_reference'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:289:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `block in resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `each'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `flat_map'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:52:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `block in initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `map'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `create_version'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:170:in `inventory'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:195:in `run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:26:in `block in run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:152:in `block in with_runner'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:150:in `with_runner'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:25:in `run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_helper.rb:225:in `block in install_agent'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_helper.rb:210:in `install_agent'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_tasks.rb:126:in `block (2 levels) in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `<main>'
Tasks: TOP => litmus:install_agent
(See full trace by running task with --trace)

To Reproduce
Download puppetlabs-motd
Add spec/fixtures/litmus_inventory.yaml with terraform plugin data:

groups:
  - name: cloud-webs
    targets:
      - _plugin: terraform
        dir: /home/srutherford/Documents/Git/Terraform/gcp_instance_test
        resource_type: google_compute_instance.default
        target_mapping:
          uri: network_interface.0.network_ip

Run pdk bundle install
Run pdk bundle exec rake litmus:install_agent
Run pdk bundle exec rake litmus:install_module

Expected behavior
The PDK should be able to run the bolt commands without error.

Additional context

  • Your PDK installation method
    installed via native package on CentOS8

  • Your PDK version (pdk --version)
    pdk --version
    2.1.0

  • Your operating system / platform
    CentOS 8

I'm able to run commands using the Bolt I've installed from Puppetlabs CentOS repository using the terraform plugin:

bolt task run puppet_agent::install -i spec/fixtures/litmus_inventory.yaml --targets cloud-webs --run-as root
CLI arguments ["run-as"] might be overridden by Inventory: /home/srutherford/Documents/Git/Puppet/puppetlabs-motd/spec/fixtures/litmus_inventory.yaml [ID: cli_overrides]
Started on 10.180.23.173...
Finished on 10.180.23.173:
  19:33:54 +0000 INFO: Version parameter not defined and agent detected. Nothing to do.
Successful on 1 target: 10.180.23.173
Ran on 1 target in 1.55 sec
@jpogran
Copy link

jpogran commented Apr 20, 2021

@shaun-rutherford I think there are a couple of different things going on here.

Reading through https://puppet.com/docs/bolt/latest/bolt_installing_modules.html, I think you're missing a step in your commands.

bolt module add <module_name>
bolt module install

That being said, this issue is about using litmus and bolt. PDK is just the thing that executes those tools. If we can't figure it out here I'll move this over to the litmus repository

@shaun-rutherford
Copy link
Author

shaun-rutherford commented Apr 20, 2021 •
edited

@jpogran Thanks for the quick reply.

I ran the bold module add terraform before but missed the install bit. Unfortunately doesn't look that solved it either :(

bolt module install
Installing project modules

  → Syncing modules from /home/srutherford/.puppetlabs/bolt/Puppetfile to
    /home/srutherford/.puppetlabs/bolt/.modules

  → Generating type references

Successfully synced modules from /home/srutherford/.puppetlabs/bolt/Puppetfile to /home/srutherford/.puppetlabs/bolt/.modules

ls /home/srutherford/.puppetlabs/bolt/.modules/terraform/
bolt_plugin.json  CHANGELOG.md  CODE_OF_CONDUCT.md  CODEOWNERS  CONTRIBUTING.md  Gemfile  lib  LICENSE  metadata.json  plans  Rakefile  README.md  spec  tasks

pdk bundle exec rake litmus:install_module
pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.5.0
Building '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd' into '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd/pkg'
Built '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd/pkg/puppetlabs-motd-5.0.0.tar.gz'
rake aborted!
Bolt::Plugin::PluginError::Unknown: Unknown plugin: 'terraform'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:231:in `get_hook'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:311:in `resolve_single_reference'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:289:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `block in resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `each'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `flat_map'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:52:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `block in initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `map'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `create_version'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:170:in `inventory'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:206:in `run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:62:in `block in run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:152:in `block in with_runner'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:150:in `with_runner'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:61:in `run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:351:in `uninstall_module'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:310:in `block in install_module'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:302:in `install_module'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_tasks.rb:209:in `block (2 levels) in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/puppetlabs/pdk/private/ruby/2.7.2/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.7.2/bin/bundle:23:in `<main>'
Tasks: TOP => litmus:install_module
(See full trace by running task with --trace)

@shaun-rutherford
Copy link
Author

Hey @jpogran just checking in to see if you had any more information on this?

@jpogran
Copy link

jpogran commented Apr 26, 2021

Apologies for the delay, was working with the bolt and litmus teams to figure this out. The short of it is that litmus needs to be modified to 'see' the moduledir Bolt uses when running bolt module install, or be modified to wrap bolt commands such that the modules be 'installed' in the correct place and time during a litmus run.

To explain what's going on currently, bolt's bolt module install command writes to your project config file at /home/srutherford/.puppetlabs/bolt/.modules, which bolt recognizes on it's modulepath when it's run on your machine. Litmus doesn't know about that file and instead uses a custom created inventory file and points to its own modulepath (https://github.com/puppetlabs/puppet_litmus/blob/main/lib/puppet_litmus/rake_tasks.rb#L128), which all gets copied over or run directly on the target, depending on which part of the execution is happening.

Linking the two together is a task for the litmus team, so I'm moving this ticket over to that repo and will work with them to get a plan together for this.

@jpogran jpogran transferred this issue from puppetlabs/pdk Apr 26, 2021
@jpogran jpogran changed the title PDK does not support bolt inventory plugins litmus does not see bolt inventory plugins Apr 26, 2021
@pmcmaw pmcmaw added the bug label Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug community
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jpogran @shaun-rutherford @chelnak @pmcmaw