Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provision.yaml expansion #393

Open
cdenneen opened this issue Feb 26, 2021 · 5 comments
Open

provision.yaml expansion #393

cdenneen opened this issue Feb 26, 2021 · 5 comments

Comments

@cdenneen
Copy link

Currently you can specify a provision.yaml in order to provision multiple images.
What would be useful is to add a collection key to the inventory of hosts so if you just run rake "litmus:install_agent" it would reference this provision file for a possible collection in order to provide different agent's across the inventory?

Current:

---
centos7-puppet6:
  provisioner: docker
  images:
  - litmusimage/centos:7
centos7-puppet7:
  provisioner: docker
  images:
  - litmusimage/centos:7

Proposal:

centos7-puppet6:
  provisioner: docker
  images:
  - litmusimage/centos:7
  collection: puppet6
centos7-puppet7:
  provisioner: docker
  images:
  - litmusimage/centos:7
  collection: puppet7

Currently in order to accomplish this you must run:
litmus:provision_list[centos7-puppet6] followed by litmus:agent_install[puppet6,localhost:2222]
and then
litmus:provision_list[centos7-puppet6] followed bylitmus:agent_install[puppet7,localhost:2223]
obviously with the target port changing this is difficult from an automation perspective

The proposal would be that by default if no key is specified for litmus:provision_list to loop through them all as well as the same loop when it comes to litmus:install_agent ... currently both of these without keys error:

provision_list:

❯ PDK_PUPPET_VERSION=6 pdk bundle exec rake litmus:provision_list
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.19.1
rake aborted!
No key  in ./provision.yaml, see https://puppetlabs.github.io/litmus/Litmus-core-commands.html#provisioning-via-yaml for examples
/Users/cdenneen/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.24.0/lib/puppet_litmus/rake_tasks.rb:28:in `block (2 levels) in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `<main>'
Tasks: TOP => litmus:provision_list

install_agent: (From the error looks like it should be attempting to install the latest agent but fails as it can't find it)

❯ PDK_PUPPET_VERSION=6 pdk bundle exec rake litmus:install_agent
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.19.1
install_agent
rake aborted!
Installation of agent failed.
Results:
  localhost:2225: {"_output"=>"12:26:47 -0500 INFO: Version parameter not defined and no agent detected. Assuming latest.\n12:26:47 -0500 INFO: Downloading Puppet latest for el...\n12:26:47 -0500 INFO: Red hat like platform! Lets get you an RPM...\n12:26:47 -0500 INFO: Downloading http://yum.puppet.com/puppet-release-el-7.noarch.rpm\n12:26:47 -0500 INFO:   to file /tmp/install.sh.86827.58897/puppet-release-el-7.noarch.rpm\n12:26:47 -0500 INFO: Trying curl...\n12:26:47 -0500 INFO: installing puppetlabs yum repo with rpm...\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: epel.mirror.constant.com\n * extras: mirror.nodespace.net\n * updates: mirror.ash.fastserv.com\nNo package puppet-agent available.\n12:26:49 -0500 INFO: Retrying... [1/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: artifactory.ap.org\n * epel: epel.mirror.constant.com\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:50 -0500 INFO: Return code: 1\n12:26:51 -0500 INFO: Retrying... [2/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: artifactory.ap.org\n * epel: epel.mirror.constant.com\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:52 -0500 INFO: Return code: 1\n12:26:53 -0500 INFO: Retrying... [3/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: artifactory.ap.org\n * epel: mirror.steadfastnet.com\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:54 -0500 INFO: Return code: 1\n12:26:55 -0500 INFO: Retrying... [4/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.vtti.vt.edu\n * epel: d2lzkl7pfhq30w.cloudfront.net\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:56 -0500 INFO: Return code: 1\n12:26:57 -0500 INFO: Retrying... [5/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: artifactory.ap.org\n * epel: mirror.csclub.uwaterloo.ca\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:58 -0500 INFO: Return code: 1\n12:26:59 -0500 CRIT: Installation failed\n", "_error"=>{"kind"=>"puppetlabs.tasks/task-error", "issue_code"=>"TASK_ERROR", "msg"=>"The task failed with exit code 1", "details"=>{"exit_code"=>1}}}
  localhost:2226: {"_output"=>"12:26:47 -0500 INFO: Version parameter not defined and no agent detected. Assuming latest.\n12:26:47 -0500 INFO: Downloading Puppet latest for el...\n12:26:47 -0500 INFO: Red hat like platform! Lets get you an RPM...\n12:26:47 -0500 INFO: Downloading http://yum.puppet.com/puppet-release-el-7.noarch.rpm\n12:26:47 -0500 INFO:   to file /tmp/install.sh.51677.49907/puppet-release-el-7.noarch.rpm\n12:26:47 -0500 INFO: Trying curl...\n12:26:47 -0500 INFO: installing puppetlabs yum repo with rpm...\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.wdc1.us.leaseweb.net\n * epel: ewr.edge.kernel.org\n * extras: artifactory.ap.org\n * updates: packages.oit.ncsu.edu\nNo package puppet-agent available.\n12:26:49 -0500 INFO: Retrying... [1/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: artifactory.ap.org\n * epel: ewr.edge.kernel.org\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:50 -0500 INFO: Return code: 1\n12:26:51 -0500 INFO: Retrying... [2/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.wdc1.us.leaseweb.net\n * epel: ewr.edge.kernel.org\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:52 -0500 INFO: Return code: 1\n12:26:53 -0500 INFO: Retrying... [3/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.wdc1.us.leaseweb.net\n * epel: iad.mirror.rackspace.com\n * extras: artifactory.ap.org\n * updates: packages.oit.ncsu.edu\nNo package puppet-agent available.\n12:26:54 -0500 INFO: Return code: 1\n12:26:55 -0500 INFO: Retrying... [4/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.wdc1.us.leaseweb.net\n * epel: epel.mirror.constant.com\n * extras: artifactory.ap.org\n * updates: artifactory.ap.org\nNo package puppet-agent available.\n12:26:56 -0500 INFO: Return code: 1\n12:26:57 -0500 INFO: Retrying... [5/5]\nLoaded plugins: fastestmirror, ovl\nLoading mirror speeds from cached hostfile\n * base: mirror.wdc1.us.leaseweb.net\n * epel: mirror.math.princeton.edu\n * extras: mirror.metrocast.net\n * updates: packages.oit.ncsu.edu\nNo package puppet-agent available.\n12:26:58 -0500 INFO: Return code: 1\n12:26:59 -0500 CRIT: Installation failed\n", "_error"=>{"kind"=>"puppetlabs.tasks/task-error", "issue_code"=>"TASK_ERROR", "msg"=>"The task failed with exit code 1", "details"=>{"exit_code"=>1}}}}

Think it would also be useful for the target key instead of having to be localhost:222X it could be the list_name so you could do pdk bundle exec rake "litmus:tear_down[centos7-puppet6]" vs pdk bundle exec rake "litmus:tear_down[localhost:2222]"

@DavidS
Copy link
Contributor

DavidS commented Mar 23, 2021

I like the idea in principle.

The problem I see is that there is no good connection from the provision_list invocation to agent_install task.

Using existing mechanisms in litmus, setting the desired puppet collection in the vars could work out. See

inventory_vars = provision_hash[args[:key]]['vars']

vars are key/value pairs that get passed through provisioning and get stored in the inventory file, so the agent_install rake task would be able to pick this up again, and default :collection to that value.

Think it would also be useful for the target key instead of having to be localhost:222X it could be the list_name so you could do pdk bundle exec rake "litmus:tear_down[centos7-puppet6]" vs pdk bundle exec rake "litmus:tear_down[localhost:2222]"

It would be great to use more of bolt's target addressing features and map provision list entries to groups and allow users to set more useful aliases.

@cdenneen
Copy link
Author

@DavidS do you have a working example of how vars could be picked up by agent_install task? Is this wired or would require some wiring. Sounds like a reasonable approach given how litmus is currently written to take advantage of rather than the original proposal.

As far as the tear_down or other tasks sometimes the localhost:222x approach is a bit clunky. Trying to figure out which host is which or even having a left over inventory file accidentally left around and then after provision the host is 2224 but during agent_install it fails because can't connect to 2222, 2223 from the inventory file.

@cdenneen
Copy link
Author

@DavidS also is vars the params mentioned in the litmus documentation?

@cdenneen
Copy link
Author

https://puppetlabs.github.io/litmus/Litmus-core-commands.html#provisioning-via-yaml

---
default:
  provisioner: docker
  images: ['litmusimage/centos:7']
centos7:
  provisioner: docker
  images: ['litmusimage/centos:7']
  params:
    param_a: someone
    param_b: something
  vars:
    var1=value1
    var2=value2

but the resulting inventory shows:

---
version: 2
groups:
- name: docker_nodes
  targets: []
- name: ssh_nodes
  targets:
  - uri: localhost:2227
    config:
      transport: ssh
      ssh:
        user: root
        password: root
        port: 2227
        host-key-check: false
    facts:
      provisioner: docker
      container_name: litmusimage_centos_7-2227
      platform: litmusimage/centos:7
      os-release:
        NAME: CentOS Linux
        VERSION: 7 (Core)
        ID: centos
        ID_LIKE: rhel fedora
        VERSION_ID: '7'
        PRETTY_NAME: CentOS Linux 7 (Core)
        ANSI_COLOR: 0;31
        CPE_NAME: cpe:/o:centos:centos:7
        HOME_URL: https://www.centos.org/
        BUG_REPORT_URL: https://bugs.centos.org/
        CENTOS_MANTISBT_PROJECT: CentOS-7
        CENTOS_MANTISBT_PROJECT_VERSION: '7'
        REDHAT_SUPPORT_PRODUCT: centos
        REDHAT_SUPPORT_PRODUCT_VERSION: '7'
    vars: var1=value1 var2=value2
- name: winrm_nodes
  targets: []

so based on that the docs need to be updated as params do not work?

@DavidS
Copy link
Contributor

DavidS commented Mar 31, 2021

  1. there's no explicit wiring around vars. litmus is all spit and bailing wire, all of the real work is done by bolt.
  2. vars vs params: yeah that seems to be an oversight in the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants