Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k8s module throwing 'This module requires the OpenShift Python client. Try pip install openshift' #50529

Closed
geerlingguy opened this issue Jan 4, 2019 · 50 comments
Labels
affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. clustering Clustering category k8s module This issue/PR relates to a module. support:community This issue/PR relates to code supported by the Ansible community.

Comments

@geerlingguy
Copy link
Contributor

SUMMARY

So, I've already read through all of the following earlier issues:

And I've been debugging this issue for my Pi Dramble project (within a Vagrant environment) locally for a couple days; see Switch to geerlingguy.k8s_manifests role. I think I'm close to a solution, but because this road has been windy, and ridiculously weird, I thought I'd post an issue here, and at least document how I solved it (if I can get it solved, ha!) for the benefit of future searchers... and myself, when I inevitably run into the problem again!

ISSUE TYPE
  • Bug Report (maybe Support Request-ish)
COMPONENT NAME

k8s

ANSIBLE VERSION
ansible 2.7.5
  config file = /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg
  configured module search path = [u'/Users/jgeerling/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15 (default, Jul 17 2018, 19:41:20) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
CONFIGURATION
ANSIBLE_NOCOWS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
ANSIBLE_PIPELINING(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
ANSIBLE_SSH_CONTROL_PATH(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = /tmp/ansible-ssh-%%h-%%p-%%r
DEFAULT_FORKS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = 10
DEFAULT_HOST_LIST(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = [u'/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/inventory']
DEFAULT_LOAD_CALLBACK_PLUGINS(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = True
DEFAULT_ROLES_PATH(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = [u'/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/roles']
DEFAULT_STDOUT_CALLBACK(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = yaml
RETRY_FILES_ENABLED(/Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/testing/vagrant/ansible.cfg) = False
OS / ENVIRONMENT

macOS Mojave (10.14)

STEPS TO REPRODUCE

Playbook:

---
- hosts: localhost
  gather_facts: false
  become: false
  connection: local

  vars_files:
    - vars/main.yml
    - config.yml

  roles:
    - name: geerlingguy.k8s_manifests
      tags: ['kubernetes', 'nfs', 'drupal', 'registry']

Inventory:

[localhost]
127.0.0.1 deploy_target=pi
EXPECTED RESULTS

Kubernetes manifests will be applied to the K8s cluster.

ACTUAL RESULTS
TASK [geerlingguy.k8s_manifests : Deploy the resources defined inside the manifest.] ***
task path: /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/roles/geerlingguy.k8s_manifests/tasks/deploy-manifest.yml:27
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/clustering/k8s/k8s.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgeerling
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/clustering/k8s/k8s.py
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
failed: [127.0.0.1] (item={'rules': [{'apiGroups': [u''], 'verbs': [u'get', u'list', u'watch'], 'resources': [u'services', u'endpoints', u'secrets']}, {'apiGroups': [u'extensions'], 'verbs': [u'get', u'list', u'watch'], 'resources': [u'ingresses']}], 'kind': u'ClusterRole', 'apiVersion': u'rbac.authorization.k8s.io/v1beta1', 'metadata': {'name': u'traefik-ingress-controller'}}) => changed=false 
  invocation:
    module_args:
      api_key: null
      api_version: v1
      cert_file: null
      context: null
      definition:
        apiVersion: rbac.authorization.k8s.io/v1beta1
        kind: ClusterRole
        metadata:
          name: traefik-ingress-controller
      ...
      verbs:
      - get
      - list
      - watch
  msg: This module requires the OpenShift Python client. Try `pip install openshift`

First thing I tried was verifying openshift was installed in the python being used:

$ /usr/local/opt/python@2/bin/python2.7 -m pip freeze | grep openshift
openshift==0.8.0

(I also tried everything with openshift==0.8.1 just to rule out version issues.).

That's odd.

So then I added a debug statement to the playbook:

  pre_tasks:
    - python_requirements_facts:
        dependencies: openshift

And the output:

TASK [python_requirements_facts] ***********************************************
task path: /Users/jgeerling/Dropbox/Development/GitHub/raspberry-pi-dramble/test.yml:12
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/system/python_requirements_facts.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: jgeerling
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 && sleep 0'
ok: [127.0.0.1] => changed=false 
  invocation:
    module_args:
      dependencies:
      - openshift
  mismatched: {}
  not_found: []
  python: /usr/local/opt/python@2/bin/python2.7
  python_system_path:
  - /var/folders/mt/rp7cb3s95qzfkrh1dk_8ckfc0000gn/T/ansible_python_requirements_facts_payload_JLohA7/ansible_python_requirements_facts_payload.zip
  - ''
  - /usr/local/lib/python2.7/site-packages
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
  - /usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
  - /Users/jgeerling/Library/Python/2.7/lib/python/site-packages
  - /usr/local/lib/python2.7/site-packages
  python_version: |-
    2.7.15 (default, Jul 17 2018, 19:41:20)
    [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
  valid:
    openshift:
      desired: null
      installed: 0.8.0

So... what gives?

@ansibot
Copy link
Contributor

ansibot commented Jan 4, 2019

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Jan 4, 2019

@ansibot ansibot added affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. k8s module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. support:community This issue/PR relates to code supported by the Ansible community. labels Jan 4, 2019
@geerlingguy
Copy link
Contributor Author

Created a test python script:

try:
    import kubernetes
    import openshift
    from openshift.dynamic import DynamicClient
    from openshift.dynamic.exceptions import ResourceNotFoundError, ResourceNotUniqueError
    print "Found it"
except ImportError:
    print "Didnt find it"

Ran it:

$ /usr/local/opt/python@2/bin/python2.7 test.py 
/usr/local/lib/python2.7/site-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Didnt find it

So... looks like there's some weird error with urllib3 and/or chardet, which tosses an exception, even though openshift and kubernetes are available. And so that fails Ansible's check, but that exception is completely gobbled up by Ansible and there's no way for me to tell that without reproducing the test code and finding the exception on my own.

Is there any way to bubble up the actual exception instead of just blindly setting HAS_K8S_MODULE_HELPER = False (which is not quite true, since openshift is verifiably present)?

@geerlingguy
Copy link
Contributor Author

geerlingguy commented Jan 4, 2019

So in my case, it was an annoying Requests-related exception (actually just a RequestsDependencyWarning, but somehow it was flagging it as an exception?), and I fixed it with:

$ pip install --upgrade requests

(For context, see psf/requests#4673)

Now everything is working happily.

But I want to leave this open in case it's deemed a bug that the k8s module doesn't bubble up exceptions and has (IMO) a little bit of a fragile check for whether openshift is actually installed (or maybe that error message could be made a little less strict—e.g. 'it could be that openshift is not installed').

@fabianvf
Copy link
Contributor

fabianvf commented Jan 4, 2019

@geerlingguy thanks a ton for the investigation, that's definitely a weird one. I'll try and get that error properly bubbled up.

@ansibot ansibot removed the needs_triage Needs a first human triage before being processed. label Jan 4, 2019
@ClaudioDeFacci
Copy link

ClaudioDeFacci commented Jan 6, 2019

Same here:

Ansible 2.7.5
Python 2.7.15rc1
Openshift 0.8.2

I am simply using the k8s module and get the above error.
Last suggestion of geerlingguy (upgrading requests) did not help.
Hope this is no bigger issue.
At this point I am pretty upset as I do not know Python at all and am unable to contribute to a solution on this.
I would appreciate any help to get this solved.
Thank you!

@ghost
Copy link

ghost commented Jan 8, 2019

+1

@ghost
Copy link

ghost commented Jan 8, 2019

Tested in 2.6.11 version also ... same error and same issue.

kylix3511  (e) venv260  ~  k8s  learning  cat ansible2.yaml 
---
- hosts: master 
  connection: local

  tasks:
    - name: Create a kubernetes namespace
      k8s_raw:
        name: testing
        api_version: v1
        kind: Namespace
        state: present
 kylix3511  (e) venv260  ~  k8s  learning 

Error Message :



 kylix3511  (e) venv260  ~  k8s  learning  ansible-playbook ansible2.yaml -vvv
ansible-playbook 2.6.11
  config file = /Users/kylix3511/k8s/learning/ansible.cfg
  configured module search path = [u'/Users/kylix3511/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible
  executable location = /Users/kylix3511/venv260/bin/ansible-playbook
  python version = 2.7.10 (default, Aug 17 2018, 19:45:58) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
Using /Users/kylix3511/k8s/learning/ansible.cfg as config file
Parsed /Users/kylix3511/k8s/learning/hosts inventory source with ini plugin

PLAYBOOK: ansible2.yaml ************************************************************************************************************************************
1 plays in ansible2.yaml

PLAY [master] **********************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************
task path: /Users/kylix3511/k8s/learning/ansible2.yaml:2
<master> ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
<master> EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
<master> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666 `" && echo ansible-tmp-1546978639.55-171007548805666="` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666 `" ) && sleep 0'
Using module file /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible/modules/system/setup.py
<master> PUT /Users/kylix3511/.ansible/tmp/ansible-local-2599671Udp6/tmpnqhyGB TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py
<master> EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py && sleep 0'
<master> EXEC /bin/sh -c '/usr/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/setup.py && sleep 0'
<master> EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978639.55-171007548805666/ > /dev/null 2>&1 && sleep 0'
ok: [master]
META: ran handlers

TASK [Create a kubernetes namespace] ***********************************************************************************************************************
task path: /Users/kylix3511/k8s/learning/ansible2.yaml:6
<master> ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
<master> EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
<master> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319 `" && echo ansible-tmp-1546978640.38-224769996401319="` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319 `" ) && sleep 0'
Using module file /Users/kylix3511/venv260/lib/python2.7/site-packages/ansible/modules/clustering/k8s/_k8s_raw.py
<master> PUT /Users/kylix3511/.ansible/tmp/ansible-local-2599671Udp6/tmpOdDMKi TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py
<master> EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py && sleep 0'
<master> EXEC /bin/sh -c '/usr/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/_k8s_raw.py && sleep 0'
<master> EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1546978640.38-224769996401319/ > /dev/null 2>&1 && sleep 0'
fatal: [master]: FAILED! => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "api_key": null, 
            "api_version": "v1", 
            "cert_file": null, 
            "context": null, 
            "force": false, 
            "host": null, 
            "key_file": null, 
            "kind": "Namespace", 
            "kubeconfig": null, 
            "name": "testing", 
            "namespace": null, 
            "password": null, 
            "resource_definition": null, 
            "src": null, 
            "ssl_ca_cert": null, 
            "state": "present", 
            "username": null, 
            "verify_ssl": null
        }
    }, 
    "msg": "This module requires the OpenShift Python client. Try `pip install openshift`"
}

PLAY RECAP *************************************************************************************************************************************************
master                     : ok=1    changed=0    unreachable=0    failed=1   

 kylix3511  (e) venv260  ~  k8s  learning  

@fabianvf
Copy link
Contributor

fabianvf commented Jan 8, 2019

If you have openshift installed in a virtual environment, you may need to specify the proper python intepreter for Ansible to use (by default it will use your system python). You can do that by setting the ansible_python_interpreter variable to ansible_playbook_python. Your new play would look like:

- hosts: master 
  connection: local
  vars:
    ansible_python_interpreter: '{{ ansible_playbook_python }}'
  tasks:
    - name: Create a kubernetes namespace
      k8s_raw:
        name: testing
        api_version: v1
        kind: Namespace
        state: present

Does that fix your issue @kylix3511 or @ClaudioDeFacci?

Edit: Including Will Thames' excellent blog post on this subject: https://willthames.github.io/2018/07/01/connection-local-vs-delegate_to-localhost.html

@ghost
Copy link

ghost commented Jan 8, 2019 via email

@fabianvf
Copy link
Contributor

fabianvf commented Jan 8, 2019

No, I haven't used AWX very extensively so I can't really be of much help there. I'd recommend asking in IRC.

@geerlingguy
Copy link
Contributor Author

geerlingguy commented Jan 9, 2019

Just hit this again today on my other Mac, after I ran a brew upgrade and the brew python was relinked. D'oh! Luckily I googled and found this issue pretty quickly :)

On this computer, the problem was that the system Python (/usr/bin/python) was being used for Ansible, even though my current environment was using the Brew-managed Python (/usr/local/bin/python).

So I ran:

$ /usr/bin/python -m pip install --upgrade --user openshift

And now I'm back in business... after also updating kubernetes to 8.0.1.

@benwsapp
Copy link

@kylix3511 we got this working in AWX by setting up a custom virtual environment, like here so you have the proper pip dependencies installed, and by setting the "ansible_python_interpreter" with an extra var in the job template in AWX. then, locally just setting the "ansible_python_interpreter as global group_var to "ansible_playbook_python" as @fabianvf had said.

it seems like it might be a bug within the AWX custom environment feature. like the error checking for the playbook isn't using the custom venv or something.

@willthames
Copy link
Contributor

@geerlingguy - rather than updating system python, it's better to set ansible_python_interpreter to ansible_playbook_python as @fabianvf suggests.

Also, for anyone reporting these issues, it's hugely helpful to put a python_requirements_facts task in your playbook so that you can see what is actually being run.

For example:

- python_requirements_facts:
    dependencies:
      - openshift
      - requests

would be helpful here

@ghost
Copy link

ghost commented Feb 3, 2019 via email

@benwsapp
Copy link

benwsapp commented Feb 3, 2019

Thanks Will. python_requriments_facts is very useful tip. Thanks so much. I try to use python_requriments_facts, it is not working as expected. Please help if you got some time. kylix3511  (e) venv280  ~  ansible  helloWorldApp  cat deployHelloWorld.yaml

--- - hosts: localhost connection: local vars: ansible_python_interpreter: '{{ ansible_playbook_python }}' - python_requirements_facts: dependencies: - openshift - requests tasks: - name: Deploying the simple HelloWorld App in Kylix3511 Namespace k8s: state: present definition: apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" labels: run: myhelloworld10 name: myhelloworld10 namespace: “kylix3511" spec: replicas: 5 selector: matchLabels: run: myhelloworld10 template: metadata: labels: run: myhelloworld10 spec: containers: - image: hub.docker.com/images/myhelloworldapp10:v5 imagePullPolicy: Always name: myhelloworld10
On Feb 2, 2019, at 4:10 PM, Will Thames @.***> wrote: @geerlingguy https://github.com/geerlingguy - rather than updating system python, it's better to set ansible_python_interpreter to ansible_playbook_python as @fabianvf https://github.com/fabianvf suggests. Also, for anyone reporting these issues, it's hugely helpful to put a python_requirements_facts task in your playbook so that you can see what is actually being run. For example: - python_requirements_facts: dependencies: - openshift - requests would be helpful here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#50529 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/Ak2Op2cUxqRbLmZghYlMDWXR5AlBQqDWks5vJikCgaJpZM4ZpCBb.

looks like you have a typo there? - here's the reference for proper syntax.

otherwise, are you getting output at all? or just an error? should show what requirements are and are not being met.

@ghost
Copy link

ghost commented Feb 3, 2019

Thanks @benwsapp. Yes. I did try that ... still same error.

ansible-playbook deployHelloWorld.yaml -vvvvv
ansible-playbook 2.8.0.dev0
  config file = None
  configured module search path = [u'/Users/kylix3511/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/kylix3511/venv280/lib/python2.7/site-packages/ansible
  executable location = /Users/kylix3511/venv280/bin/ansible-playbook
  python version = 2.7.10 (default, Aug 17 2018, 19:45:58) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]
No config file found; using defaults
setting up inventory plugins
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not
match 'all'

ERROR! Syntax Error while loading YAML.
  expected <block end>, but found '<block mapping start>'

The error appears to have been in '/Users/kylix3511/ansible/helloWorldApp/deployHelloWorld.yaml': line 10, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


    tasks:
    ^ here




```

@willthames
Copy link
Contributor

@kylix3511 - the python_requirements_facts is a task, so needs to be in the task list:



    tasks:
    - python_requirements_facts:
        dependencies:
          - openshift
          - requests

    - name: Deploying the simple HelloWorld App in Kylix3511 Namespace
      k8s:
        state: present
        ...

@ghost
Copy link

ghost commented Feb 3, 2019

Thanks @willthames

It works as expected.



TASK [python_requirements_facts] ***********************************************************************************************************************************************************************************************************
task path: /Users/kylix3511/ansible/helloWorldApp/deployHelloWorld.yaml:11
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: kylix3511
<127.0.0.1> EXEC /bin/sh -c 'echo ~kylix3511 && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430 `" && echo ansible-tmp-1549176705.49-269941163809430="` echo /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430 `" ) && sleep 0'
Using module file /Users/kylix3511/venv280/lib/python2.7/site-packages/ansible/modules/system/python_requirements_facts.py
<127.0.0.1> PUT /Users/kylix3511/.ansible/tmp/ansible-local-45759DkpzFE/tmpzttzsf TO /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/ /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/kylix3511/venv280/bin/python /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/AnsiballZ_python_requirements_facts.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/kylix3511/.ansible/tmp/ansible-tmp-1549176705.49-269941163809430/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "dependencies": [
                "openshift", 
                "requests"
            ]
        }
    }, 
    "mismatched": {}, 
    "not_found": [], 
    "python": "/Users/kylix3511/venv280/bin/python", 
    "python_system_path": [
        "/var/folders/yq/064210y10wz31j_rhz_zp8p40000gn/T/ansible_python_requirements_facts_payload_MfhH23/ansible_python_requirements_facts_payload.zip", 
        "/Users/kylix3511/venv280/lib/python27.zip", 
        "/Users/kylix3511/venv280/lib/python2.7", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-darwin", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-mac", 
        "/Users/kylix3511/venv280/lib/python2.7/plat-mac/lib-scriptpackages", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-tk", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-old", 
        "/Users/kylix3511/venv280/lib/python2.7/lib-dynload", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac", 
        "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages", 
        "/Users/kylix3511/venv280/lib/python2.7/site-packages"
    ], 
    "python_version": "2.7.10 (default, Aug 17 2018, 19:45:58) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]", 
    "valid": {
        "openshift": {
            "desired": null, 
            "installed": "0.8.2"
        }, 
        "requests": {
            "desired": null, 
            "installed": "2.20.1"
        }
    }
}




@ghost
Copy link

ghost commented Feb 3, 2019

Did any body tried and deployed a simple HelloWorld application in Kubernetes via ansible ?? I'm trying to do that last few days ...

@fabianvf
Copy link
Contributor

@Asgoret glad you solved it, just fyi I believe the delegate_to: localhost you had in there made the ansible_python_interpreter variable not matter anymore, so it was still running against your system Python at that point.

@Asgoret
Copy link

Asgoret commented Feb 12, 2019

@fabianvf I test simple construction (only hosts: localhost) and it works) so the problem was in some old packages. Is there any opportunity that this behavior will have another error message except pip install openshift?

@willthames
Copy link
Contributor

@Asgoret newer versions of the k8s module have slightly improved error messages showing python path etc.

@ansibot ansibot added the clustering Clustering category label Mar 8, 2019
@dglinder
Copy link
Contributor

@kylix3511 wrote:

For some reason, the same thing is not working in Ansible AWX, I’m creaking my head on it … still no luck.

Same here, RedHat 7.6, ansible 2.7.7, Python 2.7.5, except Tower 3.3.4 and using a custom Ansible Environment that has been setup just for this task (See playbook below) - the Tower documentation and RH support recommend this.

When we specify the custom environment via Tower/AWX we still get the error. The output from python_requirements_facts: for Towers "/var/lib/awx/venv/Kubernetes/" venv shows output like this:

{
    "_ansible_parsed": true,
    "python_version": "2.7.5 (default, Sep 12 2018, 05:31:16) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]",
    "_ansible_delegated_vars": {
        "ansible_delegated_host": "localhost",
        "ansible_host": "localhost"
    },
    "python": "/usr/bin/python2",
    "changed": false,
    "python_system_path": [
        "/tmp/ansible_python_requirements_facts_payload_n4PpL5/ansible_python_requirements_facts_payload.zip",
        "/var/lib/awx/venv/Kubernetes/lib/python2.7/site-packages",
        "/var/lib/awx/projects/_551__dglinder_test_k8s/KubernetesPlaybooks",
        "/usr/lib64/python27.zip",
        "/usr/lib64/python2.7",
        "/usr/lib64/python2.7/plat-linux2",
        "/usr/lib64/python2.7/lib-tk",
        "/usr/lib64/python2.7/lib-old",
        "/usr/lib64/python2.7/lib-dynload",
        "/usr/lib64/python2.7/site-packages",
        "/usr/lib64/python2.7/site-packages/gtk-2.0",
        "/usr/lib/python2.7/site-packages"
    ],
    "valid": {
        "requests": {
            "desired": null,
            "installed": "2.21.0"
        },
        "k8s": {
            "desired": null,
            "installed": "0.11.0"
        },
        "openshift": {
            "desired": null,
            "installed": "0.8.7"
        }
    },
    "mismatched": {},
    "invocation": {
        "module_args": {
            "dependencies": [
                "openshift",
                "requests",
                "k8s"
            ]
        }
    },
    "not_found": [],
    "_ansible_no_log": false
}

The Tower virtual environment is setup on all Tower nodes using this playbook:

#!/usr/bin/env ansible-playbook
---
- hosts: all
  gather_facts: false

  vars:
    venvs:
      # Setup the Kubernetes virtual environment
      - dirname: Kubernetes
        modules:
          - k8s
          - openshift
          - requests
        prereqs:
          - gcc


  tasks:
  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: latest

  - name: "Install pip module required packages"
    yum:
      name: "{{ item.prereqs }}"
      state: latest
    loop: "{{ venvs }}"
    when: item.prereqs is defined

  - name: "Create the venv skeleton directory"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: directory
    become: true
    loop: "{{ venvs }}"

  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
    loop: "{{ venvs }}"

  - name: "Install requested packages in virtual env"
    pip:
      name: "{{ item.modules }}"
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
    loop: "{{ venvs }}"

@ghost
Copy link

ghost commented Mar 28, 2019

+1

@mukeshmurali
Copy link

My problem was resolved with

pip install openshift==0.8.4 --ignore-installed

Havent worked out whats happening behind scenes :(

@dglinder
Copy link
Contributor

dglinder commented Apr 1, 2019

@mukeshmurali was your fix for the Tower/AWX installation, or another command line environment fix? I tried your --ignore-installed option on my Tower virtual environment (see above) but it didn't resolve things for us.

@dglinder
Copy link
Contributor

dglinder commented Apr 2, 2019

Two questions for those more knowledgeable in "pip" and Python modules:
1: Instead of running pip install openshift in the venv, how about running it in the main OS environment? Will those modules then be available to the Tower execution environment?
2: How is the check for openshift or other modules done? Is it possible that some Tower code has a hard-coded path that ignores the venv?

P.S. I'm still learning about Python and 'pip' so excuse me in advance if those questions are bad.

@willthames
Copy link
Contributor

Is ansible/awx#2462 (comment) helpful here?

@dglinder
Copy link
Contributor

dglinder commented Apr 3, 2019

So for those who have struggled with this, my Ansible Tower clusters now have a working solution too. Basically we:

  1. Use pip to install the openshift Python module into the OS environment
    • I am pretty confident that this was the main solution but I haven't bisected the changes to validate this.
  2. Setup the AWX / Ansible Tower virtual environment in /var/lib/awx/venv/Kubernetes
  3. Add the Tower required base packages (python-memcached and psutil)
  4. Add the k8s, openshift, and requests packages.

Pip package installation used the --ignore-installed option and used the site packages as well so the openshift module in the first step (above) was visible. (At least thats what I believe it happening.)

This leads me to believe that there's some "leakage" of the virtual environments within the Tower system that are not properly sandboxing the k8s module within the venv built with the openshift module. But I'm too much of a Python novice to successfully track that down.

I hope this helps others, if only as a band-aid until the real solution is found.

Below is the Ansible playbook I wrote to set this up on my Tower servers:

#!/usr/bin/env ansible-playbook
# Installs the company specific virtual environments for Python properly
# into the Tower environment following the Ansible Tower environment notes:
# URL: https://docs.ansible.com/ansible-tower/3.3.4/html/upgrade-migration-guide/virtualenv.html
# 
# NOTE: Requires the EPEL repository to install the python2-pip package.
#
# Usage:
#   $ sudo ./setup_venvs.yml -i inventory.sand_swn01.ini
#
---
- hosts:
    - tower
    - playground
  gather_facts: false

  vars:
    venvs:
      # Setup the Kubernetes virtual environment
      - dirname: Kubernetes
        modules:
          - k8s
          - openshift
          - requests
        prereqs:
          - gcc

  tasks:
  - name: "Ensure pip is installed"
    # NOTE: This is from the EPEL repository.
    yum:
      name: python2-pip
      state: latest

  - name: "Ensure pip has latest tools"
    # Notes taken from the Ansible container page:
    # https://docs.ansible.com/ansible-container/installation.html
    pip:
      name:
        - setuptools
      extra_args: --upgrade

  - name: "Install packages OUTSIDE Tower virtual environments"
    # Necessary to make 'openshift' for the 'k8s' module.
    pip:
      name:
        - openshift
      extra_args: --ignore-installed

  - name: "Install Python virtual environment packages"
    yum:
      name:
        - python-virtualenv
      state: latest

  - name: "Install pip module required packages"
    yum:
      name: "{{ item.prereqs }}"
      state: latest
    loop: "{{ venvs }}"
    when: item.prereqs is defined

  - name: "Clean out old venv directories"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: absent
    become: true
    loop: "{{ venvs }}"

  - name: "Create the venv skeleton directory"
    file:
      path: "/var/lib/awx/venv/{{ item.dirname }}"
      state: directory
    become: true
    loop: "{{ venvs }}"

  - name: "Install base Tower packages"
    pip:
      name:
        - python-memcached
        - psutil
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed
    loop: "{{ venvs }}"

  - name: "Install requested packages in virtual env"
    pip:
      name: "{{ item.modules }}"
      umask: "0022"
      state: latest
      virtualenv: "/var/lib/awx/venv/{{ item.dirname }}"
      virtualenv_site_packages: yes
      extra_args: --ignore-installed
    loop: "{{ venvs }}"

Edit 1 - Update playbook to show the "pip update setuptools" step.

@dglinder
Copy link
Contributor

dglinder commented Apr 4, 2019

So, the fix here was to add these Python modules to the underlying core OS package "World" and not just to the Tower virtual environments. Is this a problem in the way Python and/or the "k8s" module checks for pre-requisites, or is it a AWX pre-requisite checking failure?

@fabianvf
Copy link
Contributor

fabianvf commented Apr 4, 2019

As far as I know the k8s module isn't doing anything different with module discovery/loading, so I would guess the issue is something to do with Tower + virtualenv + ansible python interpreter discovery.

@mitchellmaler
Copy link

mitchellmaler commented Apr 4, 2019

I was having this issue on a Mac using the homebrew version. My fix was to remove the homebrew version of Ansible and install using pip.
brew remove ansible && pip3 install --user ansible

@dglinder
Copy link
Contributor

dglinder commented Apr 7, 2019

As far as I know the k8s module isn't doing anything different with module discovery/loading, so I would guess the issue is something to do with Tower + virtualenv + ansible python interpreter discovery.

I'm suspecting that too, but my support engineer at RedHat says that since the "k8s" module is not part of the core that RedHat provides, his options are limited. I guess I'll take my work-around for now and hope that future modules the teams need don't require a conflicting module.

If anyone has any pointers on where to start debugging the version checking within the virtualenv I'm all ears.

@willthames
Copy link
Contributor

python_requirements_facts is really the only thing we have to see what is going on within a virtualenv - but that will only tell you that something is wrong, not why.

There does seem to be a parallel AWX issue for this (ansible/awx#2462) - this problem seems to be more about AWX/Tower than the k8s module, so perhaps it's worth following up there?

@dglinder
Copy link
Contributor

dglinder commented Apr 8, 2019

There does seem to be a parallel AWX issue for this (ansible/awx#2462) - this problem seems to be more about AWX/Tower than the k8s module, so perhaps it's worth following up there?

Thanks @willthames that does seem quite familiar.

@OJFord
Copy link

OJFord commented May 5, 2019

Is there even a working workaround for this issue?

I'm not (knowingly, I've never heard of it) using AWX Tower, I'm just running ansible on a Mac against a Linux remote. Initially I was running brew ansible in a pyenv virtual environment with openshift installed, but following above comments I've tried installing both ansible and openshift to system python environment, and that ansible still gives this error.

The error doesn't make it clear if openshift should be installed on local or remote, so before I found this issue I was adding tasks to install pip and openshift on the remote, which hasn't helped either.

@willthames
Copy link
Contributor

@OJFord - usually it's almost always that if you're using connection: local, you need to set ansible_python_interpreter: {{ ansible_playbook_python }}

Openshift should be installed on whatever host ansible is running on.

As discussed earlier in the thread, you can use python_requirements_facts to ensure that you're running the version of python you think you are and the versions of openshift and kubernetes libraries that you think you are

@OJFord
Copy link

OJFord commented May 8, 2019

@willthames Thanks - so k8s[_*] modules should be run with connection: local? Initially I wasn't doing that, before flailing around 'trying to try' everything in this thread, but I suppose it does make most sense.

I still can't make it work though.
hosts:

[local]
localhost ansible_python_interpreter="{{ ansible_playbook_python }}"

python_requirements_facts:

{"valid": {"openshift": {"desired": null, "installed": "0.8.8"}, "requests": {"desired": null, "installed": "2.21.0"}, ...}

and shows the python from pyenv as expected.

/that/path/to/python -m pip install openshift confirms it's installed.

k8s_facts with connection: local still gives:

{"changed": false, "msg": "This module requires the OpenShift Python client. Try `pip install openshift`"}

@willthames
Copy link
Contributor

@OJFord can you run the playbook with -vvv - that should show what python is being run and whether it's the one you expect.

@OJFord
Copy link

OJFord commented May 9, 2019

@willthames That actually shows it's using /usr/bin/python instead of pyenv's - but I previously installed openshift to that python's environment too, and just double-checked that.

It runs:

[IP] EXEC /bin/sh -c '/usr/bin/python [.ansible]/tmp/ansible-tmp-1557386156.751379-236194994805047/AnsiballZ_k8s_facts.py && sleep 0'

and gives the same error about installing openshift.

/usr/bin/python -m pip install openshift

confirms it is installed.

@willthames
Copy link
Contributor

Ok, if you're seeing [IP] I'm pretty sure it's not running locally, it's running on a remote host which itself doesn't have openshift installed. That also explains why ansible_python_interpreter isn't being respected (I think)

Are you trying to run ansible on a target host to make kubernetes api calls? Do you have a reason to do that (either way is fine, but I'm troubleshooting a problem without all of the information)

You're only giving me minimal snippets of information, this would be much better investigated as a new separate issue, with a proper minimal reproducible test case (https://stackoverflow.com/help/mcve) - creating such a thing is often a good path to isolating the issue and finding the cause of the issue yourself.

@willthames
Copy link
Contributor

I'm going to close this issue now. If people are having an issue that appears similar to this one, please open a new issue fully describing your problem.

@ansible ansible locked and limited conversation to collaborators Aug 5, 2019
@ansible ansible deleted a comment Oct 14, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. clustering Clustering category k8s module This issue/PR relates to a module. support:community This issue/PR relates to code supported by the Ansible community.
Projects
None yet
Development

No branches or pull requests