HUAWEI CE: unable to decode json from response to exec_command

[aleita]

SUMMARY

ce_command module used through an ansible playbook does not work.
the playbook successfully connect to the device but It never get an answer.

2022-03-29 16:38:56,127 p=6818 u=root n=ansible | network_os is set to ce
2022-03-29 16:38:56,783 p=6831 u=root n=p=6831 u=root | paramiko [90.147.169.130] | Connected (version 2.0, client -)
2022-03-29 16:38:59,191 p=6831 u=root n=p=6831 u=root | paramiko [90.147.169.130] | Connected (version 2.0, client -)
2022-03-29 16:39:03,714 p=6831 u=root n=p=6831 u=root | paramiko [90.147.169.130] | Connected (version 2.0, client -)
2022-03-29 16:39:12,191 p=6831 u=root n=p=6831 u=root | paramiko [90.147.169.130] | Connected (version 2.0, client -)
2022-03-29 16:39:12,861 p=6831 u=root n=ansible | shutdown complete
2022-03-29 16:39:12,913 p=6803 u=root n=ansible | fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "msg": "Command: display version, unable to decode json from response to exec_command({\"answer\": null, \"output\": \"text\", \"prompt\": null, \"command\": \"display version\"}). received 'none'."}

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.network.ce_command.

ANSIBLE VERSION

ansible [core 2.11.10] 
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.6.8 (default, Apr 25 2019, 21:02:35) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
  jinja version = 3.0.3
  libyaml = True

COLLECTION VERSION

[root@ui03 huawei-16816]# ansible-galaxy collection list community.network
21:02:35) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by 
setting deprecation_warnings=False in ansible.cfg.

# /root/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.network 3.1.0  

# /usr/local/lib/python3.6/site-packages/ansible_collections
Collection        Version
----------------- -------
community.network 3.0.0  
[root@ui03 huawei-16816]# 

CONFIGURATION

OS / ENVIRONMENT

Centos7

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

ansible-playbook [core 2.11.10] 
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.8 (default, Apr 25 2019, 21:02:35) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
  jinja version = 3.0.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
host_list declined parsing /root/huawei-16816/inventory as it did not pass its verify_file() method
script declined parsing /root/huawei-16816/inventory as it did not pass its verify_file() method
auto declined parsing /root/huawei-16816/inventory as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /root/huawei-16816/inventory inventory source with ini plugin
Loading collection community.network from /root/.ansible/collections/ansible_collections/community/network
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.6/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: command.yaml *****************************************************************************************************************************************
Positional arguments: command.yaml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/root/huawei-16816/inventory',)
forks: 5
1 plays in command.yaml

PLAY [CloudEngine command test] ********************************************************************************************************************************
META: ran handlers

TASK [Run display version on remote devices] *******************************************************************************************************************
task path: /root/huawei-16816/command.yaml:9
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /root/.ansible/collections/ansible_collections/ansible/netcommon
redirecting (type: terminal) ansible.builtin.ce to community.network.ce
redirecting (type: cliconf) ansible.builtin.ce to community.network.ce
<*******> attempting to start connection
<*******> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /usr/local/bin/ansible-connection
<*******> local domain socket does not exist, starting it
<*******> control socket path is /root/.ansible/pc/621159a436
<*******> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<*******> Loading collection ansible.netcommon from /root/.ansible/collections/ansible_collections/ansible/netcommon
<*******> redirecting (type: terminal) ansible.builtin.ce to community.network.ce
<*******> Loading collection community.network from /root/.ansible/collections/ansible_collections/community/network
<*******> redirecting (type: cliconf) ansible.builtin.ce to community.network.ce
<*******> local domain socket listeners started successfully
<*******> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.ce from path /root/.ansible/collections/ansible_collections/community/network/plugins/cliconf/ce.py for network_os ce
<*******> ssh type is set to paramiko
<*******> 
<*******> local domain socket path is /root/.ansible/pc/621159a436
<*******> Using network group action community.network.ce for community.network.ce_command
<*******> ANSIBLE_NETWORK_IMPORT_MODULES: disabled
<*******> ANSIBLE_NETWORK_IMPORT_MODULES: module execution time may be extended
<*******> ESTABLISH LOCAL CONNECTION FOR USER: root
< ******* > EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-326239fz50t7c `"&& mkdir "` echo /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864 `" && echo ansible-tmp-1648637408.8477938-32632-75858781796864="` echo /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864 `" ) && sleep 0'
Using module file /root/.ansible/collections/ansible_collections/community/network/plugins/modules/ce_command.py
<*******> PUT /root/.ansible/tmp/ansible-local-326239fz50t7c/tmpquu2zl08 TO /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864/AnsiballZ_ce_command.py
<*******> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864/ /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864/AnsiballZ_ce_command.py && sleep 0'
<*******> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864/AnsiballZ_ce_command.py && sleep 0'
<*******> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-326239fz50t7c/ansible-tmp-1648637408.8477938-32632-75858781796864/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "display version"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 10,
            "wait_for": null
        }
    },
    "msg": "Command: display version, unable to decode json from response to exec_command('{\"command\": \"display version\", \"output\": \"text\", \"prompt\": null, \"answer\": null}'). received 'none'."
}

PLAY RECAP *****************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[ansibullbot]

Files identified in the description:
None

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[maticomba]

I have the same issue, but no experience in debugging python. If I can help somehow, just let me know. The file with the problem seems to be community.network/plugins/modules/network/cloudengine/ce_command.py
@ansibullbot !component = plugins/modules/network/cloudengine/ce_command.py

[maticomba]

I found that this happens when using Libssh as Connection Plugin. By changing it and falling back to Paramiko, it has worked again. This can be done by setting this in the ansible.cfg file:

[persistent_connection] 
ssh_type = paramiko

Source: https://www.ansible.com/blog/new-libssh-connection-plugin-for-ansible-network

[ovidioruzzier]

Unfortunately for me switching back to paramiko doesn't work.

OS VERSION

Debian 11

ANSIBLE VERSION

user@Test-debian:~$ ansible --version
ansible [core 2.13.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/user/.local/bin/ansible
  python version = 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
  jinja version = 3.1.2
  libyaml = True

OTHER

user@Test-debian:~$ cat /etc/ansible/ansible.cfg
[persistent_connection]
# ssh_type = paramiko
user@Test-debian:~$ cat huawei_network_playbook_4.yml
---
- name: Check Version
  hosts: 192.168.41.4
  connection: network_cli
  gather_facts: false
  vars:
    cli:
      port: 22
      username: ******
      password: *******
      transport: cli
    ansible_network_os: ce
    ansible_network_cli_ssh_type: paramiko
  tasks:

  - name: Retrieve CE OS version
    community.network.ce_command:
      commands:
       - display version
       - display device
      wait_for: result[0] contains *
      retries: 3
#     provider: "{{ cli }}"
#    when: ansible_network_os == 'ce'

user@Test-debian:~$ ansible-galaxy collection list | grep -e "netcommon\|community.network\|utils"
ansible.netcommon             3.0.1
ansible.utils                 2.6.1
community.network             4.0.1
ansible.netcommon 3.0.1
ansible.utils     2.6.1
community.network 3.1.0

ERROR

fatal: [192.168.41.4]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                "display version",
                "display device"
            ],
            "interval": 1,
            "match": "all",
            "provider": null,
            "retries": 3,
            "wait_for": [
                "result[0] contains *"
            ]
        }
    },
    "msg": "Command: display version, unable to decode json from response to exec_command('{\"command\": \"display version\", \"output\": \"text\", \"prompt\": null, \"answer\": null}'). received 'none'."
}

[ovidioruzzier]

Have anybody found a solution/workaround for this?

[axlgit]

I'm experiecing the same problem.
Have you found a solution for that, @ovidioruzzier ?

[Andersson007]

huawei team, could anyone take a look?

cc @QijunPan @TommyLike @edisonxiang @freesky-edward @hwDCN @niuzhenguo @xuxiaowei0512 @yanzhangi @zengchen1024 @zhongjun2

[axlgit]

Getting help from Huawei team would be wonderful!

[Andersson007]

@axlgit yep, though this part of the collection does not seem to be maintained:(

[ovidioruzzier]

@Andersson007 , actually I would say it's really the basics 'cause it seems we are experiencing problems at the SSH connection level. Which other part of the collection is supposed to work OK?

[Andersson007]

@ovidioruzzier i've no idea, this collection contains stuff moved from ansible's single repo 2 years ago.
Group of modules people/vendors wanted to maintain were moved to separate collections.
So now this collection is a kind of kitchen sink of mostly abandoned stuff: original authors/maintainers mostly do not respond unfortunately.

If you or your company are a huawei customer, you could try to inform them about the issue directly.

Alternatively, someone who has a device needed for testing can try to fix the issue themselves and submit a patch.
Here's the Quick-start guide.