NotAllowedInCurrentState exception when deploying using vmware_content_deploy_ovf_template

[eliasp]

SUMMARY

Sometimes, the creation of a VM using vmware_content_deploy_ovf_template fails with an NotAllowedInCurrentState exception.
Since this happens only sporadically, this seems to be a race condition/lack of error handling for cases when this exception is thrown.

ISSUE TYPE

• Bug Report

COMPONENT NAME

vmware_content_deploy_ovf_template

ANSIBLE VERSION

ansible [core 2.15.9]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.9.18 (main, Jan  4 2024, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True

COLLECTION VERSION

# /usr/share/ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.vmware 4.2.0  

CONFIGURATION

ANSIBLE_FORCE_COLOR(env: ANSIBLE_FORCE_COLOR) = True
CONFIG_FILE() = None

OS / ENVIRONMENT

• target system is vSphere 8.0U2
• pyvmomi-8.0.2.0.1
• vsphere-automation-sdk-python-8.0.2.0

STEPS TO REPRODUCE

The tasks below are used to spawn VMs, in some cases dozens at a time.
My feeling is, that this error occurs more often, the more VMs are created at the same time.

---
- name: Set vSphere cluster variables
  ansible.builtin.set_fact:
    vsphere: &vsphere_opts
      hostname: "{{ vsphere_hostname }}"
      username: "{{ vsphere_username }}"
      password: "{{ vsphere_password }}"
      datacenter: "{{ vsphere_datacenter }}"
      validate_certs: false
    cluster: "{{ vsphere_cluster }}"
    datastore: "{{ vsphere_datastore }}"

- name: Set VM folder variable
  ansible.builtin.set_fact:
    vm_folder: "systems-pipeline"

- name: Set VM folder path variables
  ansible.builtin.set_fact:
    vm_folder_path: "{{ vsphere['datacenter'] ~ '/' ~ vm_folder }}"
    vm_folder_path_full: "{{ vsphere['datacenter'] ~ '/vm/' ~ vm_folder }}"

- name: Set VM name
  ansible.builtin.set_fact:
    vm_name: "{{ inventory_hostname | ansible.builtin.split('.') | first }}"

- name: Get information about potentially already existing VM
  community.vmware.vmware_guest_info:
    <<: *vsphere_opts
    name: "{{ vm_name }}"
    folder: "{{ vm_folder_path_full }}"
    schema: vsphere
  register: existing_vm
  # don't fail when there's no matching VM
  ignore_errors: true
  delegate_to: localhost

- name: Delete existing VMs
  when: "'instance' in existing_vm and systems_recreate is sameas true"
  community.vmware.vmware_guest:
    <<: *vsphere_opts
    cluster: "{{ cluster }}"
    datastore: "{{ datastore }}"
    folder: "{{ vm_folder_path_full }}"
    state: absent
    force: true
    name: "{{ vm_name }}"
  delegate_to: localhost

- name: VM handling parallelization variables
  ansible.builtin.set_fact:
    # how long to wait for a VM to be created (minutes * seconds)
    vm_creation_timeout: "{{ (15 * 60) }}"
    # how often to poll for updates during creation (seconds)
    vm_creation_polltime: 10

- name: Create VM
  when: "'instance' not in existing_vm or systems_recreate is sameas true"
  community.vmware.vmware_content_deploy_ovf_template:
    <<: *vsphere_opts
    cluster: "{{ cluster }}"
    datastore: "{{ datastore }}"
    name: "{{ vm_name }}"
    folder: "{{ vm_folder_path }}"
    template: "RHEL9"
    library: "OS Templates"
  register: vm_creation_result
  delegate_to: localhost
  async: "{{ vm_creation_timeout }}"
  poll: 0

- name: Wait for VM creation
  when: "'instance' not in existing_vm or systems_recreate is sameas true"
  ansible.builtin.async_status:
    jid: "{{ vm_creation_result.ansible_job_id }}"
  register: vm_creation_async_poll
  delegate_to: localhost
  until: vm_creation_async_poll.finished
  retries: "{{ (vm_creation_timeout | int / vm_creation_polltime | int) | round(1, 'ceil') | int }}"
  delay: "{{ vm_creation_polltime }}"

EXPECTED RESULTS

I expect the VMs to be created without any of those tasks throwing an error.

ACTUAL RESULTS

In some cases, the creation of a VM runs into an error, throwing the traceback below.
I believe, this response from vSphere should be handled more gracefully (with a few seconds of sleep and a retry for 3-5 times), since it seems to represent a temporary state of the VM which doesn't allow to proceed with the operation.
See also the VMware Developer Documentation on StdErrorsNotAllowedInCurrentState.

Traceback (most recent call last):
  File "/tmp/.ansible/tmp/ansible-tmp-1711445806.3506498-635-136676944950384/AnsiballZ_vmware_content_deploy_ovf_template.py", line 107, in <module>
    _ansiballz_main()
  File "/tmp/.ansible/tmp/ansible-tmp-1711445806.3506498-635-136676944950384/AnsiballZ_vmware_content_deploy_ovf_template.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/tmp/.ansible/tmp/ansible-tmp-1711445806.3506498-635-136676944950384/AnsiballZ_vmware_content_deploy_ovf_template.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible_collections.community.vmware.plugins.modules.vmware_content_deploy_ovf_template', init_globals=dict(_module_fqn='ansible_collections.community.vmware.plugins.modules.vmware_content_deploy_ovf_template', _modlib_path=modlib_path),
  File "/usr/lib64/python3.9/runpy.py", line 225, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_community.vmware.vmware_content_deploy_ovf_template_payload_q4no8rl5/ansible_community.vmware.vmware_content_deploy_ovf_template_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_content_deploy_ovf_template.py", line 441, in <module>
  File "/tmp/ansible_community.vmware.vmware_content_deploy_ovf_template_payload_q4no8rl5/ansible_community.vmware.vmware_content_deploy_ovf_template_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_content_deploy_ovf_template.py", line 437, in main
  File "/tmp/ansible_community.vmware.vmware_content_deploy_ovf_template_payload_q4no8rl5/ansible_community.vmware.vmware_content_deploy_ovf_template_payload.zip/ansible_collections/community/vmware/plugins/modules/vmware_content_deploy_ovf_template.py", line 271, in deploy_vm_from_ovf_template
  File "/usr/local/lib/python3.9/site-packages/com/vmware/vcenter/ovf_client.py", line 3232, in filter
    return self._invoke('filter',
  File "/usr/local/lib/python3.9/site-packages/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='vapi.bindings.method.impl.unexpected', default_message='Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.NotAllowedInCurrentState', args=['com.vmware.vapi.std.errors.NotAllowedInCurrentState'], params=None, localized=None), LocalizableMessage(id='com.vmware.ovfs.ovfs-main.ovfs.invalid_state_for_operation', default_message='The attempted operation cannot be performed in the current state (IMPORT_ERROR).', args=['IMPORT_ERROR'], params=None, localized=None)], data : None, error_type : INTERNAL_SERVER_ERROR}