podman_images module fails to push image to registry when "force: true" is set

[friendlypenguin]

/kind bug

Description

podman_images module fails to push a image to a registry when set "force: true" complaining about not beeing able to pull. It works however when "force" option is not set.

Steps to reproduce the issue:

1. Non-working playbook:
   cat ee_builder_issue.yml

---
- name: build execution environment
  hosts: ee_builder
  gather_facts: true
  tasks:
    - name: push tag
      containers.podman.podman_image:
        name: "{{ default_ee }}"
        tag: "latest"
        push: true
        force: true #<-- this seems to be a bug, it doesnt work
        username: "{{ registry_username }}"
        password: "{{ registry_password }}"
        push_args:
          dest: "{{ registry_push_url }}"

The playbook works fine when omitting the line "force: true"

2. Running the whole playbook with:
   ansible-navigator run ee_builder_issue.yml

Describe the results you received:
Non-working version fails with:

│ msg: Failed to pull image default_ee:latest

Describe the results you expected:

Expecting the task to succeed like when not setting "force: true"

Additional information you deem important (e.g. issue happens only occasionally):
fails always

Content of the Execution Environment running the playbook:

Image: default_ee:latest (Information about ansible and ansible collections)                                                                                                                                                                                                                                                                                                                       
 0│---
 1│ansible:
 2│  collections:
 3│    details:
 4│      ansible.posix: 1.5.2
 5│      awx.awx: 22.2.0
 6│      community.crypto: 2.13.0
 7│      community.general: 7.0.0
 8│      community.vmware: 3.5.0
 9│      containers.podman: 1.10.1
10│      pulp.squeezer: 0.0.13
11│      theforeman.foreman: 3.10.0
12│  version:
13│    details: ansible [core 2.14.1]

Output of podman version:

Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.6
Built:        Wed Apr 26 18:50:28 2023
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 606c693de21bcbab87e31002e46663c5f2dc8a9b'
  cpuUtilization:
    idlePercent: 99.35
    systemPercent: 0.18
    userPercent: 0.48
  cpus: 1
  distribution:
    distribution: '"rhel"'
    version: "9.2"
  eventLogger: journald
  hostname: server.XXXX.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-162.23.1.el9_1.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1036140544
  memTotal: 3832860672
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.el9_2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 4209532928
  swapTotal: 4240437248
  uptime: 390h 14m 3.00s (Approximately 16.25 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 68212199424
  graphRootUsed: 12094132224
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 120
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1682527828
  BuiltTime: Wed Apr 26 18:50:28 2023
  GitCommit: ""
  GoVersion: go1.19.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-4.4.1-9.el9_2.x86_64

Playbok you run with ansible (e.g. content of playbook.yaml):

---
- name: build execution environment
  hosts: ee_builder
  gather_facts: true
  tasks:
    - name: push tag
      containers.podman.podman_image:
        name: "{{ default_ee }}"
        tag: "latest"
        push: true
        force: true #<-- this seems to be a bug, it doesnt work with force: true
        username: "{{ registry_username }}"
        password: "{{ registry_password }}"
        push_args:
          dest: "{{ registry_push_url }}"

Command line and output of ansible run with high verbosity

ansible-navigator run plays/maintenance/ee_builder_issue.yml

content of ansible-navigator logfile (with loglevel set to debug) see attached file
log.txt

[sshnaidm]

As I see from attached log, it's a problem to pull the image, not to push it:

2023-05-15T12:25:57.105053+00:00 DEBUG 'ansible_navigator.runner.base._event_handler' ansible-runner event handle: {'uuid': '18e79646-5163-4ee9-9f79-41b11b81ee26', 'counter': 8, 'stdout': '\x1b[0;31mfatal: [XXXXserver.XXXX.com]: FAILED! => {"changed": false, "msg": "Failed to pull image default_ee:latest"}\x1b[0m', 'start_line': 7, 'end_line': 8, 'runner_ident': '80f6a6ba-a0ec-48a0-be7a-17288378f573', 'event': 'runner_on_failed', 'pid': 26, 'created': '2023-05-15T12:25:57.103733', 'parent_uuid': '8acc70df-5274-5c6c-6393-000000000062', 'event_data': {'playbook': '/home/userx/ansible/plays/maintenance/ee_builder_issue.yml', 'playbook_uuid': '87fde179-5dd3-4fa3-8d55-0574c7875655', 'play': 'build execution environment', 'play_uuid': '8acc70df-5274-5c6c-6393-000000000060', 'play_pattern': 'ee_builder', 'task': 'push tag', 'task_uuid': '8acc70df-5274-5c6c-6393-000000000062', 'task_action': 'containers.podman.podman_image', 'resolved_action': 'containers.podman.podman_image', 'task_args': '', 'task_path': '/home/userx/ansible/plays/maintenance/ee_builder_issue.yml:6', 'host': 'XXXXserver.XXXX.com', 'remote_addr': 'XXXXserver.XXXX.com', 'res': {'msg': 'Failed to pull image default_ee:latest', 'invocation': {'module_args': {'name': 'default_ee', 'tag': 'latest', 'push': True, 'force': True, 'username': 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER', 'password': 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER', 'push_args': {'dest': 'registry.XXXX.com', 'compress': None, 'format': None, 'remove_signatures': None, 'sign_by': None, 'transport': None}, 'pull': True, 'state': 'present', 'executable': 'podman', 'build': {'force_rm': False, 'format': 'oci', 'cache': True, 'rm': True, 'annotation': None, 'file': None, 'volume': None, 'extra_args': None}, 'path': None, 'validate_certs': None, 'auth_file': None, 'ca_cert_dir': None}}, '_ansible_no_log': None, 'changed': False}, 'start': '2023-05-15T12:25:53.502655', 'end': '2023-05-15T12:25:57.103480', 'duration': 3.600825, 'ignore_errors': None, 'event_loop': None, 'uuid': '18e79646-5163-4ee9-9f79-41b11b81ee26'}}

"Failed to pull image default_ee:latest"

Not sure how --force is related here, it tries and can't pull the image. Please mention exactly the image you try to pull. If it succeeds without --force, it means the image is on the host, in this case podman won't try to pull it.

[sshnaidm]

@friendlypenguin please reopen the issue if it happens again with additional information