Noobaa Operator phase Rejected with Error: Persistent Error: Invalid endpoint min count (must be greater than 0)

[NgHuuAn]

Environment info

• NooBaa Operator Version: [root@helper ~]# noobaa version
  INFO[0000] CLI version: 5.14.3
  INFO[0000] noobaa-image: noobaa/noobaa-core:master-20230710
  INFO[0000] operator-image: noobaa/noobaa-operator:5.14.3

• Platform: [root@helper ~]# oc version
  Client Version: 4.15.0
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: 4.15.0
  Kubernetes Version: v1.28.6+6216ea1

Actual behavior

1. We encountered a warning that Phase was Rejected by Noobaa Operators, please help us check it

Steps to reproduce

1. Try resovled by delete the pods odf-operator-..., noobaa-operators-..., noobaa-endpoint-...

More information - Screenshots / Logs / Other output

Noobaa status:

Status:
  Accounts:
    Admin:
      Secret Ref:
        Name:       noobaa-admin
        Namespace:  openshift-storage
  Actual Image:     registry.redhat.io/odf4/mcg-core-rhel9@sha256:34c6ad754db3f7365737a3990a8bb45d99a570fb101f410253bfe7812ecebbe7
  Conditions:
    Last Heartbeat Time:   2024-03-13T04:20:06Z
    Last Transition Time:  2024-03-13T04:20:06Z
    Message:               Invalid endpoint min count (must be greater than 0)
    Reason:                InvalidEndpointsConfiguration
    Status:                Unknown
    Type:                  Available
    Last Heartbeat Time:   2024-03-13T04:20:06Z
    Last Transition Time:  2024-03-13T04:20:06Z
    Message:               Invalid endpoint min count (must be greater than 0)
    Reason:                InvalidEndpointsConfiguration
    Status:                False
    Type:                  Progressing
    Last Heartbeat Time:   2024-03-13T04:20:06Z
    Last Transition Time:  2024-03-13T04:20:06Z
    Message:               Invalid endpoint min count (must be greater than 0)
    Reason:                InvalidEndpointsConfiguration
    Status:                True
    Type:                  Degraded
    Last Heartbeat Time:   2024-03-13T04:20:06Z
    Last Transition Time:  2024-03-13T04:20:06Z
    Message:               Invalid endpoint min count (must be greater than 0)
    Reason:                InvalidEndpointsConfiguration
    Status:                Unknown
    Type:                  Upgradeable
    Last Heartbeat Time:   2024-03-11T08:40:38Z
    Last Transition Time:  2024-03-11T03:35:00Z
    Status:                k8s
    Type:                  KMS-Type
    Last Heartbeat Time:   2024-03-11T08:40:38Z
    Last Transition Time:  2024-03-11T03:35:02Z
    Status:                Sync
    Type:                  KMS-Status
  Endpoints:
    Ready Count:  1
    Virtual Hosts:
      s3.openshift-storage.svc
      s3-openshift-storage.apps.cluster.domain.local
  Observed Generation:  8
  Phase:                Rejected
  Readme:
                        ERROR: NooBaa operator cannot reconcile this system spec.

  Check out the system status.phase, status.conditions, and events with:

    kubectl -n openshift-storage describe noobaa
    kubectl -n openshift-storage get noobaa -o yaml
    kubectl -n openshift-storage get events --sort-by=metadata.creationTimestamp

  In order to retry, edit the system spec and the operator is watching and will be notified.

  NooBaa Core Version:     master-20230710
  NooBaa Operator Version: 5.14.5

All pod running well:

[root@helper ~]# oc get pod | grep noobaa
noobaa-core-0                                                     1/1     Running   0               2d
noobaa-db-pg-0                                                    1/1     Running   0               2d
noobaa-endpoint-86476cffb9-962n2                                  1/1     Running   0               52m
noobaa-operator-6c54f4454b-rnjdl                                  2/2     Running   0               52m

Noobaa operator log:

[root@helper ~]# oc logs noobaa-operator-6c54f4454b-rnjdl
...
time="2024-03-13T04:20:06Z" level=info msg="SetPhase: \"Rejected\"" sys=openshift-storage/noobaa
time="2024-03-13T04:20:06Z" level=error msg="❌ Persistent Error: Invalid endpoint min count (must be greater than 0)" sys=openshift-storage/noobaa
...

Noobaa core log:

Mar-13 4:29:25.205 [WebServer/38]    [L0] core.rpc.ice:: Got error connect EINVAL fe80::858:aff:fe60:215:60100 - Local (:::0)
Mar-13 4:30:25.514 [WebServer/38]    [L0] core.rpc.ice:: Got error connect EINVAL fe80::858:aff:fe60:215:60100 - Local (:::0)
Mar-13 4:31:25.830 [WebServer/38]    [L0] core.rpc.ice:: Got error connect EINVAL fe80::858:aff:fe60:215:60100 - Local (:::0)