Failure during Kiali CR creation with Ansible playbook errors

[genfemme]

When attempting to install the Kiali server via the Kiali operator, using the installation steps provided in the official documentation, the Custom Resource (CR) creation process fails. The status fetched using kubectl get kiali kiali -n istio-system -o jsonpath='{.status}' indicates an 'unknown playbook failure' during the Ansible reconciliation process. This results in a failure state despite the Kiali operator being active.

A clear and concise description of the bug. (provide screenshots if applicable)

Expected Behavior

The Kiali CR should be created successfully without errors, and the Kiali server should be running as configured in the CR file.

Steps to reproduce this bug?

1. Follow the Kiali Server installation via Operator instructions from the official Kiali documentation (https://kiali.io/docs/installation/installation-guide/example-install/).
2. Apply the Kiali CR with the following configuration:

 `apiVersion: kiali.io/v1alpha1
 kind: Kiali
 metadata:
   name: kiali
   namespace: istio-system
 spec:
   auth:
     strategy: "token"
   deployment:
     accessible_namespaces: ["bookinfo"]
     view_only_mode: false
   server:
     web_root: "/kiali"
`

3. Check the status of the Kiali CR using:

kubectl get kiali kiali -n istio-system -o jsonpath='{.status}'

Output -

`

{
  "conditions": [
    {
      "lastTransitionTime": "2024-04-22T12:09:51Z",
      "message": "",
      "reason": "",
      "status": "False",
      "type": "Successful"
    },
    {
      "lastTransitionTime": "2024-04-22T14:53:44Z",
      "message": "Running reconciliation",
      "reason": "Running",
      "status": "False",
      "type": "Running"
    },
    {
      "ansibleResult": {
        "changed": 2,
        "completion": "2024-04-22T14:53:48.543507",
        "failures": 1,
        "ok": 35,
        "skipped": 32
      },
      "lastTransitionTime": "2024-04-22T14:53:48Z",
      "message": "unknown playbook failure",
      "reason": "Failed",
      "status": "True",
      "type": "Failure"
    }
  ],
  "deployment": {
    "instanceName": "kiali",
    "namespace": "istio-system"
  },
  "environment": {
    "isKubernetes": true,
    "kubernetesVersion": "1.28.3",
    "operatorVersion": "v1.83.0"
  },
  "progress": {
    "duration": "0:00:02",
    "message": "2. Setting up configuration"
  },
  "specVersion": "default"

`

Environment

Environment

Kiali Operator version: 1.83.0
Istio version: 1.17.4
Kubernetes implementation: EKS
Kubernetes version: 1.28.3

[jshaughn]

Hi @genfemme, I'm not sure if it's the problem but your Kiali version is much higher than recommended given the Istio version. According to the Prerequisite Chart, you should probably try Kiali v1.66.1.

[genfemme]

Thank you for your prompt response @jshaughn
I will downgrade Kiali to v1.66.1 as recommended.
Post-downgrade, should I follow the same installation steps as described in the documentation, or are there any additional considerations I should be aware of given the change in version?
In case I encounter similar issues after downgrading, could you advise on the best steps or logs to check to troubleshoot effectively?

[jshaughn]

I think the steps are similar but you can also access the version-specific docs here: https://v1-66.kiali.io/ If you have problems let us know. Of course, if possible I would also recommend updating Istio, as 1.17 is out of support.

[genfemme]

Thanks @jshaughn
Following your recommendation, I downgraded Kiali to v1.66.1 to match the compatibility requirements. However, I am still encountering the same issue with the Kiali Custom Resource (CR) failing to create successfully.

Here are the updated details from the Kiali CR status:

`{

"conditions": [
{
"lastTransitionTime": "2024-04-22T12:58:53Z",
"message": "",
"reason": "",
"status": "False",
"type": "Successful"
},
{
"lastTransitionTime": "2024-04-22T16:07:33Z",
"message": "Running reconciliation",
"reason": "Running",
"status": "False",
"type": "Running"
},
{
"ansibleResult": {
"changed": 2,
"completion": "2024-04-22T16:07:42.978275",
"failures": 1,
"ok": 47,
"skipped": 61
},
"lastTransitionTime": "2024-04-22T16:07:43Z",
"message": "unknown playbook failure",
"reason": "Failed",
"status": "True",
"type": "Failure"
}
],
"deployment": {
"instanceName": "kiali",
"namespace": "istio-system"
},
"environment": {
"isKubernetes": true,
"kubernetesVersion": "1.25.16-eks-b9c9ed7",
"operatorVersion": "v1.66.1"
},
"progress": {
"duration": "0:00:06",
"message": "2. Setting up configuration"
},
"specVersion": "default"
}
`

[genfemme]

What is the best version of Istio to use with Kiali v1.66.1?

[jshaughn]

@genfemme , just look at the chart I linked above. First, I'd suggest choosing the most recent version of Istio that you can, and then use the highest compatible version of Kiali. To answer your question, v1.66.1 or Kiali should be the best for istio 1.17.

[jmazzitelli]

That is really weird - I have never seen that before.

To find out more detailed errors (we hope - I never saw such a generic error message from the operator SDK), go in the Kiali Operator pod and look at the logs. See what error messages it shows.

[genfemme]

@jmazzitelli
Given the extensive volume of logs of kiali-operator pod, I used grep to filter for entries containing "error"
TASK [default/kiali-deploy : Determine all accessible namespaces, expanding regex expressions to matched namespaces] *** An exception occurred during task execution. To see the full traceback, use -vvv. The error was: re.error: nothing to repeat at position 1 fatal: [localhost]: FAILED! => {"changed": false}

Please let me know if there are specific parts of the logs or additional details you would like me to provide.

[jmazzitelli]

What is the explicit value of your spec.deployment.accessible_namespaces inside the Kiali CR? Get it via kubectl, something like kubectl get kiali kiali -n istio-system -ojsonpath='{.spec.deployment.accessible_namespaces}'

Based on that error message, it seems like something is odd about the value of that setting. I can't think what it could be, but that's what the problem seems to be located.

Look in the operator logs, as it starts reconciling the Kiali CR, it should output the value of the CR values (find the output of the task named "Print some debug information").

Check to make sure there isn't some hidden character in there, some add ASCII or something like that.

[genfemme]

@jmazzitelli

Thank you for your guidance on resolving the namespace access issue. It turns out I was missing a "." when using the wildcard in the accessible_namespaces. Correcting it to [".*"] as shown below resolved that problem:

kubectl get kiali kiali -n istio-system -ojsonpath='{.spec.deployment.accessible_namespaces}'

[".*"]%

However, I've encountered a new issue that seems to involve RBAC configurations. Here is the relevant error message from the operator logs:

`

"RoleBinding kiali: Failed to create object: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"roles.rbac.authorization.k8s.io \\"kiali-viewer\\" not found","reason":"NotFound","details":{"name":"kiali-viewer","group":"rbac.authorization.k8s.io","kind":"roles"},"code":404}\n'"
`

This error is puzzling as the kiali-viewer role does indeed exist in the istio-system namespace. Further investigation revealed that the kiali-service-account in the istio-system namespace doesn't have any tokens associated with it, which is unusual for a service account that's supposed to interact with the Kubernetes API on behalf of an application like Kiali.

Do you have any suggestions on why the RoleBinding operation might fail to recognize an existing role, or why the service account might lack tokens?

[jmazzitelli]

It turns out I was missing a "." when using the wildcard in the accessible_namespaces.

But you said in your original description, your value was this:

     accessible_namespaces: ["bookinfo"]

[genfemme]

@jmazzitelli
I apologize for any confusion in my previous descriptions. To clarify, I originally had the accessible_namespaces set to include a specific namespace, ["bookinfo"], as part of my initial configuration. However, I also experimented with using a wildcard to grant access to all namespaces.
This is when I adjusted the configuration to use [""], which I later corrected to ["."] after realizing the mistake with the regex pattern.

[jmazzitelli]

(converted to a Q&A discussion as this is not a bug)