Replies: 16 comments 32 replies
-
|
Please provide actual replication procedures. Exactly what did you run (the commands to execute) in order to install the operator and Kiali server, and please indicate what the Kiali CR actually looks like. I suspect this is not a bug. But without knowing exactly how this was installed, I can't tell. Note also regarding this: |
Beta Was this translation helpful? Give feedback.
-
|
I used the following Helm values and included the Custom Resource (CR) definition as shown below, but the installation did not create the Kiali server and the resources that gets configured via the CR Therefore I had to apply the CR manually using kubectl apply. Here are the details of that CR: Noted: |
Beta Was this translation helpful? Give feedback.
-
|
Could this also be the reason why only namespaces, |
Beta Was this translation helpful? Give feedback.
-
|
(I converted this to a Q&A Discussion - this does not appear to be a bug.) |
Beta Was this translation helpful? Give feedback.
-
|
You did not tell me exactly how you installed (what command you used to install). You only gave me the values yaml you used. So I can only blindly guess at how you are installing this. But I just tested this myself - and here is what I did. I used the helm CLI directly. The Kiali Operator installed fine: The Kiali Server installed fine: The Kiali CR was created correctly - this is what it looks like after the operator fully reconciled it and installed Kiali Server: results in: apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
annotations:
ansible.sdk.operatorframework.io/verbosity: "1"
meta.helm.sh/release-name: kiali-operator
meta.helm.sh/release-namespace: kiali-operator
creationTimestamp: "2024-04-23T15:10:25Z"
finalizers:
- kiali.io/finalizer
generation: 1
labels:
app: kiali-operator
app.kubernetes.io/instance: kiali-operator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kiali-operator
app.kubernetes.io/part-of: kiali-operator
app.kubernetes.io/version: v1.83.0
helm.sh/chart: kiali-operator-1.83.0
version: v1.83.0
name: kiali
namespace: istio-system
resourceVersion: "1518"
uid: bddc578a-f6c1-4273-b60c-25562297158f
spec:
auth:
strategy: token
deployment:
accessible_namespaces:
- .*
view_only_mode: false
external_services:
custom_dashboards:
grafana:
url: http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80
prometheus:
url: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
server:
web_root: /kiali
status:
conditions:
- ansibleResult:
changed: 12
completion: 2024-04-23T15:11:08.892327
failures: 0
ok: 76
skipped: 54
lastTransitionTime: "2024-04-23T15:10:57Z"
message: Awaiting next reconciliation
reason: Successful
status: "True"
type: Running
- lastTransitionTime: "2024-04-23T15:11:09Z"
message: Last reconciliation succeeded
reason: Successful
status: "True"
type: Successful
- lastTransitionTime: "2024-04-23T15:11:09Z"
message: ""
reason: ""
status: "False"
type: Failure
deployment:
accessibleNamespaces: default,ingress-nginx,istio-system,kiali-operator,kube-node-lease,kube-public,kube-system
instanceName: kiali
namespace: istio-system
environment:
isKubernetes: true
kubernetesVersion: 1.27.3
operatorVersion: v1.83.0
progress:
duration: "0:00:10"
message: 7. Finished all resource creation
specVersion: defaultThis is the latest Kiali helm charts - but nothing should have changed between the old 1.66 version and the latest version. Those commands should still work. In fact, I just tried it myself and it worked the same way. First I uninstalled the Kiali CR via apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
annotations:
ansible.sdk.operatorframework.io/verbosity: "1"
meta.helm.sh/release-name: kiali-operator
meta.helm.sh/release-namespace: kiali-operator
creationTimestamp: "2024-04-23T15:18:23Z"
finalizers:
- kiali.io/finalizer
generation: 1
labels:
app: kiali-operator
app.kubernetes.io/instance: kiali-operator
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kiali-operator
app.kubernetes.io/part-of: kiali-operator
app.kubernetes.io/version: v1.66.1
helm.sh/chart: kiali-operator-1.66.1
version: v1.66.1
name: kiali
namespace: istio-system
resourceVersion: "2390"
uid: f4b22e1c-9bf3-41ed-a5b2-f681d61601df
spec:
auth:
strategy: token
deployment:
accessible_namespaces:
- .*
view_only_mode: false
external_services:
custom_dashboards:
grafana:
url: http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80
prometheus:
url: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
server:
web_root: /kiali
status:
conditions:
- ansibleResult:
changed: 20
completion: 2024-04-23T15:19:08.517029
failures: 0
ok: 109
skipped: 104
lastTransitionTime: "2024-04-23T15:18:53Z"
message: Awaiting next reconciliation
reason: Successful
status: "True"
type: Running
- lastTransitionTime: "2024-04-23T15:19:08Z"
message: Last reconciliation succeeded
reason: Successful
status: "True"
type: Successful
- lastTransitionTime: "2024-04-23T15:19:08Z"
message: ""
reason: ""
status: "False"
type: Failure
deployment:
accessibleNamespaces: default,ingress-nginx,istio-system,kiali-operator,kube-node-lease,kube-public,kube-system
instanceName: kiali
namespace: istio-system
environment:
isKubernetes: true
kubernetesVersion: 1.27.3
operatorVersion: v1.66.1
progress:
duration: "0:00:14"
message: 7. Finished all resource creation
specVersion: defaultOK, so I have Kiali installed and running. Since the auth.strategy is set to "token" we need to get a token so we can log into Kiali. We have instructions on our FAQ on how to do that - please see: https://kiali.io/docs/faq/authentication/#how-to-obtain-a-token-when-logging-in-via-token-auth-strategy . Note that I am using minikube running k8s version greater than 1.24 so I need to obtain the token via I point my browser to my Kiali UI and get to the login page and I copy-n-paste that token into the login field, at which point I am logged in. The masthead of the Kiali UI after I logged in looks like this: and the About Box shows us that I am logging into Kiali v1.66.1:
Finally, one last item. I don't think you have configured your Prometheus correctly. You have this: external_services:
custom_dashboards:
prometheus:
url: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
grafana:
url: "http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80"You have those configured under |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the detailed response. I installed the kiali-operator using helm and then applied the following CR file Then ran this to check the status - I am still getting this error! I can access the kiali UI ok and can view only kiali and istiod namespace.
Below are the kiali CR details -
|
Beta Was this translation helpful? Give feedback.
-
|
This sounds like it might be related to this:
But I do not know. You say you are not doing any upgrade of Kiali, its just a straight new install, so those might not be related. I cannot replicate your error installing Kiali using helm. In my earlier comment, I posted the exact commands I used to install Kiali Operator and the Server. I suggest you do that (uninstall everything and then run those exact commands I posted) and see if you get the same error. I suspect you are installing things differently than me. The only other idea I have is the error you are getting |
Beta Was this translation helpful? Give feedback.
-
|
The only alternative I can think of is try to install the latest Kiali Operator (not the old 1.66) and see if you get the same behavior. |
Beta Was this translation helpful? Give feedback.
-
|
That Also this is a good indicator: So you should have a running Kiali Server. |
Beta Was this translation helpful? Give feedback.
-
|
Note that Kiali server probably won't work correctly, because 1.83 doesn't work with the istio you have, but at least we can see Kiali Operator was able to install it successfully. So perhaps there was a problem with 1.66 that has now been fixed. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli Are you suggesting that I can avoid the last status as per below - |
Beta Was this translation helpful? Give feedback.
-
|
Saying this - Are you suggesting to downgrade now to v1.66.1 to make it compatible with the current istio version that I have and see if that works? |
Beta Was this translation helpful? Give feedback.
-
|
For the first question, yes, ignore that. I believe you are misunderstanding what that is telling you. See "status" == "False"? That means that status block is not in effect (it is False, not True). So this: mean the "failure" type is "false" - in other words, it is false that it failed. i.e. it succeeded :) It's confusing, but that's what it means. For the second question, yes again. I would say uninstall 1.83 completely and try 1.66.1 again and see what happens. |
Beta Was this translation helpful? Give feedback.
-
|
@genfemme Are you committed to Istio 1.17? If you can update Istio you can use newer versions of Kiali. |
Beta Was this translation helpful? Give feedback.
-
|
@jshaughn In the meantime, could you confirm the updated version to use would be the following - |
Beta Was this translation helpful? Give feedback.
-
|
Understood, I was just asking. As for Kiali 1.83, it is only validated against 1.21. See that compatibility chart. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli @jshaughn |
Beta Was this translation helpful? Give feedback.
-
|
I doubt 1.83 is compatible with 1.17, I would not try that combination. If you have a problem with 1.66.1 you may have success with Kiali 1.73 and Istio 1.17, as Kiali 1.73 has been tested against Istio 1.16 and is expected also to work against Istio 1.18. v1.73 is an unusual version as it is tied to Red Hat OpenShift Service Mesh offerings. |
Beta Was this translation helpful? Give feedback.
-
|
@jshaughn @jmazzitelli Following are the kiali-operator logs. I have added the last part of the logs to show the error - This is the output of the kiali server status - When I checked if the |
Beta Was this translation helpful? Give feedback.
-
|
When you say, So, there has to be something that is wrong with your installation steps. I've asked several times for the full and exact steps you perform to install, but I have yet to see it. So, I cannot help any further because I have no idea how exactly you are installing and I suspect something you are doing is broke. If you are not using the Kiali.io instructions to install (via the helm CLI commands), then look for differences in your installation method compared with the supported methods of installation. As I mentioned earlier, I gave the steps I used (with 1.66.1) and it works for me on minikube. So you need to check the installation steps you are performing and find out what is different. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli Detailed Steps to Deploy Kiali Using ArgoCD ArgoCD application.yaml file I am adding the below to the As ArgoCD is set to automatically synchronize with our GitHub repository, it detects changes committed to the application.yaml and values.yaml files and initiates the deployment process without manual triggers. Since adding the CR via the values.yaml file is not working, I am manually applying the file to the cluster using the below details - And then creating the token for the service account using the below file - Then I check the status by running this - |
Beta Was this translation helpful? Give feedback.
-
|
I don't know what ArgoCD is doing under the covers, and the fact that Do you have a staging environment that mimics your production environment? You need to run the documented/supported install instructions and see if you get the same error. If not, then we can be sure something within ArgoCD is breaking (and I've seen a case in the past where ArgoCD doesn't do correct things related to Kiali). So, in your staging environment, do this. But first make sure nothing Kiali related is installed in your cluster, so uninstall all Kiali-related resources from your cluster before running this test. I cannot stress this enough - the cluster must not have anything Kiali related installed - no ClusterRoles, no ClusterRoleBindings, no ServiceAccounts, etc. Then get up-to-date Kiali helm charts installed on your machine - either with "helm repo add" or "helm repo update" helm repo add kiali https://kiali.org/helm-charts
helm repo updateNow ensure you are connected to your cluster with a kubeadmin account, and then install the operator with your values.yaml: helm install --version 1.66.1 --namespace kiali-operator --create-namespace -f ./values.yaml kiali-operator kiali/kiali-operatorwhere cr:
namespace: istio-system
name: kiali
create: true
spec:
auth:
strategy: "token"
deployment:
accessible_namespaces: [".*"]
view_only_mode: false
server:
web_root: "/kiali"
external_services:
prometheus:
url: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"
grafana:
url: "http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local:80"Now follow the status of the Kiali CR via: See if that error shows up in the Kiali CR status, or if the Kiali CR status shows everything finishes successfully and you have a Kiali Server installed (via If you do NOT get the error, then something is broken within the ArgoCD system. You would then want to look at the differences between what ArgoCD is invoking (with respect to helm and how it configured things like the operator helm chart and kiali CR) compared to running the helm CLI. You would then need to address those differences. If you DO STILL get the error, then the problem must be within the old Kiali helm charts or operator which have apparently since been fixed (since you do not see this error when you try with 1.83). In that case, we can only suggest to upgrade Istio and Kiali because Kiali 1.75 and under as well as Istio 1.19 and under are obsolete and no longer supported or maintained. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli The deployment is required to be done using the ArgoCD setup that we have. There is nothing in the deployment logs or pod logs that I could find to provide any reference. Could you suggest any steps to check the differences? I am not sure how to proceed with looking at the differences between what ArgoCD is invoking compared to running helm Cli as you mentioned. |
Beta Was this translation helpful? Give feedback.
-
|
I have never used ArgoCD so I can't speak to that. The fact that 1.66.1 installs fine on minikube for you makes me further convinced it is either:
Since you cannot test on a staging environment that mimics your production environment in which you can test the helm CLI, you cannot eliminate that (2.) above as a possible source of the problem. FWIW: I do know people are successfully installing Kiali using ArgoCD. Perhaps you can ask on some ArgoCD online forum or the Istio slack to find those people and get their ideas. |
Beta Was this translation helpful? Give feedback.
-
|
Thank @jmazzitelli I have asked in the istio slack channel now. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli We don't have default Istio Dashboards in the Grafana instead some custom dashboards with complex queries setup in the variables section for each dashboard. Can custom dashboards be added to the CR? Can you also provide the reason of this error - "User does not have permission to create Istio Config on namespace: istio-system" and where to see the config to configure this correctly? |
Beta Was this translation helpful? Give feedback.
-
|
waiting for response |
Beta Was this translation helpful? Give feedback.
-
|
Can you start another discussion - we don't want to bleed mutiple issues into a single discussion. Helps people in the future look up/google problems and get taken to discussions related onto to a single issue. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli |
Beta Was this translation helpful? Give feedback.
-
|
@genfemme have you tried just deploying the kiali server helm chart and not using the operator: https://kiali.io/docs/installation/installation-guide/install-with-helm/#standalone-kiali-installation? |
Beta Was this translation helpful? Give feedback.
-
|
I haven't as we want to get all the functionalities that the operator provides and it is recommended only for convenience as per the screenshot from the official doc - https://kiali.io/docs/installation/installation-guide/install-with-helm/#standalone-kiali-installation. We have helm set up with ArgoCD. |
Beta Was this translation helpful? Give feedback.
-
|
Something isn't right -- look at the Kiali CR status field: "deployment": {
"accessibleNamespaces": "istio-system",
"instanceName": "kiali",
"namespace": "istio-system"
},
For some reason, accessible_namespaces is not being processed in the way you think. Do those namespaces exist at the time you create the Kiali CR? This is behavior I would expect to see if namespaces specified in accessible_namespaces do NOT exist at the time the Kiali CR is processed by the operator. I would suggest running your Kiali CR through the validator. Maybe something is incorrect with it. See: https://kiali.io/docs/configuration/kialis.kiali.io/#validating-your-kiali-cr |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli The output of the validator -
|
Beta Was this translation helpful? Give feedback.
-
|
Wait.. I just looked at your output: |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli But the role "kiali-viewer" is still not being added to the namespaces stated in the CR. |
Beta Was this translation helpful? Give feedback.
-
|
Why are you expecting a |
Beta Was this translation helpful? Give feedback.
-
|
FWIW: I just tried it and it works with v1.73. Which makes sense because here you see the permissions in the role template that the operator will set: https://github.com/kiali/kiali-operator/blob/master/roles/v1.73/kiali-deploy/templates/kubernetes/role.yaml#L58-L71 role-viewer is similar (but no write access) - but this is only created if view_only_mode is true: https://github.com/kiali/kiali-operator/blob/master/roles/v1.73/kiali-deploy/templates/kubernetes/role-viewer.yaml#L55-L65 In either case, those permissions should be getting set. The errors you show are happening only in the Something else must be going on that I can't see. But given your last problem was due to user error (that long namespace name with the commas in it), and your other error appears non-Kiali related with ArgoCD, I can't rule out those types of issues here either. I would just say double check everything, consult the EKS cluster documentation to see if any limitations are on the "default" namespace, and check the operator logs for any install errors that might indicate where the problem is. Honestly, at this point, with the amount of support you are requiring, it might benefit you to obtain some official product support from an Istio vendor. I cannot personally attend to these issues much anymore as it is becoming a large time sink. |
Beta Was this translation helpful? Give feedback.
-
|
@jmazzitelli The reason to set this as false as I want to provide read-write access. I want to provide full access to one account at our end to check the all functionality. I tried using other namespaces as well one and nothing changed. Your help is greatly appreciated so far, and I understand the constraints on your time. |
Beta Was this translation helpful? Give feedback.
-
|
Btw, If in the docs you find an omission, mistake, or something that could be made clear, we encourage submissions! Kiali is a community project. |
Beta Was this translation helpful? Give feedback.
-
Describe the bug
I am encountering an issue while deploying a Kiali Server via a Kiali Custom Resource (CR).
When applying the CR, the operation fails due to an RBAC configuration error, stating that the
kiali-viewerrole cannot be found, despite its existence. Additionally, I discovered that thekiali-service-account, which is crucial for Kiali's operation, does not generate any tokens, preventing proper authentication and authorization processes.Error Message with
Expected Behavior
The Kiali operator should successfully acknowledge and utilize the existing
kiali-viewerrole to configure the RoleBinding as specified in the CR. Thekiali-service-accountshould automatically generate tokens upon creation.Environment
Kiali Operator version: 1.83.0
Istio version: 1.17.4
Kubernetes implementation: EKS
Kubernetes version: 1.28.3
Beta Was this translation helpful? Give feedback.
All reactions