Release "prometheus-operator" failed: rpc error: code = Canceled #6130
Comments
|
We have the same issue on minikube so it does not seem to be specific to AWS. |
|
We have the same issue on kubespray-deployed clusters. |
|
I'm also seeing the issue on both k8s 12.x and 13.x k8s kubespray deployed clusters in our automated pipeline - 100% failure rate. The previous version of prometheus-operator(0.30.1) works without issues. |
|
Saw there was an update to promethus chart today. I bumped it to and i'm no longer seeing the issue. |
|
@rnkhouse Can you check with the latest chart version as mentioned by @dlevene1 in #6130 (comment)? |
|
I have this same issue with version 6.8.1 on AKS. |
|
We have the same issue on kubespray-deployed clusters. Kubernete version: Prometheus-operator version: |
|
I have the same issue on aks. |
|
Can anyone reproduce this issue in Helm 3, or does it propagate as a different error? My assumption is that with the removal of tiller this should no longer be an issue. |
|
@bacongobbler This is still an issue in Helm 3. bash$ helm install r-prometheus-operator stable/prometheus-operator --version 6.8.2 -f prometheus-operator/helm/prometheus-operator.yaml
manifest_sorter.go:179: info: skipping unknown hook: "crd-install"
Error: apiVersion "monitoring.coreos.com/v1" in prometheus-operator/templates/exporters/kube-controller-manager/servicemonitor.yaml is not available |
|
That seems to be a different issue than the issue raised by the OP, though.
Can you check and see if you're using the latest beta release as well? That error was seemingly addressed in #6332 which was released in 3.0.0-beta.3. If not can you open a new issue? |
|
@bacongobbler i'm using the latest Helm v3.0.0-beta.3. |
|
I had to go back to --version 6.7.3 to get it to install properly |
|
Our workaround is to keep prometheus operator image on v0.31.1. |
|
helm.log After some fiddling with install options --debug and such, am now getting:
Edit: May try updating my helm versions, currently at v2.12.3 related: helm/charts#15977 |
|
After doing some digging with @cyp3d it appears that the issue could be caused by a helm delete timeout that's too short for some clusters. I cannot reproduce the issue anywhere, so if someone who is experiencing this could validate a potential fix in the linked pull request branch I would much appreciate it! |
|
Same here on several Clusters created with kops on AWS. |
|
Could you try the potential fix in this PR? helm/charts#17090 |
|
I gave the PR a run through and still the same |
|
@vsliouniaev Nope, does not fix the issue here |
|
Just checked the updated PR still seeing the following on our infra: FYI we are on Kuber 1.14.3 |
|
I was able to get around this issue by following the 'Helm fails to create CRDs' section in readme.md. I'm not sure how they're related, but it worked. Step 1: Manually create the CRDS Step 2: Step 3: |
|
@vsliouniaev Still same issue! Though the workaround from lethalwire works. |
|
The lethalwire workaround has me resolved as well. |
|
So 4 days a part the workaround worked and stopped working I had to use the CRDs file from |
|
I tried on chart v8.2.4: if Also, like what vsliouniaev said, what does |
|
@truealex81 Since helm3 is meant to give more information about this, can you please post verbose logs from the install process? |
|
I am receiving the same issue deploying 8.2.4 on Azure AKS. Helm Version: Helm --debug produces this output: I can reproduce this reliably. If there is a way to get more verbose logs, please let me know and i post the output here |
|
@pather87 thanks a lot! Here's the order of what's meant to happen in the chart:
Could you please check if you have any failed jobs still present? From the logs it reads like you shouldn't because they were all successful. Are there any other resources present in the cluster after the |
|
Same here when install prometheus-operator:
|
|
@vsliouniaev thanks for your answer!
kubectl get all -lrelease=prometheus-operator |
|
Installation with debug: After, then I execute: |
|
What I've also discovered by trying to work around this: The issue persists, if i delete the chart and the CRDs afterwards and install the chart again, but the issue does not persist, if i do not delete the crds. I tried out and installed the crds beforehand, and do a |
|
The next log line I would expect after this is about post-install,post-upgrade hooks, but it does not appear in your case. I'm not certain what helm is waiting on here |
|
Manual CRDs creation helps at least on Azure. |
|
Thanks @truealex81 ! That works on Azure. |
|
myenv: but: i guess the lastest version of prometheus-operator is not compatible?!!! |
|
@bierhov please can you post the resources in the namespace after a failure? |
|
yes! |
|
Can you please post the resources though? |
sorry,i cant reappear,unless i remove my stable helm env and do it again! |
|
@bierhov do you have any failed jobs left after the install? |
my k8s version is 1.11.2 helm an tiller version is 2.13.1 |
|
Not reproducable using: Kubernetes version:
|
|
@gramozkrasniqi |
|
@vsliouniaev if you dont create them you will get the error. |
|
We use prometheus-operator in our deployment, in a nutshell, we upgraded prom-op from 6.9.3 to 8.3.3 and always failed with "Error: context canceled". I try to refresh crds, which in 'github.com/helm/charts/tree/master/stable/prometheus-operator' mentions ( like this kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/alertmanager.crd.yaml ), but these don't exists anymore. I almost gave up, but with these crds, helm deploy succeeded ! yeyyyy My setup: Kubernetes version: Purge prometheus-operator from k8s ! Then: That's all ! |
|
Does this mean that it's necessary to do a clean install and lose historical metrics data? |
|
Аfter upgarding AKS k8s to 1.15.5, helm to 3.0.1 and Prometheus-operator chart to 8.3.3 the problem is gone. |
worked for me as well on AKS |
in azure kubernetes works, thanks |
Thanks, this worked for me with AKS cluster. had to change the URL for the CRD's. kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.37/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml --validate=false helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring --set prometheusOperator.createCustomResource=false |
|
Closing. Looks like this has been since resolved, according to the last three commenters. Thanks! |
and others
Describe the bug
When I try to install prometheus operator on AKS with
helm install stable/prometheus-operator --name prometheus-operator -f prometheus-operator-values.yamlI am getting this error:I checked with history:
Chart
[stable/prometheus-operator]
Additional Info
I am using below configurations to deploy a chart:
In values file:
createCustomResourceis set tofalse,Output of
helm version:Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Output of
kubectl version:Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.7", GitCommit:"4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState:"clean", BuildDate:"2019-06-06T01:39:30Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud Provider/Platform (AKS, GKE, Minikube etc.):
AKS
The text was updated successfully, but these errors were encountered: