Error recreating deleted extendedsecret

[drnic]

I had deleted scf's scf.var-cc-bridge-tps secret and extendedsecret, and when I upgraded cf-operator it finally attempted to recreate the secrets (is there another way?)

But it fails/errors creating the secret:

2019-10-04T06:04:45.639Z        DEBUG   ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:368 Creating secret 'scf.var-cc-bridge-tps'
2019-10-04T06:04:45.641Z        INFO    ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:148 Error generating certificate secret: could not create or update secret 'scf.var-cc-bridge-tps': Post https://10.20.0.1:443/api/v1/namespaces/scf/secrets: context deadline exceeded
2019-10-04T06:04:45.645Z        ERROR   controller-runtime.controller   controller/controller.go:218    Reconciler error        {"controller": "extendedsecret-controller", "request": "scf/scf.var-cc-bridge-tps", "error": "generating certificate secret.: could not create or update secret 'scf.var-cc-bridge-tps': Post https://10.20.0.1:443/api/v1/namespaces/scf/secrets: context deadline exceeded", "errorVerbose": "Post https://10.20.0.1:443/api/v1/namespaces/scf/secrets: context deadline exceeded\ncould not create or update secret 'scf.var-cc-bridge-tps'\ncode.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret.(*ReconcileExtendedSecret).createSecret\n\t/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret/extendedsecret_reconciler.go:385\ncode.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret.(*ReconcileExtendedSecret).createCertificateSecret\n\t/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret/extendedsecret_reconciler.go:331\ncode.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret.(*ReconcileExtendedSecret).Reconcile\n\t/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret/extendedsecret_reconciler.go:142\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337\ngenerating certificate secret.\ncode.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret.(*ReconcileExtendedSecret).Reconcile\n\t/go/src/code.cloudfoundry.org/cf-operator/pkg/kube/controllers/extendedsecret/extendedsecret_reconciler.go:149\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88
2019-10-04T06:04:45.654Z        INFO    ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:86  Reconciling ExtendedSecret scf/scf.var-cc-bridge-tps
2019-10-04T06:04:45.654Z        INFO    ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:141 Generating certificate
2019-10-04T06:04:45.654Z        DEBUG   ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:311 Generating certificate scf.var-cc-bridge-tps
2019-10-04T06:06:17.653Z        INFO    ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:86  Reconciling ExtendedSecret scf/scf.var-cc-bridge-tps
2019-10-04T06:06:17.653Z        INFO    ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:141 Generating certificate
2019-10-04T06:06:17.653Z        DEBUG   ext-secret-reconciler   extendedsecret/extendedsecret_reconciler.go:311 Generating certificate scf.var-cc-bridge-tps

Hmm, eventually the secret was created, about 6.5m into the new runtime of cf-operator.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/168944888

The labels on this github issue will be updated when the story is started.

[viovanov]

@drnic this sounds more like a feature request for something like bosh cloudcheck.

e.g. we're thinking of forcing a change on a BOSHDeployment CRD in the form of an annotation: fissile.cloudfoundry.org/cloudcheck: true