You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm setting up an infrastructure based on scale-to-zero, and therefore scale-from-zero too.
To do this, we're using the now-familiar "cluster autoscaler", coupled with cluster API (specifically the machineDeployment resource with some annotations).
The node scaling is working fine.
For the moment, I'm just trying to create an "autoscaler-go" knative service, from the cluster where no node is available.
The resource is then "pending", which is expected.
NAME READY STATUS RESTARTS AGE
user-service-00001-deployment-6f6d577c45-rtjvz 0/2 Pending 0 1m32s
Here is the configuration I used to create the service:
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON ACTUAL REPLICAS DESIRED REPLICAS
user-service-00001 user-service 1 False Unschedulable 1 0
This is the events I get from the revision
Warning InternalError 7m29s revision-controller failed to update deployment "user-service-00001-deployment": Operation cannot be fulfilled on deployments.apps "user-service-00001-deployment": the object has been modified; please apply your changes to the latest version and try again
Warning InternalError 7m29s revision-controller failed to update PA "user-service-00001": Operation cannot be fulfilled on podautoscalers.autoscaling.internal.knative.dev "user-service-00001": the object has been modified; please apply your changes to the latest version and try again
The PodAutoscaler resource is not ready, and the DesiredScale is 0.
NAME DESIREDSCALE ACTUALSCALE READY REASON
user-service-00001 0 1 False NoTraffic
the events from the PodAutoscaler resource
Status:
Actual Scale: 1
Conditions:
Last Transition Time: 2024-02-23T16:32:02Z
Message: The target is not receiving traffic.
Reason: NoTraffic
Status: False
Type: Active
Last Transition Time: 2024-02-23T16:32:02Z
Message: The target is not receiving traffic.
Reason: NoTraffic
Status: False
Type: Ready
Last Transition Time: 2024-02-23T16:38:03Z
Status: True
Type: SKSReady
Last Transition Time: 2024-02-23T16:32:02Z
Status: True
Type: ScaleTargetInitialized
Desired Scale: 0
Metrics Service Name: user-service-00001-private
Observed Generation: 2
Service Name: user-service-00001
I got error logs from the autoscaler pod
{"severity":"ERROR","timestamp":"2024-02-23T15:55:24.847361414Z","logger":"autoscaler","caller":"podautoscaler/reconciler.go:314","message":"Returned an error","commit":"239b73e","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"2c39855d-329c-43a0-99a9-204f4944e4af","knative.dev/key":"3010eb09/user-service-00001","targetMethod":"ReconcileKind","error":"error scaling target: failed to get scale target {Deployment user-service-00001-deployment apps/v1 }: error fetching Pod Scalable 3010eb09/user-service-00001-deployment: deployments.apps \"user-service-00001-deployment\" not found","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler/reconciler.go:314\nmain.(*leaderAware).Reconcile\n\tknative.dev/serving/cmd/autoscaler/leaderelection.go:44\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2024-02-23T15:55:24.847442144Z","logger":"autoscaler","caller":"controller/controller.go:566","message":"Reconcile error","commit":"239b73e","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"2c39855d-329c-43a0-99a9-204f4944e4af","knative.dev/key":"3010eb09/user-service-00001","duration":"787.035µs","error":"error scaling target: failed to get scale target {Deployment user-service-00001-deployment apps/v1 }: error fetching Pod Scalable 3010eb09/user-service-00001-deployment: deployments.apps \"user-service-00001-deployment\" not found","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20231023151236-29775d7c9e5c/controller/controller.go:491"}
I started to play with the configuration in an attempt to find the parameter that would unlock everything, but this was not successful. Please note that the values are intentionally exaggerated in an attempt to highlight a pattern.
I'm not sure what I'm doing wrong. It looks like the revision has no reconciler, but I'm not sure.
The pod is running and the service is created, but the revision is not, which is why the service is not ready, and it's a bit of a mystery.
Could you please help me understand what is wrong with my configuration?
The text was updated successfully, but these errors were encountered:
Ask your question here:
Hello,
I'm setting up an infrastructure based on scale-to-zero, and therefore scale-from-zero too.
To do this, we're using the now-familiar "cluster autoscaler", coupled with cluster API (specifically the machineDeployment resource with some annotations).
The node scaling is working fine.
For the moment, I'm just trying to create an "autoscaler-go" knative service, from the cluster where no node is available.
The resource is then "pending", which is expected.
Here is the configuration I used to create the service:
After a few minutes, the pod is still pending, but we get an event that says the cluster autoscaler has been triggered.
When the node is available, the pod is created and running.
However, the service is not ready, and the revision is not created.
This is the events I get from the revision
The
PodAutoscaler
resource is not ready, and theDesiredScale
is 0.the events from the
PodAutoscaler
resourceI got error logs from the autoscaler pod
PodAutoscaler
resource:Changed reachability manually from "Unreachable" to "" and Changed desiredScale manually from "0" to "1"
The configuration I tried
I started to play with the configuration in an attempt to find the parameter that would unlock everything, but this was not successful. Please note that the values are intentionally exaggerated in an attempt to highlight a pattern.
config-autoscaler:
config-deployment:
config-defaults:
The problem I'm facing
I'm not sure what I'm doing wrong. It looks like the revision has no reconciler, but I'm not sure.
The pod is running and the service is created, but the revision is not, which is why the service is not ready, and it's a bit of a mystery.
Could you please help me understand what is wrong with my configuration?
The text was updated successfully, but these errors were encountered: