Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TimeSeriesPrediction status PredictPartial message is not all metric predicted #886

Open
wxuedong opened this issue Jan 11, 2024 · 1 comment

Comments

@wxuedong
Copy link

Describe the feature

K8s version 1.20.0
Craned version 0.11.0

Create ehpa yaml:

apiVersion: autoscaling.crane.io/v1alpha1
kind: EffectiveHorizontalPodAutoscaler
metadata:
  name: biz-service
  namespace: ptest
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: biz-service
  minReplicas: 1        
  maxReplicas: 18      
  scaleStrategy: Auto    
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
  prediction:
    predictionWindowSeconds: 3600   
    predictionAlgorithm:
      algorithmType: dsp
      dsp:
        sampleInterval: "60s"
        historyLength: "7d"

kubectl describe ehpa biz-service

Annotations:  autoscaling.crane.io/effective-hpa-current-metrics:
                - resource:
                    current:
                      averageUtilization: 12
                      averageValue: 516m
                    name: cpu
                  type: Resource
...
Status:
  Conditions:
    Last Transition Time:  2024-01-11T01:34:34Z
    Message:               Effective HPA is ready
    Reason:                EffectiveHorizontalPodAutoscalerReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-01-11T01:34:34Z
    Message:               not all metric predicted
    Reason:                PredictPartial
    Status:                False
    Type:                  PredictionReady
    Last Transition Time:  2024-01-11T01:34:34Z
    Message:               recommended size matches current size
    Reason:                ReadyForNewScale
    Status:                True
    Type:                  AbleToScale
    Last Transition Time:  2024-01-11T01:34:34Z
    Message:               the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
    Reason:                ValidMetricFound
    Status:                True
    Type:                  ScalingActive
    Last Transition Time:  2024-01-11T01:34:34Z
    Message:               the desired replica count is less than the minimum replica count
    Reason:                TooFewReplicas
    Status:                True
    Type:                  ScalingLimited
  Current Replicas:        16
  Expect Replicas:         16
Events:
  Type    Reason             Age    From                      Message
  ----    ------             ----   ----                      -------
  Normal  PredictionCreated  6m22s  effective-hpa-controller  Create TimeSeriesPrediction successfully
  Normal  HPACreated         6m22s  effective-hpa-controller  Create HorizontalPodAutoscaler successfully

kubectl describe hpa ehpa-biz-service

Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  12% (517m) / 50%
...

kubectl describe TimeSeriesPrediction ehpa-biz-service

Spec:
  Prediction Metrics:
    Algorithm:
      Algorithm Type:  dsp
      Dsp:
        Estimators:
        History Length:   2d
        Sample Interval:  60s
    Expression Query:
      Expression:             sum(irate(container_cpu_usage_seconds_total{namespace="ptest",pod=~"^biz-service-.*",container!=""}[3m]))
    Resource Identifier:      resource.cpu
    Type:                     ExpressionQuery
  Prediction Window Seconds:  3600
  Target Ref:
    API Version:  argoproj.io/v1alpha1
    Kind:         Rollout
    Name:         biz-service
    Namespace:    ptest
Status:
  Conditions:
    Last Transition Time:  2024-01-11T01:31:33Z
    Message:               not all metric predicted
    Reason:                PredictPartial
    Status:                False
    Type:                  Ready
  Prediction Metrics:
    Ready:                false
    Resource Identifier:  resource.cpu

prometheus data:

image

ps: biz-service has been running for more than three months

Why is there no crane_autoscaling_prediction Metric in HPA resources ?

Under what circumstances will there be the message "not all metric predicted"?

@yangyegang
Copy link

I have the same problem, have you solved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants