Acceptance test failure "When scaling by memoryused it should scale out ..."

[corlettb]

Subject of the issue

go_app isn't reaching the set memory to allow the acceptance tests to pass.

Test case:

https://github.com/cloudfoundry/app-autoscaler-release/blob/main/src/acceptance/app/dynamic_policy_test.go#L51

When using:

node_memory_limit: 256

  [2023-12-08 15:27:55.63 (UTC)]> cf bind-service autoscaler-1-dynamic-policy-69d05a732f89ac88 autoscaler-1-service-2fbfe96f7de3c61f -c {"instance_min_count":1,"instance_max_count":2,"scaling_rules":[{"metric_type":"memoryused","breach_duration_secs":60,"threshold":200,"operator":">=","cool_down_secs":60,"adjustment":"+1"},{"metric_type":"memoryused","breach_duration_secs":60,"threshold":115,"operator":"<","cool_down_secs":60,"adjustment":"-1"}]}
   
  Binding service instance autoscaler-1-service-2fbfe96f7de3c61f to app autoscaler-1-dynamic-policy-69d05a732f89ac88 in org ASATS-1-ORG-9f4f58e8ad4197bf / space ASATS-1-SPACE-fd4611c5a432f27d as ASATS-1-USER-ccbb9c446b3aff56...
  OK

...
  [2023-12-08 15:28:02.02 (UTC)]> curl -H Expect: -s https://autoscaler-1-dynamic-policy-69d05a732f89ac88.dev02.dev.cloudpipelineapps.digital/memory/200/5 -H X-Cf-App-Instance: 00855bde-2eaf-44cb-a6b7-53158e1ad9c4:0 -f --connect-timeout 5 --max-time 10 --retry 5 --retry-delay 0 --retry-max-time 15 
  {"memoryMiB":200,"minutes":5}

cf autoscaling-metrics autoscaler-1-dynamic-policy-69d05a732f89ac88 memoryused
Retrieving aggregated memoryused metrics for app autoscaler-1-dynamic-policy-69d05a732f89ac88...
Metrics Name     	Value     	Timestamp     	
memoryused       	198MB     	2023-12-08T15:32:49Z     	
memoryused       	198MB     	2023-12-08T15:32:09Z     	
memoryused       	198MB     	2023-12-08T15:31:29Z     	
memoryused       	198MB     	2023-12-08T15:30:49Z     	
memoryused       	198MB     	2023-12-08T15:30:09Z     	
memoryused       	198MB     	2023-12-08T15:29:29Z     	
memoryused       	198MB     	2023-12-08T15:28:49Z 

Memory does not reach 200MB to trigger scale up

When using:

node_memory_limit: 128

  [2023-12-08 15:34:41.51 (UTC)]> curl -H Expect: -s https://autoscaler-1-dynamic-policy-dcec3f630328ab50.dev02.dev.cloudpipelineapps.digital/memory/100/5 -H X-Cf-App-Instance: 27403589-5b59-4ce3-be47-6b8ba7c654f5:0 -f --connect-timeout 5 --max-time 10 --retry 5 --retry-delay 0 --retry-max-time 15 
  {"memoryMiB":100,"minutes":5}
...
 [2023-12-08 15:34:32.15 (UTC)]> cf bind-service autoscaler-1-dynamic-policy-dcec3f630328ab50 autoscaler-1-service-1d3e70a74f9240e6 -c {"instance_min_count":1,"instance_max_count":2,"scaling_rules":[{"metric_type":"memoryused","breach_duration_secs":60,"threshold":100,"operator":">=","cool_down_secs":60,"adjustment":"+1"},{"metric_type":"memoryused","breach_duration_secs":60,"threshold":70,"operator":"<","cool_down_secs":60,"adjustment":"-1"}]}
   
  Binding service instance autoscaler-1-service-1d3e70a74f9240e6 to app autoscaler-1-dynamic-policy-dcec3f630328ab50 in org ASATS-1-ORG-040deec86b0e6ce3 / space ASATS-1-SPACE-d80dc2be2e82029d as ASATS-1-USER-6ac21c8e731fa950...
  OK

cf autoscaling-metrics autoscaler-1-dynamic-policy-dcec3f630328ab50 memoryused
Retrieving aggregated memoryused metrics for app autoscaler-1-dynamic-policy-dcec3f630328ab50...
Metrics Name     	Value     	Timestamp     	
memoryused       	94MB      	2023-12-08T15:38:09Z     	
memoryused       	94MB      	2023-12-08T15:37:29Z     	
memoryused       	94MB      	2023-12-08T15:36:49Z     	
memoryused       	94MB      	2023-12-08T15:36:09Z     	
memoryused       	94MB      	2023-12-08T15:35:29Z 

Memory does not reach 100MB to trigger scale up

I don't know if this is a problem with the go_app or the metrics being reported.

Your environment

Version: v11.4.0
Database: postgres 13
Release: is using app-autoscaler-deployment.yml?
With custom operations
Acceptance tests: https://github.com/alphagov/paas-cf/blob/BC-app-autoscaler-base-manifest-switch/concourse/tasks/app-autoscaler-acceptance-tests-run.yml#L73

Steps to reproduce

Run the acceptance tests

Expected behaviour

Test passes.

Actual behaviour

Test fails