Abnormal Response Time Percentiles Reported After a Load Increase

[dilinade]

Prerequisites

• I am using the latest version of Locust
• I am reporting a bug, not asking a question

Description

Issue Description:
I've configured a custom load shape in Locust with a specific pattern. Initially, within the first 30 seconds, load is generated by 72 users and then spikes to approximately 136 users, nearly doubling the load, before returning back to the 72 users. However, after the spike, although the number of users returns to 72 (initial level), the response time percentiles are significantly lower compared to the initial normal load phase. This behavior is unexpected as I anticipate the response times to be similar to the initial phase of the normal load.

Steps to Reproduce:

Set up a custom load shape in Locust with the following characteristics:
Ramps up to 72 users within the first 30 seconds.
Spikes to approximately 136 users, doubling the load.
Returns to a stable load of 72 users after the spike.
Observe the response time percentiles during the spike and after the spike.

Expected Behavior:
Response time percentiles should remain consistent with the initial phase of the normal load throughout the test.
I can confirm that it's not originating from the web server application itself.

Actual Behavior:
Response time percentiles are significantly lower after the spike in user count, despite the number of users returning to normal levels.

Additional Information:

Using constant_throughtput(2) per user.
CURRENT_RESPONSE_TIME_PERCENTILE_WINDOW = 2

Command line

locust -f mylocustfile.py

Locustfile contents

import json
from locust import FastHttpUser, TaskSet, task, constant_throughput
from locust import LoadTestShape


class UserTasks(TaskSet):
    @task
    def get_root(self):
        self.client.get("/")


class WebsiteUser(FastHttpUser):
    wait_time = constant_throughput(2)
    tasks = [UserTasks]


class StagesShape(LoadTestShape):
    stages = [
{"duration": 30, "users": 72, "spawn_rate": 30},
{"duration": 60, "users": 136, "spawn_rate": 30},
{"duration": 90, "users": 72, "spawn_rate": 30}
]

    def tick(self):
        run_time = self.get_run_time()

        for stage in self.stages:
            if run_time < stage["duration"]:
                tick_data = (stage["users"], stage["spawn_rate"])
                return tick_data

        return None

Python version

3.8.10

Locust version

2.24.1

Operating system

Ubuntu

[cyberw]

Oh. That looks very strange.

• can you try with the default CURRENT_RESPONSE_TIME_PERCENTILE_WINDOW setting and see if it makes a difference?
• can you output the stats to a csv file and add it here? (Using --csv filename)

[dilinade]

Thank you for the response.

There's a bit difference when it's set to the default value of 10. However, response latencies remain lower than previous levels.

• 

• CSV with CURRENT_RESPONSE_TIME_PERCENTILE_WINDOW = 2
  output_2_new_stats.csv
  output_2_new_stats_history.csv

• csv stats output for CURRENT_RESPONSE_TIME_PERCENTILE_WINDOW = 10
  output_10_new_stats.csv
  output_10_new_stats_history.csv

[cyberw]

Ok, two things:

• CSV stats align with the UI, so it is not a presentation issue. Are you completely sure you dont actually HAVE lower response times after the ramp down? Perhaps you have some auto scaling going on?
• I think there might be an error in the presentation of average response times. It seems the line displays the average for the whole test and not just for the relevant interval. @andrewbaldwin44 can I bother you to take a look?