Artillery doesn't override phases in environments

[milosgajdos]

Version info:

VERSION INFO:

Artillery: 2.0.11
Node.js:   v18.20.2
OS:        linux

Running this command:

docker run -it --rm -v ${PWD}:/scripts artilleryio/artillery:2.0.11 run --environment local /scripts/script.yaml

I expected to see this happen:

I was hoping the phases in the specified environment would FULLY override the ones defined in the default target. Instead, it would appear if your default target settings contain a list of phases and your environment defines say, only one phase, only the first one from the default target is replaced and the remaining two are happily used in the load test. This seems like a very strange UX.

If your environment defines number of phases larger than the default target they all override the default target

Instead, this happened:

I can see in the output the default target phases are happily being used

Phase started: Local test (index: 0, duration: 30s) 16:25:51(+0000)
Phase completed: Local test (index: 0, duration: 30s) 16:26:21(+0000)
Phase started: Ramp up (index: 1, duration: 60s) 16:26:21(+0000)
Phase completed: Ramp up (index: 1, duration: 60s) 16:27:21(+0000)
Phase completed: Spike up (index: 2, duration: 30s) 16:27:51(+0000)
...

Files being used:

config:
  target: https://foo-stage
  phases:
      - duration: 60
        arrivalRate: 1
        rampTo: 5
        name: Warm up
      - duration: 60
        arrivalRate: 5
        rampTo: 10
        name: Ramp up
      - duration: 30
        arrivalRate: 10
        rampTo: 30
        name: Spike up
  environments:
    local:
      target: http://localhost:3000
      phases:
        - duration: 30
          arrivalRate: 1
          rampTo: 10
          name: Local test
  variables:
    token: '{{ $env.API_KEY }}'
    json_ct: 'application/json'
    version: v1
 
scenarios:
  - name: '/{{ version }}/foos'
    flow:
      - get:
          url: '/{{ version }}/foos'
          headers:
            authorization: 'Bearer {{ token }}'

[hassy]

thanks for the bug report @milosgajdos!

[hassy]

Took a bit of time to refresh my memory - the behavior is inherited from Lodash merge, the relevant code is here: https://github.com/artilleryio/artillery/blob/main/packages/artillery/lib/util.js#L129 It won't be easy to change without the risk of breaking existing scripts relying on that exact behavior, as that code has been there for >4 years at this point.

For your specific scenario, the best thing to do would be to extract the phase definition under config into its own environment, and switch between the two as needed.

[milosgajdos]

Interesting! Thanks for looking into that and suggesting a workaround. I still feel like this isn't particularly intuitive but I get that maintaining compatibility with the existing scripts is important. Do you want me to close this report or should I keep it open? It would be nice if this was documented in the docs better, maybe.

[milosgajdos]

For your specific scenario, the best thing to do would be to extract the phase definition under config into its own environment, and switch between the two as needed.

So you're suggesting ditching the default phases config and simply focus fully on environments only i.e. something like this:

config:
  environments:
    default:
      target: https://foo-stage
      phases:
          - duration: 60
            arrivalRate: 1
            rampTo: 5
            name: Warm up
          - duration: 60
            arrivalRate: 5
            rampTo: 10
            name: Ramp up
          - duration: 30
            arrivalRate: 10
            rampTo: 30
            name: Spike up
    local:
      target: http://localhost:3000
      phases:
        - duration: 30
          arrivalRate: 1
          rampTo: 10
          name: Local test
  variables:
    token: '{{ $env.API_KEY }}'
    json_ct: 'application/json'
    version: v1

scenarios:
  - name: '/{{ version }}/foos'
    flow:
      - get:
          url: '/{{ version }}/foos'
          headers:
            authorization: 'Bearer {{ token }}'

And then drive the load test via --environment <default|local>

[hassy]

@milosgajdos yep something like that! and if that doesn't work either then we're in trouble 😆