Change to mappings breaks canary routes

[andrazjelenc]

Describe the bug
On each modification of existing mappings, creating of new mapping or deleting a mapping, all canary mappings get an additional route. Additional route is seen on Diagnostics UI, in logs and even in generated envoy.json.

To Reproduce
Steps to reproduce the behavior:

1. Apply following manifest to the k8s cluster where you have Emissary Ingress installed. This will create one canary with two services and one basic one.

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: single
  namespace: develop
spec:
  ambassador_id:
  - ambassador
  host: another-host
  prefix: /single/
  service: random-service-A:8081
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: canary-one
  namespace: develop
spec:
  ambassador_id:
  - ambassador
  host: random-host
  prefix: /test/
  service: random-service-B:8080
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: canary-two
  namespace: develop
spec:
  ambassador_id:
  - ambassador
  host: random-host
  prefix: /test/
  service: random-service-C:8081

2. On Diagnostics UI inspect routes and see that everything is fine.

3. Modify property of one of the existing mappings. For example we can modify service on mapping single. Any modification would trigger the issue.

4. On Diagnostics UI inspect routes again and see additional mapping on canary.

6. I know that there is a bug on UI about weights (diag console's Weight column doesn't reflect Mapping weight attributes #2142). But this is not only frontend bug, this duplicated mapping is also seen in the logs if we run Emissary Ingress with environment variable AMBASSADOR_DEBUG set to "diagd".

ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <Mapping single.develop.1>: GID 21ca1689d6fa02eda3784f8eb1baae3ec29503f8 route_weight [0, 8, 22, 0, '/single/', 'GET', ':authority-another-host'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <InternalMapping ambassador.ambassador.1>: GID a2c3d327956b3acbfedb3662b5f50528dae653b5 route_weight [0, 26, 11, 0, '/ambassador/v0/check_alive', 'GET', ':authority-*'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: <InternalMapping ambassador.ambassador.1>: GID 65f77b4cfda77405d7353cc1c83de745ee74b141 route_weight [0, 26, 11, 0, '/ambassador/v0/check_ready', 'GET', ':authority-*'], resolver <KubernetesServiceResolver ir.resolver>
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRAuth: svc auth, weight 100, grpc False, ctx_name None, location auth.ambassador.1
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 1d55ae3c39e58c6de3c526c16c37ec66c83feb2d
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning calculated weight 33 of remaining weight 100 to each of 3 weightless mappings
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 33 to weightless mapping canary-one
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 66 to weightless mapping canary-two
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to weightless mapping canary-two
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 21ca1689d6fa02eda3784f8eb1baae3ec29503f8
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping single in group
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for a2c3d327956b3acbfedb3662b5f50528dae653b5
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping internal_liveness_probe_mapping in group
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: IRHTTPMappingGroup: normalizing weights for 65f77b4cfda77405d7353cc1c83de745ee74b141
ambassador 2024-02-22 07:15:43 diagd 3.8.1 [P32TAEW] DEBUG: Assigning weight 100 to single mapping internal_readiness_probe_mapping in group

Exactly the same issue appears if we create new mapping or delete one of the mappings. And on each change, all canary mappings get one more duplication.

If we then restart Emissary Ingress the duplicated routes disappear and everything is fine again.

Expected behavior
Creating, deleting or modifying mappings should not produce duplicated routes on canary mappings.

Versions (please complete the following information):

• Ambassador: 3.8.1
• Kubernetes environment AWS EKS
• Version 1.25