Endpoint metadata #1559

AlexanderYastrebov · 2020-10-21T11:16:09Z

Is your feature request related to a problem? Please describe.

Load balancing algorithms (#557) require or would benefit from additional endpoint information besides currently available
scheme, ip address and port. For example slow start aka fadeIn algorithm (#1207) has to know when endpoint became available to receive traffic. Other algorithms need endpoint metrics and thus endpoint identity is required (ip address is not suitable as identity because it can be reused).

Useful endpoint metadata values are:

identity (e.g. targetRef.uid in k8s)
creation date (e.g. createdAt of targetRef in k8s)
name (e.g. name of targetRef in k8s)
weight (for weighted load balancing algorithms)

The mechanism for passing metadata should support possible future metadata extensions.

Describe the solution you would like

Endpoint metadata may be queried by skipper via infrastructure API and/or delivered to skipper proxy/algorithms via route definitions.

Within route definition the following ways are possible to pass endpoint metadata:

Marker filters

Example route:

r0: * 
-> endpointCreated("http://10.0.0.1:8080", "2020-12-18T15:30:00Z01:00") -> endpointId("http://10.0.0.1:8080", "6b6c2568-138b-11eb-835b-9f5a8a575060")
-> endpointCreated("http://10.0.0.2:8080", "2020-12-18T15:31:00Z01:00") -> endpointId("http://10.0.0.2:8080", "7daa5380-138b-11eb-b85d-7739503adcb7")
-> <roundRobin, "http://10.0.0.1:8080", "http://10.0.0.2:8080">;

pro: readily available
con: semantically incorrect (hacky, its not a filter after all)

Eskip backend definition

Example route:

r0: * 
-> <roundRobin, 
    "http://10.0.0.1:8080" ("createdAt=2020-12-18T15:30:00Z01:00", "id=6b6c2568-138b-11eb-835b-9f5a8a575060"), 
    "http://10.0.0.2:8080" ("createdAt=2020-12-18T15:31:00Z01:00", "id=7daa5380-138b-11eb-b85d-7739503adcb7")
>;

Syntax is subject to be discussed.

pro: established endpoint metadata as a first class route concept
con: requires eskip extension

Endpoint url parameters

It is possible to use url parameters because only scheme and host of endpoint url are currently used for routing.

Example route:

r0: * 
-> <roundRobin, 
    "http://10.0.0.1:8080?createdAt=2020-12-18T15:30:00Z01:00&id=6b6c2568-138b-11eb-835b-9f5a8a575060", 
    "http://10.0.0.2:8080?createdAt=2020-12-18T15:31:00Z01:00&id=7daa5380-138b-11eb-b85d-7739503adcb7"
>;

pro: readily available
pro: standard url parsing machinery is available
con: hacky?

Would you like to work on it?
I would like to validate, discuss and possibly extend the ideas first.

The text was updated successfully, but these errors were encountered:

szuecs · 2020-11-08T19:38:13Z

The idea to mix kubernetes IDs into backend id seem not reasonable to me.

it leaks data from dataclient
IMO IP address is enough and not explained well enough that this doesn’t make sense.
the IDs by kubernetes does not tell running nor ready (endpoints will be ready), how long this takes per pod is not clear.

AlexanderYastrebov · 2020-11-10T17:18:53Z

@szuecs

it leaks data from dataclient

Well, routes themselves are a leak from dataclient then :)

IMO IP address is enough and not explained well enough that this doesn’t make sense.

Not sure I understand, IP reuse appears to be a very real thing:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ip-reuse
spec:
  replicas: 0
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: busybox
        image: busybox
        command: ["sleep", "3600"]
        imagePullPolicy: IfNotPresent

$ kubectl scale deployment ip-reuse --replicas=10
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-6pc99   1/1     Running   0          12s   172.17.0.11   minikube   <none>           <none>
ip-reuse-5844f5c9c5-b6njc   1/1     Running   0          12s   172.17.0.7    minikube   <none>           <none>
ip-reuse-5844f5c9c5-d4g46   1/1     Running   0          12s   172.17.0.5    minikube   <none>           <none>
ip-reuse-5844f5c9c5-f5ldl   1/1     Running   0          12s   172.17.0.6    minikube   <none>           <none>
ip-reuse-5844f5c9c5-k7zn7   1/1     Running   0          12s   172.17.0.4    minikube   <none>           <none>
ip-reuse-5844f5c9c5-nh55r   1/1     Running   0          12s   172.17.0.10   minikube   <none>           <none>
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          12s   172.17.0.8    minikube   <none>           <none>
ip-reuse-5844f5c9c5-sbnhf   1/1     Running   0          12s   172.17.0.12   minikube   <none>           <none>
ip-reuse-5844f5c9c5-w5sv7   1/1     Running   0          12s   172.17.0.3    minikube   <none>           <none>
ip-reuse-5844f5c9c5-xlckj   1/1     Running   0          12s   172.17.0.9    minikube   <none>           <none>
$ kubectl scale deployment ip-reuse --replicas=1
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          30s   172.17.0.8   minikube   <none>           <none>
$ kubectl scale deployment ip-reuse --replicas=10
$ kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
ip-reuse-5844f5c9c5-2p4lh   1/1     Running   0          21s   172.17.0.10   minikube   <none>           <none>
ip-reuse-5844f5c9c5-64pkh   1/1     Running   0          21s   172.17.0.12   minikube   <none>           <none>
ip-reuse-5844f5c9c5-fqjzh   1/1     Running   0          21s   172.17.0.9    minikube   <none>           <none>
ip-reuse-5844f5c9c5-gmsnd   1/1     Running   0          21s   172.17.0.7    minikube   <none>           <none>
ip-reuse-5844f5c9c5-k7fmm   1/1     Running   0          21s   172.17.0.11   minikube   <none>           <none>
ip-reuse-5844f5c9c5-qxjl6   1/1     Running   0          54s   172.17.0.8    minikube   <none>           <none>
ip-reuse-5844f5c9c5-r996g   1/1     Running   0          21s   172.17.0.5    minikube   <none>           <none>
ip-reuse-5844f5c9c5-s7bpz   1/1     Running   0          21s   172.17.0.6    minikube   <none>           <none>
ip-reuse-5844f5c9c5-sjkmq   1/1     Running   0          21s   172.17.0.4    minikube   <none>           <none>
ip-reuse-5844f5c9c5-z9r9x   1/1     Running   0          21s   172.17.0.3    minikube   <none>           <none>

the IDs by kubernetes does not tell running nor ready (endpoints will be ready), how long this takes per pod is not clear.

I am not sure I can read it correctly, but what I propose are:
a) pass endpoint identity from infrastructure (IP address can not serve as an identity due to reuse)
b) pass endpoint creation time from infrastructure to reliably know when endpoint became ready to receive traffic (i.e. not guess it but trust k8s)

AlexanderYastrebov · 2021-10-26T13:58:52Z

In Kubernetes endpoint metadata might also include pod name which, if logged along with endpoint IP, would simplify correlation with application and event logs that contain pod name.

szuecs · 2021-10-26T14:01:31Z

Yes ep addresses has pod names:

subsets:
- addresses:
  - ip: 10.2.10.18
    nodeName: ip-172-31-21-23.eu-central-1.compute.internal
    targetRef:
      kind: Pod
      name: <pod-name>
      namespace: <pod-namespace>

Adds a special route with a predefined id `kube__metadata` that contains `False` predicate and no filters. The first argument of `False` predicate (which it ignores anyway) contains endpoint metadata encoded as JSON. Example (formatted): ``` kube__metadata: False("{ \"addresses\":[ { \"address\":\"1.2.3.4\", \"zone\":\"zoneA\", \"nodeName\":\"node1\", \"targetRef\":{ \"kind\":\"Pod\", \"name\":\"foo-app\", \"namespace\":\"default\", \"uid\":\"83a6408f-8a5b-405c-93f4-1199f7712956\" } }, ... ]") ``` This route could be used to obtain zone, node and pod names for a given address. E.g. EndpointRegistry postprocessos may check this route and store zone, node and pod name for each endpoint. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

Adds a special route with a predefined id `kube__metadata` that contains `False` predicate and no filters. The first argument of `False` predicate (which it ignores anyway) contains endpoint metadata encoded as JSON. Example (formatted): ``` kube__metadata: False("{ \"addresses\":[ { \"address\":\"1.2.3.4\", \"zone\":\"zoneA\", \"nodeName\":\"node1\", \"targetRef\":{ \"kind\":\"Pod\", \"name\":\"foo-app\", \"namespace\":\"default\", \"uid\":\"83a6408f-8a5b-405c-93f4-1199f7712956\" } }, ... ]") ``` This route could be used to obtain zone, node and pod names for a given address. E.g. EndpointRegistry postprocessor may check this route and store zone, node and pod name for each endpoint. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

Adds a special route with a predefined id `kube__metadata` that contains `False` predicate, no filters and endpoints metadata. Endpoints metadata is encoded as JSON via [data URI scheme](https://en.wikipedia.org/wiki/Data_URI_scheme) into the route backend address field. Example eskip: ``` kube__metadata: False() -> "data:application/json;base64,eyJhZGRyZXNzZXMiO..."; ... ``` This route could be used to obtain zone, node and pod names for a given address. A special route pre-processor detects and removes this route, decodes metadata and stores it as EndpointRegistry metrics. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

Adds a special route with a predefined id `kube__metadata` that contains `False` predicate, no filters and endpoints metadata. Endpoints metadata is encoded as JSON via [data URI scheme](https://en.wikipedia.org/wiki/Data_URI_scheme) into the route backend address field. Example eskip: ``` kube__metadata: False() -> "data:application/json;base64,eyJhZGRyZXNzZXMiO..."; ... ``` A special route pre-processor detects and removes this route, decodes metadata and stores it as EndpointRegistry metrics. The endpoint metrics then used to obtain endpoint zone, node and pod names for tracing and logging. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

Adds a special route with a predefined id `kube__metadata` that contains `False` predicate, no filters and endpoints metadata. Endpoints metadata is encoded as JSON via [data URI scheme](https://en.wikipedia.org/wiki/Data_URI_scheme) into the route backend address field. Example eskip: ``` kube__metadata: False() -> "data:application/json;base64,eyJhZGRyZXNzZXMiO..."; ... ``` A special route pre-processor detects and removes this route, decodes metadata and stores it as EndpointRegistry metrics. The endpoint metrics then used to obtain endpoint node and pod names for tracing and logging. TODO: - [ ] add flag to enable metadata route - [ ] tests Fixes #1559 Signed-off-by: Alexander Yastrebov <alexander.yastrebov@zalando.de>

AlexanderYastrebov mentioned this issue Oct 22, 2020

More fine-grained timeout handling #1041

Closed

AlexanderYastrebov linked a pull request Jan 11, 2024 that will close this issue

dataclients/kubernetes: add metadata route #2845

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Endpoint metadata #1559

Endpoint metadata #1559

AlexanderYastrebov commented Oct 21, 2020 •

edited

szuecs commented Nov 8, 2020

AlexanderYastrebov commented Nov 10, 2020

AlexanderYastrebov commented Oct 26, 2021

szuecs commented Oct 26, 2021

Endpoint metadata #1559

Endpoint metadata #1559

Comments

AlexanderYastrebov commented Oct 21, 2020 • edited

Marker filters

Eskip backend definition

Endpoint url parameters

szuecs commented Nov 8, 2020

AlexanderYastrebov commented Nov 10, 2020

AlexanderYastrebov commented Oct 26, 2021

szuecs commented Oct 26, 2021

AlexanderYastrebov commented Oct 21, 2020 •

edited