HTTPProxy: add cluster outlierDetection #5575

yangyy93 · 2023-07-24T08:41:29Z

add cluster outlierDetection.
ref: #5317
design: #5460

codecov · 2023-07-24T09:08:45Z

Codecov Report

Attention: Patch coverage is 75.29412% with 21 lines in your changes are missing coverage. Please review.

Project coverage is 81.53%. Comparing base (5f1b981) to head (3a2f324).
Report is 82 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5575      +/-   ##
==========================================
- Coverage   81.56%   81.53%   -0.04%     
==========================================
  Files         133      133              
  Lines       15801    15886      +85     
==========================================
+ Hits        12888    12952      +64     
- Misses       2617     2629      +12     
- Partials      296      305       +9

Files	Coverage Δ
cmd/contour/servecontext.go	`85.98% <100.00%> (+0.03%)`	⬆️
internal/dag/dag.go	`98.40% <ø> (ø)`
internal/envoy/v3/cluster.go	`96.62% <100.00%> (+0.29%)`	⬆️
internal/protobuf/helpers.go	`92.30% <100.00%> (+0.64%)`	⬆️
pkg/config/parameters.go	`88.03% <ø> (ø)`
cmd/contour/serve.go	`22.74% <50.00%> (+0.08%)`	⬆️
internal/dag/httpproxy_processor.go	`91.11% <33.33%> (-0.30%)`	⬇️
internal/envoy/cluster.go	`94.66% <77.77%> (-5.34%)`	⬇️
internal/dag/policy.go	`93.57% <67.56%> (-1.96%)`	⬇️

github-actions · 2023-08-16T00:17:26Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2023-09-06T00:18:07Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2023-10-06T00:18:37Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

yangyy93 · 2023-10-08T01:55:41Z

not stale

davinci26 · 2023-10-23T12:55:48Z

Drive by comment:

Can we also add https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#config-bootstrap-v3-clustermanager-outlierdetection

From the Envoy docs

A log of outlier ejection events can optionally be produced by Envoy. This is extremely useful during daily operations since global stats do not provide enough information on which hosts are being ejected and for what reasons. The log is structured as protobuf-based dumps of OutlierDetectionEvent messages. Ejection event logging is configured in the Cluster manager outlier detection configuration.

Which makes the above o11y seem like a good idea

davinci26 · 2023-10-24T19:43:48Z

Also it would be nice if we could have this also applied to extension services as well such ext_auth

davinci26

LGTM, I would love to have the logs mentioned above and some small changes

davinci26 · 2023-10-26T14:37:31Z

apis/projectcontour/v1/httpproxy.go

+}
+
+// OutlierDetection defines the configuration for outlier detection on a service.
+type OutlierDetection struct {


a link to the Envoy docs would be nice here

Yes, I will add a link to envoy's documentation above the comment

davinci26 · 2023-10-26T14:51:13Z

internal/dag/policy.go

+	var interval, baseEjectionTime, maxEjectionTime, maxEjectionTimeJitter time.Duration
+
+	if outlierDetection.Interval != nil {
+		interval, err = time.ParseDuration(ref.Val(outlierDetection.Interval, "10s"))


export the constant here into a variable?

I don’t quite understand what needs to be modified. Could you please explain it in detail?

I am suggesting we should take "10s" and all other default constants to cost so we can easily inspect those

davinci26 · 2023-10-26T14:51:21Z

internal/dag/policy.go

+	}
+
+	if outlierDetection.BaseEjectionTime != nil {
+		baseEjectionTime, err = time.ParseDuration(ref.Val(outlierDetection.BaseEjectionTime, "30s"))


davinci26 · 2023-10-26T14:51:29Z

internal/dag/policy.go

+		out.BaseEjectionTime = baseEjectionTime
+	}
+
+	if outlierDetection.MaxEjectionTime != nil {


yangyy93 · 2023-10-27T02:20:02Z

Drive by comment:

Can we also add https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#config-bootstrap-v3-clustermanager-outlierdetection

From the Envoy docs

A log of outlier ejection events can optionally be produced by Envoy. This is extremely useful during daily operations since global stats do not provide enough information on which hosts are being ejected and for what reasons. The log is structured as protobuf-based dumps of OutlierDetectionEvent messages. Ejection event logging is configured in the Cluster manager outlier detection configuration.

Which makes the above o11y seem like a good idea

thanks for your review, this is a good idea, I will add a switch to control whether logging is turned on or not.
outlier ejection events

yangyy93 · 2023-10-27T02:23:04Z

Also it would be nice if we could have this also applied to extension services as well such ext_auth

Now I just add outlier detection configuration on the services to be routed. I think we can discuss whether we should add relevant configuration on extension services.

davinci26 · 2023-10-27T12:41:01Z

Now I just add outlier detection configuration on the services to be routed. I think we can discuss whether we should add relevant configuration on extension services.

Works for me!

github-actions · 2023-11-11T00:18:44Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions · 2024-01-04T00:20:28Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

# Conflicts: # cmd/contour/serve.go # internal/dag/dag.go # internal/dag/httpproxy_processor.go # internal/dag/policy.go # internal/dag/policy_test.go

github-actions · 2024-01-27T00:18:24Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

# Conflicts: # site/content/docs/main/config/api-reference.html

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions · 2024-02-20T00:19:02Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-03-13T00:40:47Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

# Conflicts: # cmd/contour/serve.go # internal/dag/policy_test.go # internal/protobuf/helpers.go

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions · 2024-04-12T00:19:39Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

izturn · 2024-04-28T06:42:41Z

any progress?

github-actions · 2024-05-13T00:21:33Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

yangyy93 requested a review from a team as a code owner July 24, 2023 08:41

yangyy93 requested review from tsaarni and stevesloka and removed request for a team July 24, 2023 08:41

yangyy93 added the release-note/minor A minor change that needs about a paragraph of explanation in the release notes. label Jul 24, 2023

izturn requested review from skriss and sunjayBhatia and removed request for tsaarni and stevesloka August 1, 2023 03:02

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 16, 2023

yangyy93 marked this pull request as draft August 22, 2023 09:20

yangyy93 removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 22, 2023

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 6, 2023

github-actions bot closed this Oct 6, 2023

yangyy93 reopened this Oct 8, 2023

github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 9, 2023

yangyy93 force-pushed the outlier_detection branch from 1cf3340 to c0d066a Compare October 18, 2023 03:00

yangyy93 marked this pull request as ready for review October 20, 2023 03:06

davinci26 suggested changes Oct 26, 2023

View reviewed changes

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 19, 2023

update

2b9111f

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 4, 2024

yangyy93 self-assigned this Jan 9, 2024

yangyy93 removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 9, 2024

Merge branch 'main' into outlier_detection

0dc1120

# Conflicts: # cmd/contour/serve.go # internal/dag/dag.go # internal/dag/httpproxy_processor.go # internal/dag/policy.go # internal/dag/policy_test.go

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 27, 2024

yangyy93 added 4 commits February 2, 2024 14:28

Merge branch 'main' into outlier_detection

d6a9341

# Conflicts: # site/content/docs/main/config/api-reference.html

resolve code conflicts

096857d

Signed-off-by: yangyang <yang.yang@daocloud.io>

fix test

8facf74

Signed-off-by: yangyang <yang.yang@daocloud.io>

fix lint

7c132c9

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 20, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 13, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 13, 2024

yangyy93 added 2 commits March 22, 2024 10:17

Merge branch 'main' into outlier_detection

a556f32

# Conflicts: # cmd/contour/serve.go # internal/dag/policy_test.go # internal/protobuf/helpers.go

conflict resolution

3a2f324

Signed-off-by: yangyang <yang.yang@daocloud.io>

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 12, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 13, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPProxy: add cluster outlierDetection #5575

HTTPProxy: add cluster outlierDetection #5575

yangyy93 commented Jul 24, 2023

codecov bot commented Jul 24, 2023 •

edited

github-actions bot commented Aug 16, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Oct 6, 2023

yangyy93 commented Oct 8, 2023

davinci26 commented Oct 23, 2023

davinci26 commented Oct 24, 2023

davinci26 left a comment

davinci26 Oct 26, 2023

yangyy93 Oct 27, 2023

davinci26 Oct 26, 2023

yangyy93 Oct 27, 2023

davinci26 Oct 27, 2023

davinci26 Oct 26, 2023

davinci26 Oct 26, 2023

yangyy93 commented Oct 27, 2023

yangyy93 commented Oct 27, 2023

davinci26 commented Oct 27, 2023

github-actions bot commented Nov 11, 2023

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 27, 2024

github-actions bot commented Feb 20, 2024

github-actions bot commented Mar 13, 2024

github-actions bot commented Apr 12, 2024

izturn commented Apr 28, 2024

github-actions bot commented May 13, 2024

HTTPProxy: add cluster outlierDetection #5575

Are you sure you want to change the base?

HTTPProxy: add cluster outlierDetection #5575

Conversation

yangyy93 commented Jul 24, 2023

codecov bot commented Jul 24, 2023 • edited

Codecov Report

github-actions bot commented Aug 16, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Oct 6, 2023

yangyy93 commented Oct 8, 2023

davinci26 commented Oct 23, 2023

davinci26 commented Oct 24, 2023

davinci26 left a comment

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 commented Oct 27, 2023

yangyy93 commented Oct 27, 2023

davinci26 commented Oct 27, 2023

github-actions bot commented Nov 11, 2023

github-actions bot commented Jan 4, 2024

github-actions bot commented Jan 27, 2024

github-actions bot commented Feb 20, 2024

github-actions bot commented Mar 13, 2024

github-actions bot commented Apr 12, 2024

izturn commented Apr 28, 2024

github-actions bot commented May 13, 2024

codecov bot commented Jul 24, 2023 •

edited