Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure full functionality of AntreaProxy with proxyAll enabled when kube-proxy presents #6308

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Ensure full functionality of AntreaProxy with proxyAll enabled when kube-proxy presents #6308

wants to merge 1 commit into from

Conversation

hongliangl
Copy link
Contributor

@hongliangl hongliangl commented May 8, 2024 •
edited

Depends on #6381

Resolve #6232

To ensure the full functionality of AntreaProxy with proxyAll enabled even when kube-proxy presents, there are some key changes when proxyAll is enabled:

The jump rules for the chains managed by Antrea, ANTREA-PREROUTING and ANTREA-OUTPUT in nat table, are installed by inserting instead of appending to bypass the chain KUBE-SERVICES performing Service DNAT managed by kube-proxy. Antrea ensures that the jump rules take precedence over those managed by kube-proxy.

The iptables rules of nat table chain ANTREA-PREROUTING are like below, and it is similar in chain ANTREA-OUTPUT.

1. -A ANTREA-PREROUTING -m comment --comment "Antrea: bypass external to kube Service traffic when kube Service Endpoint is not override" -d 10.96.0.1 -p tcp --dport 443 -j KUBE-SERVICES
2. -A ANTREA-PREROUTING -m comment --comment "Antrea: accept external to ClusterIP packets" -d 10.96.0.0/12 -j ACCEPT
3. -A ANTREA-PREROUTING -m comment --comment "Antrea: DNAT external to NodePort packets" -m set --match-set ANTREA-NODEPORT-IP dst,dst -j DNAT --to-destination 169.254.0.252
4. -A ANTREA-PREROUTING -m comment --comment "Antrea: accept external to LoadBalancer packets" -m set --match-set ANTREA-LOADBALANCER-IP dst,dst -j ACCEPT
5. -A ANTREA-PREROUTING -m comment --comment "Antrea: accept external to externalIP packets" -m set --match-set ANTREA-EXTERNAL-IP dst,dst -j ACCEPT
  • Rule 1 is to bypass kube Service when kubeAPIServerOverride is not set.
  • Rule 2 is to accept ClusterIP traffic and bypass chain KUBE-SERVICES. The Service CIDR is got from serviceCIDRProvider. TODO: whether to sync iptables immediately after the update of Service CIDR.
  • Rule 3 is to accept NodePort traffic and bypass chain KUBE-SERVICES. This is not a new rule, and we have this rule before.
  • Rule 4 is to accept LoadBalancer traffic and bypass chain KUBE-SERVICES. We use an ipset to match the traffic.
  • Rule 5 is similar to rule 4.

The iptables rules of raw table chainANTREA-PREROUTING are like below:

1. -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track incoming encapsulation packets" -m udp -p udp --dport 6081 -m addrtype --dst-type LOCAL -j NOTRACK
2. -A ANTREA-PREROUTING -m comment --comment "Antrea: drop Pod multicast traffic forwarded via underlay network" -m set --match-set CLUSTER-NODE-IP src -d 224.0.0.0/4 -j DROP
3. -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track external to LoadBalancer request packets" -m set --match-set ANTREA-LOADBALANCER-IP dst,dst -j NOTRACK
4. -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track external to externalIP request packets" -m set --match-set ANTREA-EXTERNAL-IP dst,dst -j NOTRACK
5. -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track external to LoadBalancer reply packets" -m set --match-set ANTREA-LOADBALANCER-IP src,src -j NOTRACK
6. -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track external to externalIP reply packets" -m set --match-set ANTREA-EXTERNAL-IP src,src -j NOTRACK
  • Rules 1-2 are not new rules.
  • Rules 3-6 are to bypass conntrack for LoadBalancer/externalIP traffic.

TODO:

  • In iptables nat table, ensure that Antrea iptables chain ANTREA-PREROUTING and ANTREA-OUTPUT are in the first rule of corresponding default iptables chains to skip kube-proxy chains.
  • Add unit tests
  • Add e2e tests? Whether to add a new Kind test [proxyAll=true, LoadBalancerMode=DSR, NodeIPAM=true, kube-proxy presents]
  • Update related document.
  • Update manifests
  • PR / commit description
  • Benchmark of LoadBalancer

@hongliangl hongliangl requested review from tnqn and antoninbas May 8, 2024 09:22
@hongliangl hongliangl added area/proxy Issues or PRs related to proxy functions in Antrea action/release-note Indicates a PR that should be included in release notes. area/OS/linux Issues or PRs related to the Linux operating system. labels May 8, 2024
@hongliangl hongliangl added this to the Antrea v2.1 release milestone May 8, 2024
@hongliangl hongliangl force-pushed the 20240429-bypass-kube-proxy branch 2 times, most recently from ece78aa to e5a6949 Compare May 9, 2024 01:38
@hongliangl hongliangl requested review from tnqn and antoninbas May 9, 2024 09:43
@hongliangl hongliangl marked this pull request as ready for review May 9, 2024 09:44
@hongliangl hongliangl marked this pull request as draft May 17, 2024 05:48
@hongliangl hongliangl force-pushed the 20240429-bypass-kube-proxy branch 2 times, most recently from e4b5ccb to e8b24b1 Compare May 20, 2024 11:23
@hongliangl hongliangl marked this pull request as ready for review May 20, 2024 11:27
@hongliangl hongliangl force-pushed the 20240429-bypass-kube-proxy branch 2 times, most recently from 7a890bc to 1b705a5 Compare May 21, 2024 08:50
@hongliangl
Copy link
Contributor Author

/test-all

@hongliangl hongliangl force-pushed the 20240429-bypass-kube-proxy branch 3 times, most recently from 2254f1d to 716b05a Compare May 30, 2024 03:20
@hongliangl hongliangl mentioned this pull request May 30, 2024
Open
@hongliangl
Ensure full functionality of AntreaProxy with proxyAll enabled when k…
5780454 
…ube-proxy presents

Signed-off-by: Hongliang Liu <lhongliang@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tnqn tnqn Awaiting requested review from tnqn

@antoninbas antoninbas Awaiting requested review from antoninbas

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
action/release-note Indicates a PR that should be included in release notes. area/OS/linux Issues or PRs related to the Linux operating system. area/proxy Issues or PRs related to proxy functions in Antrea
Projects
None yet
Milestone
Antrea v2.1 release
Development

Successfully merging this pull request may close these issues.

Make proxyAll/LoadBalancerModeDSR work with kube-proxy present
1 participant
@hongliangl