Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Routing loop with L2 Announced VIP #32195

Open
2 of 3 tasks
zviratko opened this issue Apr 25, 2024 · 0 comments
Open
2 of 3 tasks

Routing loop with L2 Announced VIP #32195

zviratko opened this issue Apr 25, 2024 · 0 comments
Labels
feature/l2-announcement kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.

Comments

@zviratko
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

Traffic destined to an L2 Announced VIP, but not pointing to an open port or even ICMP echo pointing to the IP, results in the traffic getting routed back through the default gateway, causing a loop and ending with Time To Live exceeded.

Cilium Version

1.15.3

Kernel Version

6.6.28

Kubernetes Version

1.30.0

Regression

No response

Sysdump

sysdump was too large so I uploaded it here
https://www.jabberwocky.cz/link/cilium-sysdump-20240425-172351.zip

Relevant log output

Port that is not used by a service:
17:26:02.636296 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.636747 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 61, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637036 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 59, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637138 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 57, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637202 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 55, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637253 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 53, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637302 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 51, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637350 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 49, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637401 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 47, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637486 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 45, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637559 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 43, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637647 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 41, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637698 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 39, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637767 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 37, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637808 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 35, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637848 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 33, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637888 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 31, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637927 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 29, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.637968 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 27, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638008 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 25, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638047 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 23, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638087 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 21, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638128 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 19, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638167 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 17, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638206 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 15, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638246 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 13, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638287 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 11, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638326 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 9, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638366 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 7, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638425 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 5, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638487 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 3, id 26238, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.50382 > 10.64.20.50.81: Flags [S], cksum 0x28e8 (incorrect -> 0xd64b), seq 57194114, win 26280, options [mss 8760,sackOK,TS val 1176371366 ecr 0,nop,wscale 7], length 0
17:26:02.638529 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 1, id 26238, offset 0, flags [DF], proto TCP (6), length 60)


vs port that is used by a service:

17:26:53.344525 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 16880, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.0.8.44890 > 10.64.20.50.80: Flags [S], cksum 0x28e8 (incorrect -> 0x2376), seq 1509253675, win 26280, options [mss 8760,sackOK,TS val 1176422074 ecr 0,nop,wscale 7], length 0
17:26:53.344816 00:00:36:27:8d:f4 > a2:ba:47:2f:2c:4f, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.64.20.50.80 > 10.64.0.8.44890: Flags [S.], cksum 0x28e8 (incorrect -> 0xff4f), seq 2976285363, ack 1509253676, win 17496, options [mss 8760,sackOK,TS val 4081852156 ecr 1176422074,nop,wscale 9], length 0
17:26:53.345065 a2:ba:47:2f:2c:4f > 00:00:36:27:8d:f4, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 16881, offset 0, flags [DF], proto TCP (6), length 52)
    10.64.0.8.44890 > 10.64.20.50.80: Flags [.], cksum 0x28e0 (incorrect -> 0x8e2b), ack 1, win 206, options [nop,nop,TS val 1176422075 ecr 4081852156], length 0

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct
@zviratko zviratko added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels Apr 25, 2024
@youngnick youngnick added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. feature/l2-announcement labels Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature/l2-announcement kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zviratko @youngnick