Kind cluster LB with metallb within corporate network

[tppalani]

What happened:

I have kind cluster and up and running in my local widows system, according to the document https://kind.sigs.k8s.io/docs/user/loadbalancer/ i have installed MetalLB manifest file and using podamn i have teken the Ip address range. if you see ip address range its 10 series and my system is connected to VPN network internally.

podman network inspect -f '{{range .Subnets}}{{if eq (len .Subnet.IP) 4}}{{.Subnet}}{{end}}{{end}}' kind
10.XX.XX.XX/24 

According to the document i have applied below configuration inside my kind cluster

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: example
  namespace: metallb-system
spec:
  addresses:
  - 10.XX.XX.XX-10.XX.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system

What you expected to happen:

After deploying sample yaml file i can see the pod and service with external Ip address but not able to perform curl or not able to access the adddress from browser too.

kind: Pod
apiVersion: v1
metadata:
  name: foo-app
  labels:
    app: http-echo
spec:
  containers:
  - name: foo-app
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=foo"
---
kind: Pod
apiVersion: v1
metadata:
  name: bar-app
  labels:
    app: http-echo
spec:
  containers:
  - name: bar-app
    image: hashicorp/http-echo:0.2.3
    args:
    - "-text=bar"
---
kind: Service
apiVersion: v1
metadata:
  name: foo-service
spec:
  type: LoadBalancer
  selector:
    app: http-echo
  ports:
  # Default port used by the image
  - port: 5678

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind v0.18.0 go1.20.2 windows/amd64):
• Runtime info: (use `host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  • cpuset
  • cpu
  • cpuacct
  • blkio
  • memory
  • devices
  • freezer
  • net_cls
  • perf_event
  • net_prio
  • hugetlb
  • pids
  • rdma
  • misc
    cgroupManager: cgroupfs
    cgroupVersion: v1
    conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
    cpuUtilization:
    idlePercent: 98.13
    systemPercent: 0.86
    userPercent: 1.02
    cpus: 12
    databaseBackend: sqlite
    distribution:
    distribution: fedora
    variant: container
    version: "39"
    eventLogger: journald
    freeLocks: 2030
    hostname: LDD4C6G3
    idMappings:
    gidmap: null
    uidmap: null
    kernel: 5.15.146.1-microsoft-standard-WSL2
    linkmode: dynamic
    logDriver: journald
    memFree: 6397227008
    memTotal: 16566702080
    networkBackend: netavark
    networkBackendInfo:
    backend: netavark
    dns:
    package: aardvark-dns-1.9.0-1.fc39.x86_64
    path: /usr/libexec/podman/aardvark-dns
    version: aardvark-dns 1.9.0
    package: netavark-1.9.0-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.9.0
    ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
    crun version 1.12
    commit: ce429cb2e277d001c2179df1ac66a470f00802ae
    rundir: /run/crun
    spec: 1.0.0
    +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
    os: linux
    pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231230.gf091893-1.fc39.x86_64
    version: |
    pasta 0^20231230.gf091893-1.fc39.x86_64
    Copyright Red Hat
    GNU General Public License, version 2 or later
    https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    remoteSocket:
    exists: true
    path: /run/podman/podman.sock
    security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
    serviceIsRemote: true
    slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
    slirp4netns version 1.2.2
    commit: 0ee2d87523e906518d34a6b423271e4826f71faf
    libslirp: 4.7.0
    SLIRP_CONFIG_VERSION_MAX: 4
    libseccomp: 2.5.3
    swapFree: 4294967296
    swapTotal: 4294967296
    uptime: 7h 7m 49.00s (Approximately 0.29 days)
    variant: ""
    plugins:
    authorization: null
    log:
  • k8s-file
  • none
  • passthrough
  • journald
    network:
  • bridge
  • macvlan
  • ipvlan
    volume:
  • local
    registries:
    search:
  • docker.io
    store:
    configFile: /usr/share/containers/storage.conf
    containerStore:
    number: 6
    paused: 0
    running: 2
    stopped: 4
    graphDriverName: overlay
    graphOptions:
    overlay.mountopt: nodev,metacopy=on
    graphRoot: /var/lib/containers/storage
    graphRootAllocated: 1081101176832
    graphRootUsed: 24834830336
    graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
    imageCopyTmpDir: /var/tmp
    imageStore:
    number: 16
    runRoot: /run/containers/storage
    transientStore: false
    volumePath: /var/lib/containers/storage/volumes
    version:
    APIVersion: 4.8.3
    Built: 1704291100
    BuiltTime: Wed Jan 3 19:41:40 2024
    GitCommit: ""
    GoVersion: go1.21.5
    Os: linux
    OsArch: linux/amd64
    Version: 4.8.3`):
• OS (e.g. from /etc/os-release):
• Kubernetes version: (use Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.1):
• Any proxies or other special environment settings?: system connected with VPN

[stmcginnis]

This is a limitation when running on Windows or macOS. From the linked loadbalancer documentation:

On macOS and Windows, docker does not expose the docker network to the host. Because of this limitation, containers (including kind nodes) are only reachable from the host via port-forwards

Due to the containers not being directly on the host network, there are some workarounds required to expose anything from inside the cluster.

[BenTheElder]

Right, we are exploring on the backburner how https://github.com/kubernetes-sigs/cloud-provider-kind could employ trickery to be reachable on the host on windows/mac, but reachability of the IP addresses from the host is a limitation of the docker / podman install, so we'd be building some custom workaround for the networking environment.

You can either employ some form of tunnel into the docker/podman VM or you have to run tests from another container (the most portable option)

[tppalani]

This is a limitation when running on Windows or macOS. From the linked loadbalancer documentation:

On macOS and Windows, docker does not expose the docker network to the host. Because of this limitation, containers (including kind nodes) are only reachable from the host via port-forwards

Due to the containers not being directly on the host network, there are some workarounds required to expose anything from inside the cluster.

sorry for the trouble do we need make any changes from my side to make it work?

[BenTheElder]

Please see the linked docs which discuss this in more detail. This is an expected limitation of docker / podman on windows when executing commands from the host and isn't kind specific.