Running K3s Mutlinode cluster with Cilium and LoxiLB

[JLCode-tech]

I'm currently trying to get my head around how to deploy this.

k3s testing lab.pdf

I want to replace metallb inside the cluster with loxilb. I currently have cilium as the CNI. I have read through all of the blogs and how to's however it seems loxilb is always outside of the cluster ?

Any guidance would be great.

[backguynn]

LoxiLB is currently used as external load balancer.
This means that the topology configuration requires a node for the load balancer separate from the k8s cluster.
The topology in this post may serve as examples.

Therefore, it is recommended that external packets whose destination is the IP of the load balancer service pass through the LoxiLB node.
If you want to keep your LBPool bandwidth, you could also set it up like this:

After configuring the topology, you can control the loxilb by deploying kube-loxilb to k8s and creating a load balancer service.

[UltraInstinct14]

I think OP's question was already answered in the previous post. But since there are many similar requests, we will try to give a quick way for users to create and validate a "lab" topology with Cilium and Loxilb in approx. 10 mins. The following instructions will use a single node with multiple vms for loxilb, k3s based master/workers (with cilium) alongwith some sample apps deployment.

Prerequisites:

• Decent multi-core system with 8GB RAM or more
• Running any Linux OS
• Virtualbox and Vagrant packages installed

Steps :

1. Create the lab test-setup

git clone https://github.com/loxilb-io/loxilb.git
cd loxilb/cicd/k3s-cilium-cluster/
./config.sh

2. This will create the following topology

3. Check k3s cluster is up and running

$ vagrant status 
Current machine states: 

loxilb                    running (virtualbox)
master                    running (virtualbox)
worker1                   running (virtualbox)

# Check rules and other things in loxilb node
$  vagrant ssh loxilb -c "sudo docker exec -it loxilb loxicmd get lb -o wide"

|  EXTERNAL IP  | SECONDARY IPS | PORT  | PROTOCOL | BLOCK | SELECT |  MODE  |  ENDPOINT IP  | TARGET PORT | WEIGHT | STATE  |
|---------------|---------------|-------|----------|-------|--------|--------|---------------|-------------|--------|--------|
| 123.123.123.1 |               | 55002 | tcp      |     0 | rr     | onearm | 192.168.80.10 |       32064 |     10 | -      |
| 123.123.123.1 |               | 55003 | udp      |     0 | rr     | onearm | 192.168.80.10 |       30512 |     10 | active |

# Check k3s  svc, pods etc
$ vagrant ssh master -c "sudo kubectl get pods -A"
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   cilium-operator-66d8bc46f6-bsxr4          1/1     Running   0          8m7s
kube-system   cilium-6g7p2                              1/1     Running   0          8m7s
kube-system   local-path-provisioner-76d776f6f9-2p8jg   1/1     Running   0          8m7s
kube-system   coredns-59b4f5bbd5-kgnx4                  1/1     Running   0          8m7s
kube-system   kube-loxilb-754d58b99d-r5hm2              1/1     Running   0          7m15s
kube-system   metrics-server-7b67f64457-msp7d           1/1     Running   0          8m7s
default       udp-test                                  1/1     Running   0          5m46s
default       nginx-test                                1/1     Running   0          5m46s

$ vagrant ssh master -c "sudo kubectl get svc"
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)           AGE
kubernetes   ClusterIP      10.43.0.1       <none>          443/TCP           9m11s
nginx-lb1    LoadBalancer   10.43.86.30     123.123.123.1   55002:32064/TCP   6m33s
udp-lb1      LoadBalancer   10.43.242.128   123.123.123.1   55003:30512/UDP   6m33s

$ vagrant ssh master -c "sudo kubectl get nodes"
NAME      STATUS   ROLES                  AGE     VERSION
master    Ready    control-plane,master   30m     v1.26.5+k3s1
worker1   Ready    <none>                 29m   v1.26.5+k3s1

4. Check external traffic (From the node hosting the VMs)

curl -s --connect-timeout 10 http://123.123.123.1:55002

5. Explore vagrant scripts for further automation
   cicd/k3s-cilium-cluster/Vagrantfile - The vagrant file for VM definitions
   cicd/k3s-cilium-cluster/loxilb.sh - Vagrant's helper script to install loxilb node
   cicd/k3s-cilium-cluster/master.sh - Vagrant's helper script for k3s control node installation
   cicd/k3s-cilium-cluster/worker.sh - Vagrant's helper script for k3s worker node installation

Thanks !!

[JLCode-tech]

This is great ! thank you.