PR for implementation #1

webvictim · 2018-04-02T19:49:59Z

No description provided.

…ec-cni into implementation

eldios · 2018-04-03T10:37:18Z

Vagrantfile

+    master.vm.hostname = $master_hostname
+    master.vm.network "private_network", ip: $master_ip
+    master.vm.synced_folder "config", "/config"
+    master.vm.provider "virtualbox" do |v|


@webvictim would you be able to also add libvirt as a provider, please?
Many of us tend to stick to that as opposed to use virtualbox, nowadays.
Thanks

@eldios Sure - I'm just testing out the changes as I've had to use a different base image for libvirt. It's also hard to make libvirt work on a Mac so I've had to set it up on a Linux machine. I should be able to commit the updated code shortly.

webvictim · 2018-04-03T15:10:24Z

@eldios The Vagrantfile should work with libvirt now.

eldios · 2018-04-03T16:20:46Z

README.md

+vagrant@kube-master:~$ kubectl get nodes
+NAME          STATUS     ROLES     AGE       VERSION
+kube-master   NotReady   master    8m        v1.10.0


In the exercise specification it was not specified to use a different node as a master.
We're interested in knowing your ideas in going down this path.

Yeah, this was more of an oversight than a deliberate decision. I've changed it now so that there are three nodes total and the master runes on the first node.

eldios · 2018-04-03T16:21:10Z

README.md

+vagrant@kube-node1:~$ ping 10.128.3.4
+PING 10.128.3.4 (10.128.3.4) 56(84) bytes of data.
+64 bytes from 10.128.3.4: icmp_seq=1 ttl=63 time=0.436 ms


Why are you not using IPSEC for the master node?

knisbet · 2018-04-03T16:32:09Z

Hey Gus,

I've been doing some testing, and I found a sort of interesting issue. It looks to me like pod-to-pod traffic is bypassing the ipsec tunnels.

If you look at ipsec statusall the tunnel counters are all 0's.

   mesh-left[1]: ESTABLISHED 34 minutes ago, 192.168.64.10[192.168.64.10]...192.168.64.20[192.168.64.20]
   mesh-left[1]: IKEv2 SPIs: dd0ac2baaba214eb_i* 994afed63b77ec51_r, pre-shared key reauthentication in 16 minutes
   mesh-left[1]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
   mesh-left{7}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: cb1b31ac_i cd327d66_o
   mesh-left{7}:  AES_CBC_128/HMAC_SHA1_96, 0 bytes_i, 0 bytes_o, rekeying in 7 minutes
   mesh-left{7}:   10.128.1.0/24 === 10.128.2.0/24

The way I'm testing, is just using docker exec to ping a pod IP on another host:

docker exec -it 0027bfa14374 ping 10.128.3.2
PING 10.128.3.2 (10.128.3.2): 56 data bytes
64 bytes from 10.128.3.2: icmp_seq=0 ttl=62 time=0.557 ms
64 bytes from 10.128.3.2: icmp_seq=1 ttl=62 time=0.590 ms
64 bytes from 10.128.3.2: icmp_seq=2 ttl=62 time=0.620 ms
64 bytes from 10.128.3.2: icmp_seq=3 ttl=62 time=0.483 ms
64 bytes from 10.128.3.2: icmp_seq=4 ttl=62 time=0.844 ms
^C--- 10.128.3.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.483/0.619/0.844/0.121 ms

Also, tcpdump appears to pick up the traffic in the clear

root@kube-node1:~# tcpdump -vni eth1 icmp
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
08:51:59.261402 IP (tos 0x0, ttl 63, id 647, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.64.10 > 10.128.3.2: ICMP echo request, id 48, seq 98, length 64
08:51:59.261952 IP (tos 0x0, ttl 63, id 45133, offset 0, flags [none], proto ICMP (1), length 84)
    10.128.3.2 > 192.168.64.10: ICMP echo reply, id 48, seq 98, length 64
08:52:00.263974 IP (tos 0x0, ttl 63, id 731, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.64.10 > 10.128.3.2: ICMP echo request, id 48, seq 99, length 64
08:52:00.264661 IP (tos 0x0, ttl 63, id 45161, offset 0, flags [none], proto ICMP (1), length 84)
    10.128.3.2 > 192.168.64.10: ICMP echo reply, id 48, seq 99, length 64
08:52:01.265630 IP (tos 0x0, ttl 63, id 914, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.64.10 > 10.128.3.2: ICMP echo request, id 48, seq 100, length 64
08:52:01.266334 IP (tos 0x0, ttl 63, id 45381, offset 0, flags [none], proto ICMP (1), length 84)
    10.128.3.2 > 192.168.64.10: ICMP echo reply, id 48, seq 100, length 64

Cheers,

…nstead of policy

webvictim · 2018-04-04T01:22:44Z

@knisbet Thanks for the feedback and the detail. I've switched over to using proper routed IPSEC now with tunnels and marking the traffic explicitly as you suggested. My testing seems to show that this has fixed the issue and all the traffic is now being properly encrypted - the counters increase and I see ESP packets on the wire rather than clear ICMP.

eldios · 2018-04-04T10:30:19Z

@webvictim I still don't get why you're not using the VPN for all the kubernetes requests while you still address the 192.168.64.10:6443 directly.
Is there any specific reason? Do you think it's better to route traffic to the API server through the VPN or it's better to keep it outside of it? Could you explain your answer?
Thanks

webvictim · 2018-04-04T12:56:52Z

@eldios There are a couple of reasons I guess:

The project description doesn't explicitly say that API communication between nodes and master must be encrypted, it just covers pod-to-pod communications so I guess it's open to interpretation
API traffic between Kubernetes master and nodes is (AFAIK) already encrypted anyway so it didn't seem completely necessary to send it over the tunnels

With that said, we're not verifying the master's CA hash here so in theory a MITM attack would be possible. To counter this, we could modify the bootstrapping process to scrape the CA hash when it's output by kubeadm, pass it to the nodes somehow and verify it when they join. I didn't do this because it seemed a bit unreliable and messy - there's an open issue against kubeadm to provide a way to do this properly (kubernetes/kubeadm#659)

I think I also thought that it'd be much tougher to get Kubernetes to establish a cluster over the tunnels but actually, I just did it and it was really simple...

In terms of what I think would be better, on reflection it's probably better to just put everything inside the IPSEC tunnels as they could potentially be in a situation where the master isn't on the same network as the nodes, isn't routable or maybe the network isn't trusted.

What do you think?

eldios · 2018-04-04T14:12:46Z

I also think that it's cleaner to put everything inside IPSEC but the fact that the communication is encrypted via HTTPs makes it less mandatory.
You're answer is more or less what I expected 👍

webvictim · 2018-04-04T14:29:07Z

I've added a commit to send all Kubernetes communication over the IPSEC tunnels now as it's tidier.

webvictim added 5 commits April 2, 2018 20:33

Initial implementation

176af52

Merge branch 'implementation' of https://github.com/webvictim/k8s-ips…

dafdd56

…ec-cni into implementation

Implementation on branch

1ab927b

Merge branch 'implementation' of https://github.com/webvictim/k8s-ips…

39af00f

…ec-cni into implementation

Fix up curl commands

0133dd7

eldios reviewed Apr 3, 2018

View reviewed changes

Updates to support libvirt as a Vagrant provisioner

e37476a

eldios reviewed Apr 3, 2018

View reviewed changes

Updated to remove separate master node and use routed IPSEC tunnels i…

4efcc05

…nstead of policy

webvictim added 2 commits April 4, 2018 02:23

Remove unused variable

745cd71

Typo

a13c3dc

Use IPSEC for master <-> node Kubernetes API communication too

88ff93a

webvictim merged commit 2d4118f into master Apr 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR for implementation #1

PR for implementation #1

webvictim commented Apr 2, 2018

eldios Apr 3, 2018

webvictim Apr 3, 2018 •

edited

eldios Apr 3, 2018

webvictim commented Apr 3, 2018

eldios Apr 3, 2018

webvictim Apr 4, 2018

eldios Apr 3, 2018

webvictim Apr 4, 2018

knisbet commented Apr 3, 2018

webvictim commented Apr 4, 2018

eldios commented Apr 4, 2018 •

edited

webvictim commented Apr 4, 2018 •

edited

eldios commented Apr 4, 2018

webvictim commented Apr 4, 2018

PR for implementation #1

PR for implementation #1

Conversation

webvictim commented Apr 2, 2018

eldios Apr 3, 2018

Choose a reason for hiding this comment

webvictim Apr 3, 2018 • edited

Choose a reason for hiding this comment

eldios Apr 3, 2018

Choose a reason for hiding this comment

webvictim commented Apr 3, 2018

eldios Apr 3, 2018

Choose a reason for hiding this comment

webvictim Apr 4, 2018

Choose a reason for hiding this comment

eldios Apr 3, 2018

Choose a reason for hiding this comment

webvictim Apr 4, 2018

Choose a reason for hiding this comment

knisbet commented Apr 3, 2018

webvictim commented Apr 4, 2018

eldios commented Apr 4, 2018 • edited

webvictim commented Apr 4, 2018 • edited

eldios commented Apr 4, 2018

webvictim commented Apr 4, 2018

webvictim Apr 3, 2018 •

edited

eldios commented Apr 4, 2018 •

edited

webvictim commented Apr 4, 2018 •

edited