Add Windows support to kube-proxy #36079

jbhurat · 2016-11-02T18:49:16Z

What this PR does / why we need it:
This is the first stab at supporting kube-proxy (userspace mode) on Windows

Which issue this PR fixes :
fixes #30278

Special notes for your reviewer:
The MVP uses netsh portproxy to redirect traffic from ServiceIP:ServicePort to a LocalIP:LocalPort.
For the next version we are expecting to have guidance from Microsoft Container Networking team.

Limitations:
Current implementation does not support DNS queries over UDP as netsh portproxy currently only supports TCP. We are working with Microsoft to remediate this.

cc: @brendandburns @dcbw

Release note:

This change is

googlebot · 2016-11-02T18:49:21Z

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

pires · 2016-11-02T18:55:20Z

I'm okay with these commits being contributed to Google.

Also, sorry @dcbw for cc'ing you but your feedback on #sig-network was important.

k8s-ci-robot · 2016-11-02T19:15:48Z

Jenkins unit/integration failed for commit 93945b5. Full PR test history.

The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-02T22:21:48Z

Jenkins verification failed for commit 1bcba7d. Full PR test history.

The magic incantation to run this job again is @k8s-bot verify test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

smarterclayton · 2016-11-03T00:38:19Z

Is this on the 1.5 release feature list? Otherwise it's unlikely this will make feature freeze on monday because review bandwidth is fairly limited.

@kubernetes/sig-network

smarterclayton · 2016-11-03T00:39:25Z

pkg/util/netsh/netsh.go

+// GetInterfaceToAddIP returns the interface name where Service IP needs to be added
+// IP Address needs to be added for netsh portproxy to redirect traffic
+// Reads Environment variable INTERFACE_TO_ADD_SERVICE_IP, if it is not defined then "vEthernet (HNSTransparent)" is returned
+func (runner *runner) GetInterfaceToAddIP() string {


Is there a reason you are using an environment variable for this? Generally this would be a flag to the proxy (we don't use environment variables except in rare conditions).

smarterclayton · 2016-11-03T00:41:31Z

pkg/proxy/userspace/proxier.go

@@ -260,8 +263,10 @@ func CleanupLeftovers(ipt iptables.Interface) (encounteredError bool) {

 // Sync is called to immediately synchronize the proxier state to iptables
 func (proxier *Proxier) Sync() {
-	if err := iptablesInit(proxier.iptables); err != nil {
-		glog.Errorf("Failed to ensure iptables: %v", err)
+	if runtime.GOOS != "windows" {


Seems like these are details of the proxier implementation (or should be hidden under an interface method like .init()).

Yeah, I'd agree that there are too many:

if runtime.GOOS != windows

for my liking, I'd prefer to see both iptables and netsh implement some sort of init() interface and just call:"

if iptables != nil { iptables.init() }

or something like that...

jbhurat · 2016-11-03T01:38:59Z

This is one of the PRs to introduce Windows Server Containers support for k8s and which will be introduced in release 1.5 in alpha (kubernetes/enhancements#116)

brendandburns · 2016-11-03T03:07:15Z

pkg/proxy/userspace/proxier.go

@@ -260,8 +263,10 @@ func CleanupLeftovers(ipt iptables.Interface) (encounteredError bool) {

 // Sync is called to immediately synchronize the proxier state to iptables
 func (proxier *Proxier) Sync() {
-	if err := iptablesInit(proxier.iptables); err != nil {
-		glog.Errorf("Failed to ensure iptables: %v", err)
+	if runtime.GOOS != "windows" {


Yeah, I'd agree that there are too many:

if runtime.GOOS != windows

for my liking, I'd prefer to see both iptables and netsh implement some sort of init() interface and just call:"

if iptables != nil { iptables.init() }

or something like that...

brendandburns · 2016-11-03T03:08:21Z

pkg/proxy/userspace/proxier_windows.go

+		return nil
+	}
+
+	/*	if local, err := isLocalIP(portal.ip); err != nil {


delete dead code.

brendandburns · 2016-11-03T03:12:42Z

vendor/BUILD

@@ -90,6 +90,7 @@ go_library(
        "github.com/Azure/azure-sdk-for-go/arm/network/version.go",
        "github.com/Azure/azure-sdk-for-go/arm/network/virtualnetworkgatewayconnections.go",
        "github.com/Azure/azure-sdk-for-go/arm/network/virtualnetworkgateways.go",
+        "github.com/Azure/azure-sdk-for-go/arm/network/virtualnetworkpeerings.go",


I don't think this is actually used in this PR? Revert?

It's not used in this PR but it seems hack/update-bazel.sh wants that to be there.

brendandburns · 2016-11-03T03:13:06Z

@smarterclayton I'm happy to take the review for this one.

thockin · 2016-11-03T04:12:25Z

This is awesome, but I am afraid I am not going to have time to review it
this week, which means it is going to miss 1.5 almost certainly.

On Thu, Nov 3, 2016 at 4:13 AM, Brendan Burns notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton I'm happy to take the
review for this one.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#36079 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVGEomjhkgwzghl8b7p1nUJBO2b8rks5q6VFigaJpZM4KnnN3
.

brendandburns · 2016-11-03T04:14:24Z

@thockin I'm happy to do the review, is your review required?

brendandburns · 2016-11-03T04:16:53Z

@thockin fwiw, I did a first pass and it's nearly entirely additive, with some minor refactoring, I don't see it interfering w/ the existing iptables on linux stuff.

thockin

From a very high level: I'd rather see this as a whole distinct proxy-mode. The userspace proxy is exceedingly fragile and is overdue for deprecation. I don't want a) to change it b) to keep it because Windows needs it. I'd much rather see this copied and customized for Windows. Less conditionals, less scaryness, and more freedom for you all.

Is there a feture-repo issue to track this and other windows work?

thockin · 2016-11-03T04:14:39Z

pkg/util/netsh/netsh.go

+limitations under the License.
+*/
+
+package netsh


REALLY needs a pkg comment

thockin · 2016-11-03T04:18:40Z

If you can fork this so it is almost entirely self-contained, I would be
less concerned. The userspace Linux code is too delicate...

On Thu, Nov 3, 2016 at 5:14 AM, Brendan Burns notifications@github.com
wrote:

@thockin https://github.com/thockin I'm happy to do the review, is your
review required?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#36079 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVMaxmwPqnx5c52YFpyGimKsiYOU0ks5q6V_AgaJpZM4KnnN3
.

brendandburns · 2016-11-03T04:20:46Z

@thockin I think it's basically structured that way already. The linux-specific stuff is copy/pasted out into a separate linux-specific file, but it's basically unchanged. There's an interface extracted and it is then implemented by a Linux version and a Windows version.

(besides, isn't userspace proxy in-active for basically all Linux clusters?)

feiskyer · 2016-11-03T04:23:51Z

If you can fork this so it is almost entirely self-contained, I would be
less concerned. The userspace Linux code is too delicate...

Is it possible to add a proxy mode dedicated for windows ?

brendandburns · 2016-11-06T15:04:36Z

LGTM, thanks for the patience.

@pires please make sure to come through with the cleanups asap.

pires · 2016-11-06T15:25:13Z

@brendanburns should we do it before 1.5 gets cut? Am under the impression refactors should happen after 1.5. But please, advise.

…kube-proxy is running.

pires · 2016-11-07T09:16:17Z

@feiskyer the PR needed rebase so I went ahead and fixed copyright headers.

feiskyer · 2016-11-07T12:09:16Z

ping @brendandburns for add lgtm back

brendandburns · 2016-11-07T19:46:08Z

LGTMz

brendandburns · 2016-11-07T19:46:57Z

fwiw, github is confused, I added LGTM, not jbhurat...

pires · 2016-11-07T19:48:16Z

I've said this before, Github just can't handle projects like Kubernetes.

saad-ali · 2016-11-08T05:57:48Z

Release-czar approved post-code freeze merge--This was LGTMed and in the merge-queue at code freeze time for 1.5. Adding 1.5 milestone to let it gets merged after code freeze.

k8s-github-robot · 2016-11-09T08:48:40Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2016-11-09T09:26:26Z

Automatic merge from submit-queue

Automatic merge from submit-queue Powershell script to start kubelet and kube-proxy **What this PR does / why we need it**: This PR adds a powershell script to run kubelet and kube-proxy on Windows. It expects the required arguments like `API Server` location and uses appropriate defaults. **Which issue this PR fixes** : fixes # #34270 **Special notes for your reviewer**: This PR is for supporting Windows Server Containers for k8s, the work for which is covered under kubernetes/enhancements#116 This PR should be merged after #31707 and #36079 PRs are merged **Release note**: ```release-note ```

googlebot added the cla: no label Nov 2, 2016

pires force-pushed the windows_kube_proxy branch from d90e1ce to 93945b5 Compare November 2, 2016 18:54

k8s-github-robot assigned smarterclayton Nov 2, 2016

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. release-note-label-needed labels Nov 2, 2016

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 2, 2016

smarterclayton assigned thockin Nov 3, 2016

smarterclayton reviewed Nov 3, 2016

View reviewed changes

brendandburns added cla: human-approved release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed cla: no labels Nov 3, 2016

brendandburns reviewed Nov 3, 2016

View reviewed changes

brendandburns self-assigned this Nov 3, 2016

thockin reviewed Nov 3, 2016

View reviewed changes

brendandburns added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed cla: no labels Nov 6, 2016

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 7, 2016

Jitendra Bhurat and others added 3 commits November 7, 2016 09:11

Added netsh util package.

4cf9b3c

Added new userspace proxy mode specifically for Windows.

acf3f36

Added automatic seletion of userspace proxy mode depending on the OS …

23c35f2

…kube-proxy is running.

pires force-pushed the windows_kube_proxy branch from 966495a to 5d4c784 Compare November 7, 2016 09:14

googlebot added the cla: no label Nov 7, 2016

k8s-github-robot removed lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 7, 2016

Fixed copyright headers.

562d075

pires force-pushed the windows_kube_proxy branch from 5d4c784 to 562d075 Compare November 7, 2016 09:18

jbhurat added lgtm "Looks good to me", indicates that a PR is ready to be merged. and removed cla: no labels Nov 7, 2016

saad-ali added this to the v1.5 milestone Nov 8, 2016

k8s-github-robot merged commit c52efa5 into kubernetes:master Nov 9, 2016

chentao1596 mentioned this pull request Dec 5, 2016

WIP:kubelet: support multi-headers when getting pod from HTTP source #38089

Closed

pires mentioned this pull request Jan 25, 2017

WIP: kube-proxy: remove OS conditionals. #40414

Closed

pires deleted the windows_kube_proxy branch February 27, 2017 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Windows support to kube-proxy #36079

Add Windows support to kube-proxy #36079

jbhurat commented Nov 2, 2016 •

edited by k8s-oncall

googlebot commented Nov 2, 2016

pires commented Nov 2, 2016

k8s-ci-robot commented Nov 2, 2016

k8s-ci-robot commented Nov 2, 2016

smarterclayton commented Nov 3, 2016

smarterclayton Nov 3, 2016

smarterclayton Nov 3, 2016

brendandburns Nov 3, 2016

jbhurat commented Nov 3, 2016

brendandburns Nov 3, 2016

brendandburns Nov 3, 2016

pires Nov 3, 2016

brendandburns Nov 3, 2016

pires Nov 3, 2016

brendandburns commented Nov 3, 2016

thockin commented Nov 3, 2016

brendandburns commented Nov 3, 2016

brendandburns commented Nov 3, 2016

thockin left a comment

thockin Nov 3, 2016

pires Nov 3, 2016

thockin commented Nov 3, 2016

brendandburns commented Nov 3, 2016

feiskyer commented Nov 3, 2016

brendandburns commented Nov 6, 2016

pires commented Nov 6, 2016

pires commented Nov 7, 2016

feiskyer commented Nov 7, 2016

brendandburns commented Nov 7, 2016

brendandburns commented Nov 7, 2016 •

edited

pires commented Nov 7, 2016 •

edited

saad-ali commented Nov 8, 2016

k8s-github-robot commented Nov 9, 2016

k8s-github-robot commented Nov 9, 2016

Add Windows support to kube-proxy #36079

Add Windows support to kube-proxy #36079

Conversation

jbhurat commented Nov 2, 2016 • edited by k8s-oncall

googlebot commented Nov 2, 2016

pires commented Nov 2, 2016

k8s-ci-robot commented Nov 2, 2016

k8s-ci-robot commented Nov 2, 2016

smarterclayton commented Nov 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbhurat commented Nov 3, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendandburns commented Nov 3, 2016

thockin commented Nov 3, 2016

brendandburns commented Nov 3, 2016

brendandburns commented Nov 3, 2016

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Nov 3, 2016

brendandburns commented Nov 3, 2016

feiskyer commented Nov 3, 2016

brendandburns commented Nov 6, 2016

pires commented Nov 6, 2016

pires commented Nov 7, 2016

feiskyer commented Nov 7, 2016

brendandburns commented Nov 7, 2016

brendandburns commented Nov 7, 2016 • edited

pires commented Nov 7, 2016 • edited

saad-ali commented Nov 8, 2016

k8s-github-robot commented Nov 9, 2016

k8s-github-robot commented Nov 9, 2016

jbhurat commented Nov 2, 2016 •

edited by k8s-oncall

brendandburns commented Nov 7, 2016 •

edited

pires commented Nov 7, 2016 •

edited