Where does kubeadm take the proxy settings from?

[erikbgithub]

It's not /etc/environment, it's not the current bash session that kubeadm is running in, it's not docker or kubelet environment. I verified this by setting no_proxy to a different value in all these instances. And for some reason after a kubeadm init it still continues to set another value for no_proxy. Restart, daemon-reload, restarting the services all doesn't change that fact.

Honestly it's really annoying that it only prints the line "the ip address fo.oo.ba.rr has a proxy set to blubb" instead of saying where it takes the value from. And why doesn't it simply read the value from /etc/environment, which is the one true source of truth when it comes to proxy setting, or the current bash session in which I call kubeadm which is the easiest place to make changes to?

What I expect would be something like this:

1. kubeadm checks the current env variable http_proxy. (or https_proxy if secure communication is configured)
2. kubeadm checks the current env variable HTTP_PROXY and warns if it is different.
3. kubeadm checks http_proxy in /etc/environment. It warns if it is different.
4. similar for upper case.
5. if there is neither variable in neither context it assumes there is no proxy and informs about it.
6. kubeadm writes the manifest files (I assume this is done before creating the docker containers) with the given proxy, giving preference to the process environment setting in lower case, then upper case, then lower case /etc/environment, then upper case /etc/environment.
7. kubeadm starts the pods.
8. kubeadm checks whether the controller-manager can talk to the api server. If it gets a "forbidden" or "timeout" it assumes the proxy settings are wrong and erros, calling a kubeadm reset internally.
9. there is no event where it just waits forever without any output. It can at least reasonably well figure out if there are continuous errors in the logs of api-server and controller-manager, as well as whether there are any new logs for >= 10 minutes. And then it can error out with a corresponding error message.
10. kubeadm internally prepends the advertise address to all no_proxy settings (add the end it may get cut of). <-- Also it would be so much better to use a hostname if possible, since no_proxy is actually meant for names, not IPs.

I seriously can't express how many working hours it would save people in enterprise networks.

[luxas]

@erikbgithub Thanks a lot for this issue!
Up front I must say that I'm no proxy expert as I haven't experimented in such environments much.

So I can't comment on the exact statements above really, but I'd be very glad if you wanted to contribute to kubeadm to make the behavior behind a proxy better.

To answer your question, here is the relevant go code:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go#L432

func getProxyEnvVars() []v1.EnvVar {
	envs := []v1.EnvVar{}
	for _, env := range os.Environ() {
		pos := strings.Index(env, "=")
		if pos == -1 {
			// malformed environment variable, skip it.
			continue
		}
		name := env[:pos]
		value := env[pos+1:]
		if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" {
			envVar := v1.EnvVar{Name: name, Value: value}
			envs = append(envs, envVar)
		}
	}
	return envs
}

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/preflight/checks.go#L291

// HTTPProxyCheck checks if https connection to specific host is going
// to be done directly or over proxy. If proxy detected, it will return warning.
type HTTPProxyCheck struct {
	Proto string
	Host  string
	Port  int
}

func (hst HTTPProxyCheck) Check() (warnings, errors []error) {

	url := fmt.Sprintf("%s://%s:%d", hst.Proto, hst.Host, hst.Port)

	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return nil, []error{err}
	}

	proxy, err := http.DefaultTransport.(*http.Transport).Proxy(req)
	if err != nil {
		return nil, []error{err}
	}
	if proxy != nil {
		return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
	}
	return nil, nil
}

I seriously can't express how many working hours it would save people in enterprise networks.

Couldn't agree more

[luxas]

cc @kad @timothysc

[erikbgithub]

@luxas Thanks I'll work through that when I get a round tuit. Before I can supply patches I need to learn some go though, so I would appreciate if others can churn in for now. ;-)

First sub-question I'll look into is what go actually gets via os.Environ().

[luxas]

@erikbgithub Let me know if you need some help with creating patches and I'll help

[kad]

@erikbgithub as original author of that check, I'll be happy to answer any questions.
First few answers:

• kubeadm gets and checks environment from your currently running session. You can see what do you have if you execute $ env | grep -i _proxy= | sort. E.g. inside our company firewall I have something like this:

$ env | grep -i _proxy= | sort
ALL_PROXY=http://proxy-ir.example.com:911
FTP_PROXY=http://proxy-ir.example.com:911
HTTPS_PROXY=http://proxy-ir.example.com:911
HTTP_PROXY=http://proxy-ir.example.com:911
NO_PROXY=.example.com
all_proxy=http://proxy-ir.example.com:911
ftp_proxy=http://proxy-ir.example.com:911
http_proxy=http://proxy-ir.example.com:911
https_proxy=http://proxy-ir.example.com:911
no_proxy=.example.com
$

• Usual issue that people are stepping on without realizing it, that no_proxy variable does NOT support network ranges. so putting something like NO_PROXY=10.0.0.0/8, 192.168.0.0/16 will not have any effect and will still produce warning in pre-flight check.
• Content difference between upper/lowercase *_proxy variables doesn't matter. Go code that handles proxy environment variables have its internal logic in which order it processes it (as I recall, first upper case, then lower case, but that shouldn't matter, you can't guarantee in which order each app processing them)
• files like /etc/environment are distro-specific and not read by each individual binary. they are read and injected into process environment variables by login scripts, PAM modules, etc. Environment variables can be also coming from e.g. SSH sessions or from external management tools (like ansible) while calling other binaries, like kubeadm. So, be dependant on particular /etc/environment or similar is neither feasible nor logical.
• For 8 point, preflight check is specifically about that. It gives warning if kubeadm detects that it is going over proxy to API server. While it was discussion about this preflight check initially, it was agreed that those cases might be legitimate, thus it only warns, not producing error.
• for 10: it was also objections about manipulating those variables from multiple people. (my initial idea was at all to drop all *_proxy settings to force direct connections, but it was turned down, as it might be legitimate reasons to connect over proxies).

[ErikOShaughnessy]

I "fixed" this problem by including all my cluster node IPs in NO_PROXY and using the same NO_PROXY on all the minions when joining the cluster.

$ export NO_PROXY='ip,ip,ip,ip,.example.com'
[master]$ kubeadm init
[minion]$ kubeadm join --token={token} a.b.c.d:6443

To be honest, I'm not sure if it's all the IP addresses being enumerated or the .example.com that fixed the problem.

[kad]

if PR kubernetes/kubernetes#52788 will be merged, it will be possible to specify in NO_PROXY IP ranges for your nodes. it will simplify things a lot.

[ted-jung]

A little bit weired. if i look into the code "checks.go".
it always return error message if there are value in proxy.

if proxy != nil {
return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil
}
return nil, nil

In enterprise...there are necessarily three proxy options. (http_proxy, https_proxy, no_proxy)
http_* is mandatory option to pull images for connection to internet.
if there is no_proxy option be set...then it should return error message.

" pl, set option(no_proxy) not to be routed to the proxy for internal connection"

[rmxhaha]

I want to ask if kubeadm join supports http_proxy ?

I manage to get kubeadm init to work with http_proxy and no_proxy but it seems kubeadm join produce errors such as

kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Get https://gcr.io/v1/_ping: read tcp <my-ip>:58742->74.125.68.82:443: read: connection reset by peer

and also
/etc/environment is empty instead of filled with configuration like in the master.

which let me to believe maybe http_proxy and no_proxy is not yet supported for kubeadm join.

[erikbsap]

Running into this problem once more. It still uses the proxy incorrectly and I seem to be unable to modify proxy and no_proxy settings.

[erikbsap]

see #687, #182, or stack overflow: https://stackoverflow.com/questions/45580788/how-to-install-kubernetes-cluster-behind-proxy-with-kubeadm

[PhilippeDo]

From my experience, kubeadm use the proxy defined in /etc/environment

[earthling42]

From my experience, kubeadm use the proxy defined in /etc/environment

Yup - in my case it is also /etc/environment

[durairajasivam]

in my case, it is /etc/environment as well.

I have to remove the entries from the/etc/environment. It will be good if the kube init/Join has a flag to override this

[arbourp]

I must be missing something obvious or not obviously missing something.
I have managed to find all of the possible locations that proxys are configured for docker and other tools and docker is 100% able to access the registry on the internet.
I can wget, curl etc... they work fine.

kubeadm, can't seem to figure out the proxy settings.

kubeadm config images pull -v 4
times out every time
I0519 14:50:12.755255 54368 version.go:186] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt

I can wget or curl on that file with no problems, but the kubeadm tool doesn't seem to use the proxies. what am I missing?

I have them set in my .bashrc in the /etc/profile.d in /etc/environment. They all have correct settings for the proxy, but kubeadm isn't using them.

[neolit123]

Kubeadm shells to crictl, but we just fixed a bug where it did not pass the proxy env vars to the command.
#2697

Workaround for older versions is to list and pull images manually. See comments on the issue.

[neolit123]

kubeadm config images pull -v 4
times out every time
I0519 14:50:12.755255 54368 version.go:186] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt

That seems like a different problem. To fetch the version kubeadm constructs a go http client. It does respect the proxy env vars on the host. So make sure those are set.

[alicommit-malp]

see my answer here