New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Forbidden error when retrieving logs from non-master node's pods #211
Comments
I suspect the minion is not being given serving certs the master apiserver trusts and is simply generating its own |
same issue here $ kubeadm version
$ kubectl version
list pods runnings:
Logs from a pod running on the master:
try to get logs for a pod running on a node
|
I found the reason. |
@gousse Could you document that on the kubeadm reference page, please? |
I'm hitting this at the moment - a work around would be great! |
I spent a while trying to use |
@gousse So setting |
@jamiehannaford @tomdee |
Which components are involved doing the kubectl logs command? So does the master nodes need to have the worker nodes in there no_proxy only? Does master node means the api_server or any other controller? |
kubectl > apiserver > node hosting the pod |
thx |
Is there a way to make sure kubectl logs goes for DNS instead of IPs? Autoscaling and IPs don't work well. |
Nodes report their network addresses in their Node API object status. The apiserver contacts nodes using the preferred address type as determined by the --kubelet-preferred-address-types flag:
|
Not all kubelet cloud providers report dns addresses currently. |
\o/ Awesome. Saved my day thx! |
@jamiehannaford you're working in the troubleshooting doc. Could you add this to the list? |
@yanhongwang I'm hitting the same proxy issue. My cluster runs well so far but i can't retrieve logs. The no_proxy ip's are set. Do I really need to recreate my cluster? Or is there any other way to get this running? |
Hi @Snipes999 My environment: http_proxy, https_proxy was set by some default value in my network environment. So I add master ip and minion ip to "no_proxy" environment variable to all kubernetes cluster machine. Because I don't know what exactly "kubeadm init" done with system. I use Ansible to deploy machine automatically. So it is not difficult in my case. Otherwise, probably you can do "kubeadm reset". And then try again. Hope this can help. Hong |
I'm using Ubuntu 17.04 and Kubernetes 1.8.1 |
@Snipes999 I'll close this issue as solved then. Thank you! |
@luxas I don't think this is solved. Unless I'm not understanding this correctly...
Shouldn't this be actually fixed so that |
@tomdee I'm constantly hitting issues where something doesn't work if person is in isolated network behind proxies, and trying to fix as much as I can. We have several patches that are already merged into 1.9 and some even backported to 1.8.x to get it better. Some PRs are still under review, but hopefully will soon be merged in 1.9. If you hit something, please feel free to open issue and assign to me or CC me. |
@tomdee open support issue with details about your environment (Vagrant file, network connectivity, distro, vagrant plugins installed, etc). we will see what might be an issue. |
What keywords did you search in kubeadm issues before filing this one?
kubectl logs
logs forbidden curl insecure
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version (use
kubeadm version
):kubeadm version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2074+a092d8e0f95f52", GitCommit:"a092d8e0f95f5200f7ae2cba45c75ab42da36537", GitTreeState:"clean", BuildDate:"2016-12-13T17:03:18Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Environment:
kubectl version
):Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.5", GitCommit:"894ff23729bbc0055907dd3a496afb725396adda", GitTreeState:"clean", BuildDate:"2017-03-23T16:14:24Z", GoVersion:"go1.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-07T23:34:32Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
uname -a
): Linux 4.4.0-47-generic kubeadm init hangs on creating kube-discovery and fails to create kube-dns #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/LinuxKubernetes cluster consists of a single master node and minion node, joined together by
kubeadm
.What happened?
From a remote machine (that is not the master or minion), when doing a
kubectl logs
on any pods that lives on the minion node, the following error occurs:When doing a
kubectl logs
on any of the pods that lives on themaster
node, no error occurs and logs can be retrieved as expected.When doing a curl of the URL returned in the error above with a
--insecure
, I am able to pull the logs from the affected node.What you expected to happen?
Should be able to retrieve logs of a pod from a non-master node.
Anything else we need to know?
The text was updated successfully, but these errors were encountered: