Support service-node-port-range when initializing kubeadm

[anandanand84]

Currently kubeadm doesn't support service-node-port-range flag in init.

kubeadm init --api-service-node-port-range=30000-32767
Error: unknown flag: --api-service-node-port-range

[luxas]

You can do this with the config file, see: https://kubernetes.io/docs/admin/kubeadm/

[chinglinwen]

Documents about config file part is unclear ( provide an example to config service-node-port-range? )

[neolit123]

@chinglinwen, @anandanand84

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

apiServerExtraArgs:
  service-node-port-range: 30000-32767

it's a API server argument and kubadm is just delegating it:
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

[chinglinwen]

Thank you for the reply, can I set the range to include 80, so I can directly map dns name to that node. I deployed Kong ingress controller, It's service node port is not 80, service IP is not reachable, so it need outside nginx for ingress is not good (no just extra layer adds latency, but duplicated nginx config for every service too ) I'm not sure if it's the Kong's problem, just starting to figure out Kong ingress. : )

…

On Thu, May 17, 2018, 9:00 PM Lubomir I. Ivanov ***@***.***> wrote: @chinglinwen <https://github.com/chinglinwen>, @anandanand84 <https://github.com/anandanand84> https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file apiServerExtraArgs: service-node-port-range: 30000-32767 it's a API server argument and kubadm is just delegating it: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APlkQdRiRwxoqTCt2F_OBaCMuK27XUWJks5tzXRsgaJpZM4Ll323> .

[3dbrows]

@chinglinwen I am also trying to deploy the Kong ingress controller. I am using kubeadm to create the cluster, not using GKE or AWS (so I cannot simply use LoadBalancer to expose it). In case it helps, here is what I did. First I started the cluster using a port range of 80-32767 (as root):

$ cat <<EOF > /tmp/kubeadm-init-args.conf
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerExtraArgs:
  service-node-port-range: 80-32767
networking:
  podSubnet: 192.168.0.0/16
EOF

$ kubeadm init --config /tmp/kubeadm-init-args.conf

Then, I used nodePort: 80 on the kong-proxy service:

apiVersion: v1
kind: Service
metadata:
  name: kong-proxy
  namespace: kong
spec:
  type: NodePort
  ports:
  - name: kong-proxy
    port: 80
    targetPort: 8000
    nodePort: 80
    protocol: TCP
  selector:
    app: kong

Now I can access the kong ingress on port 80. This feels a bit like a "dirty hack", but I have struggled to think of a better way. I am also still learning this. I also considered using iptables to force port 80 traffic to a NodePort in the usual range of 30000-32767. It seems to me like all documentation assumes you have something outside K8S that can map 80 --> 3xxxx.

[chinglinwen]

@Dag24 Thank you. I did it in the same way as you do( in early days), it's working as expected, very thanks to @neolit123 's helpful advice. ( during that time encounter a issue of CNI not clean up enough while re-create the cluster ).

It's now expose 80 and 443 on every nodes. so DNS can simply resolve to node's ip. Everything is working as expected, I'm not sure if Kube-proxy will be the bottleneck or not, (Also I've notice the sessionAffinity to clientip may not work for Kong ingress right now. )

[nthienan]

I initialized k8s cluster with default service-node-port-range. How can I modify service-node-port-range without re-initialize the cluster?

[AdrianBalcan]

@nthienan you can run: kubectl edit cm kubeadm-config -n kube-system and add service-node-port-range: 80-32767 under apiServerExtraArgs:.
kube-apiserver will restart in few moments: watch "kubectl get pods -n kube-system | grep apiserver"

[yanhongwang]

Hello,

Is it possible to indicate port range like below?
--service-node-port-range=80,30000-32767

So that could avoid to expose other system port except 80?

[rrichardson]

@AdrianBalcan - I have edited the kubeadm-config as you recommended. However the apiserver hasn't restarted. Also, the change is not reflected in /etc/kubernetes/manifests/kube-apiserver.yaml.

Is there a way to force this reconfiguration and restart? What is the mechanism that monitors the kubeadm-config and makes the changes? Maybe that isn't running.

edit - also note that kubeadm config view reflects the appropriate changes.

[yanhongwang]

Same with @rrichardson

[philipsahli]

Had the same behaviour like @rrichardson. I added the line manually in /etc/kubernetes/manifests/kube-apiserver.yaml on the master and deleted the apiserver pod. Now I can use NodePort on port 80 and above.

[philipsahli]

I had some troubes again after recreating the cluster. This time following worked:

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp
vi /tmp/kube-apiserver.yaml
kubectl delete pod -l component=kube-apiserver --namespace kube-system
sleep 10
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests
kubectl get pod -l component=kube-apiserver --namespace kube-system
ps -ef | grep "service-node-port-range"

[molon]

I had some troubes again after recreating the cluster. This time following worked:

mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp
vi /tmp/kube-apiserver.yaml
kubectl delete pod -l component=kube-apiserver --namespace kube-system
sleep 10
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests
kubectl get pod -l component=kube-apiserver --namespace kube-system
ps -ef | grep "service-node-port-range"

vi /etc/kubernetes/manifests/kube-apiserver.yaml
add `--service-node-port-range=80-32767` then save
systemctl restart kubelet

[TiagoTT]

The full steps I needed to take to update the service-node-port-range:
1 - Edit the kubeadm configuration map and add service-node-port-range: 80-32767 under data, ClusterConfiguration, apiServer, extraArgs:

# kubectl edit configmap kubeadm-config -n kube-system
...
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
        service-node-port-range: 80-32767
      timeoutForControlPlane: 4m0s
...

2 - Dump the cluster configuration to a file and force kubeadm to regenerate the manifest for the apiserver, on each and every controller node:

# kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadmcurrent.yaml
# kubeadm init phase control-plane apiserver --config kubeadmcurrent.yaml

3 - Check that the expected value was placed in the manifest and wait for the kube apiserver to get restarted, on each and every controller node:

# grep service-node-port-range /etc/kubernetes/manifests/kube-apiserver.yaml 
    - --service-node-port-range=80-32767
# pgrep -a apiserver | grep -o service-node-port-range=80-32767

[yborges]

@chinglinwen I am also trying to deploy the Kong ingress controller. I am using kubeadm to create the cluster, not using GKE or AWS (so I cannot simply use LoadBalancer to expose it). In case it helps, here is what I did. First I started the cluster using a port range of 80-32767 (as root):

$ cat <<EOF > /tmp/kubeadm-init-args.conf
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
apiServerExtraArgs:
  service-node-port-range: 80-32767
networking:
  podSubnet: 192.168.0.0/16
EOF

$ kubeadm init --config /tmp/kubeadm-init-args.conf

Then, I used nodePort: 80 on the kong-proxy service:

apiVersion: v1
kind: Service
metadata:
  name: kong-proxy
  namespace: kong
spec:
  type: NodePort
  ports:
  - name: kong-proxy
    port: 80
    targetPort: 8000
    nodePort: 80
    protocol: TCP
  selector:
    app: kong

Now I can access the kong ingress on port 80. This feels a bit like a "dirty hack", but I have struggled to think of a better way. I am also still learning this. I also considered using iptables to force port 80 traffic to a NodePort in the usual range of 30000-32767. It seems to me like all documentation assumes you have something outside K8S that can map 80 --> 3xxxx.

Your comment about the documentation is important. For example, Ingress Nginx has a whole tutorial on how to use in on bare metal with service type NodePort. It's a one-line code to install it. However, it cannot work without tweaking the standard settings for kubeadm init and they don't mention that anywhere. I've been struggling with ports and ip's for 3 days now because of this.

[atline]

Hello,

Is it possible to indicate port range like below?
--service-node-port-range=80,30000-32767

So that could avoid to expose other system port except 80?

So, is it possible?

[david-2000]

@atline

Hello,
Is it possible to indicate port range like below?
--service-node-port-range=80,30000-32767
So that could avoid to expose other system port except 80?

So, is it possible?

According to this:

#122 (comment)

kubeadm is just delegating the flag to the API server argument. From https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ (search for --service-node-port-range) it looks like it is expecting a specific format for the range as "MIN-MAX". So I doubt port 80 can be included along with a separate range of numbers.