Kubernetes ClusterをVMで作ってみる #1

aokabin · 2018-09-20T03:36:05Z

How to install a Kubernetes cluster on CentOS 7 - TechRepublic

上記のサイトを参考にKubetenetes ClusterをCentOS7上で作ってみる

aokabin · 2018-09-27T03:12:45Z

今回はVagrantでサーバーを3つ立てて、その中でネットワークを組んで試してみようと思う

環境を作るにあたって参考にしたのはこちら

MacにVagrantでCentOS7環境を作成 - Qiita

$ vagrant box add centos/7
==> box: Loading metadata for box 'centos/7'
    box: URL: https://vagrantcloud.com/centos/7
This box can work with multiple providers! The providers that it
can work with are listed below. Please review the list and choose
the provider you will be working with.

1) hyperv
2) libvirt
3) virtualbox
4) vmware_desktop

Enter your choice: 3
==> box: Adding box 'centos/7' (v1804.02) for provider: virtualbox
    box: Downloading: https://vagrantcloud.com/centos/boxes/7/versions/1804.02/providers/virtualbox.box
==> box: Box download is resuming from prior download progress
    box: Download redirected to host: cloud.centos.org
==> box: Successfully added box 'centos/7' (v1804.02) for 'virtualbox'!
[Vagrant] vagrant box list
bento/centos-6.7 (virtualbox, 2.2.7)
centos/7         (virtualbox, 1804.02)
ubuntu/xenial64  (virtualbox, 20180713.0.0)

virtualboxを選択

aokabin · 2018-09-27T03:22:06Z

3つのVMを立てるので、以下のようにディレクトリを構成

$ mkdir kubeclu_{1..3}
$ tree -L 2
kubernetes-clusters
├── kubeclu_1
├── kubeclu_2
└── kubeclu_3

iTerm2なので、それぞれのディレクトリに入って⌥⌘Iで以下を実行

$ vagrant init centos/7
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

Vagrantfileが作られた

aokabin · 2018-09-27T03:23:53Z

statusを確認

$ vagrant status
Current machine states:

default                   not created (virtualbox)

The environment has not yet been created. Run `vagrant up` to
create the environment. If a machine is not created, only the
default provider will be shown. So if a provider is not listed,
then the machine is not created for that environment.

aokabin · 2018-09-27T03:29:35Z

vagrantを立ち上げる

$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos/7'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/7' is up to date...
==> default: Setting the name of the VM: kubeclu_1_default_1538018591475_19767
==> default: Fixed port collision for 22 => 2222. Now on port 2201.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2201 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2201
    default: SSH username: vagrant
    default: SSH auth method: private key
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: No guest additions were detected on the base box for this VM! Guest
    default: additions are required for forwarded ports, shared folders, host only
    default: networking, and more. If SSH fails on this machine, please install
    default: the guest additions and repackage the box to continue.
    default:
    default: This is not an error message; everything may continue to work properly,
    default: in which case you may ignore this message.
==> default: Rsyncing folder: /Users/xx/Develop/Vagrant/kubernetes-clusters/kubeclu_1/ => /vagrant

statusを確認

$ vagrant status
Current machine states:

default                   running (virtualbox)

The VM is running. To stop this VM, you can run `vagrant halt` to
shut it down forcefully, or you can run `vagrant suspend` to simply
suspend the virtual machine. In either case, to restart it again,
simply run `vagrant up`.

aokabin · 2018-09-27T03:31:39Z

sshして入ってみる

$ vagrant ssh
[vagrant@401 ~]$ ls
[vagrant@401 ~]$ pwd
/home/vagrant

いけてますね！

aokabin · 2018-09-27T04:11:21Z

再度こちらに戻ります

How to install a Kubernetes cluster on CentOS 7 - TechRepublic

まず、それぞれのVMのipアドレスの設定が必要そうなのでVagrantfileにて設定する

# Vagrantfile
  config.vm.box = "centos/7"
  config.vm.network "private_network", ip: "ここがIPアドレス", virtualbox__intnet: "kubecluster"

それぞれに、プライベートネットワークを設定、ipアドレスは以下のようにした

kubeclu_1: 192.168.50.2
kubeclu_2: 192.168.50.3
kubeclu_3: 192.168.50.4

aokabin · 2018-09-27T04:45:29Z

/etc/hostsに以下を追記

$ vi /etc/hosts
192.168.50.2 kubemaster
192.168.50.3 kube1
192.168.50.4 kube2

SELinuxとswapをdisableにする

$ setenforce 0
$ sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
$ swapoff -a

swapをoffにするのは、どうやらSwapが有効だとkubeletが起動しないような変更がKubernetes1.8.0から有効になったそうな。(中腹の「Kubernetesの設定」にて説明)

参考:

自宅PCクラスタのKubernetesを1.9にバージョンアップしたログ - あさのひとりごと
- Kubelet needs to allow configuration of container memory-swap · Issue #7294 · kubernetes/kubernetes

aokabin · 2018-09-27T06:58:31Z

なぜSELinuxをoffにしないといけないかは、慣例にしてもいいのだけど、詳しいことはこちらに書いてありそうなので、別途勉強したい

関連issue: #2
「SELinuxのせいで動かない」撲滅ガイド - Qiita

aokabin · 2018-09-27T07:02:52Z

上記コマンド実行をしても、reboot時にswapがonになるっぽいので、以下の行をコメントアウトしろとのこと

# /etc/fstab
/dev/mapper/centos-swap swap swap defaults 0 0

aokabin · 2018-09-27T12:11:14Z

次に、br_netfilterをenableに変更

Docker-ceのインストール

最初に関係パッケージ(ライブラリやミドルウェア)をインストール

$ yum install -y yum-utils device-mapper-persistent-data lvm2

次にリポジトリを追加

$ yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

リポジトリってのは、パッケージの依存関係が書いてあるものっぽい(イメージは湧く)
参考: 【パッケージ】Linuxの「パッケージ」と「yum」と「rpm」について勉強したのでまとめてみた。 - Qiita

そしてDocker-ceをインストール

$ yum install -y docker-ce

aokabin · 2018-09-27T12:47:32Z

Docker-ceとは？

UbuntuにDocker CEをインストールする - Qiita
コミュニティエディション(Community Edition)の略らしい

無償のやつってことか(EEってのもある)

aokabin · 2018-09-27T12:49:51Z

Kubernetesのインストール

$ vi /etc/yum.repos.d/kubernetes.repo

# 以下を追記
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

そしてインストール

$ yum install -y kubelet kubeadm kubectl

aokabin · 2018-10-10T09:34:21Z

久々に再開

Cgroupの変更

Docker-ceとKubernetesを同じcgroupに所属させる必要があるらしい

チェックのために、docker infoして見たところ、dockerが起動していなかったので起動

$ service docker start
$ docker info | grep -i cgroup
Cgroup Driver: cgroupfs

dockerのcgroupはcgroupfsだったので、Kubernetesも同じcgroupに変更
以下のコマンドでできるらしいけど

$ sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

念のため、実際のkubeadm.confを確認しておく

$ less /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

この中に既にcgroup-driver=systemdが...ない...！？
sedで置換してるものだと思ってたのだけど...

まぁいいや、実行してみよう
...中身は変わってないっぽい...
ちょっとdockerdのcgroupを確認してみる

$ ps -aux | grep docker
$ cat /proc/docerのPID/cgroup
11:cpuset:/
10:memory:/system.slice/docker.service
9:pids:/system.slice/docker.service
8:devices:/system.slice/docker.service
7:freezer:/
6:hugetlb:/
5:cpuacct,cpu:/system.slice/docker.service
4:blkio:/system.slice/docker.service
3:perf_event:/
2:net_prio,net_cls:/
1:name=systemd:/system.slice/docker.service

何かわかるかなと思ったけど読めねぇ

aokabin · 2018-10-10T10:16:25Z

もしかしたらこれでいいのかもしれないし、一旦進めてみよう

$ systemctl daemon-reload
$ systemctl restart kubelet

そしてKubernetesクラスタを初期化する

以下をKubemasterにて実行

$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24

参考資料と違うのは、adminのipとpodのipのcidrを変更しているところ

aokabin · 2018-10-10T10:31:25Z

$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
I1010 10:16:47.602736    1831 kernel_validator.go:81] Validating kernel version
I1010 10:16:47.602791    1831 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.1-ce. Max validated version: 17.03
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [401.pc.ie.u-ryukyu.ac.jp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.50.2]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [401.pc.ie.u-ryukyu.ac.jp localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [401.pc.ie.u-ryukyu.ac.jp localhost] and IPs [192.168.50.2 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

		Unfortunately, an error has occurred:
			timed out waiting for the condition

		This error is likely caused by:
			- The kubelet is not running
			- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
			- No internet connection is available so the kubelet cannot pull or find the following control plane images:
				- k8s.gcr.io/kube-apiserver-amd64:v1.11.3
				- k8s.gcr.io/kube-controller-manager-amd64:v1.11.3
				- k8s.gcr.io/kube-scheduler-amd64:v1.11.3
				- k8s.gcr.io/etcd-amd64:3.2.18
				- You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
				  are downloaded locally and cached.

		If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
			- 'systemctl status kubelet'
			- 'journalctl -xeu kubelet'

		Additionally, a control plane component may have crashed or exited when started by the container runtime.
		To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
		Here is one example how you may list all Kubernetes containers running in docker:
			- 'docker ps -a | grep kube | grep -v pause'
			Once you have found the failing container, you can inspect its logs with:
			- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

失敗したやで、どうやらkubeletが動いてなかったっぽいので、起動させる
3台同時

$ service kubelet start

そしてもっかい試したところエラー

$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
I1010 10:27:53.042929    3499 kernel_validator.go:81] Validating kernel version
I1010 10:27:53.043007    3499 kernel_validator.go:96] Validating kernel config
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.1-ce. Max validated version: 17.03
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Some fatal errors occurred:
	[ERROR Port-10251]: Port 10251 is in use
	[ERROR Port-10252]: Port 10252 is in use
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
	[ERROR Port-10250]: Port 10250 is in use
	[ERROR Port-2379]: Port 2379 is in use
	[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

systemctl enable kubelet.serviceを試してもほぼ同じエラー

どうやらDockerのバージョンが新しすぎるっぽいので、グレードダウンする必要があるみたい
When I use kubeadm join. Unfortunately, an error has occurred: timed out waiting for the condition · Issue #67808 · kubernetes/kubernetes

aokabin · 2018-10-10T12:39:33Z

というわけでダウングレードする

$ yum list docker-ce --showduplicates | sort -r
Failed to set locale, defaulting to C
docker-ce.x86_64            18.06.1.ce-3.el7                   docker-ce-stable
docker-ce.x86_64            18.06.1.ce-3.el7                   @docker-ce-stable
docker-ce.x86_64            18.06.0.ce-3.el7                   docker-ce-stable
docker-ce.x86_64            18.03.1.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            18.03.0.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.12.1.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.12.0.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.09.1.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.09.0.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.06.2.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.06.1.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.06.0.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.03.3.ce-1.el7                   docker-ce-stable
docker-ce.x86_64            17.03.2.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.03.1.ce-1.el7.centos            docker-ce-stable
docker-ce.x86_64            17.03.0.ce-1.el7.centos            docker-ce-stable
Loading mirror speeds from cached hostfile
Loaded plugins: fastestmirror
Installed Packages
Available Packages
 * updates: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * base: ftp.iij.ad.jp

17.03系のstable版の17.03.3.ce-1.el7にスイッチしよう

aokabin · 2018-10-10T12:49:25Z

なんかこんな感じで怒られた

インストールしてダウングレードらしいので、まずバージョンを指定してインストールする

$ yum install docker-ce-17.03.2.ce
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
Package matching docker-ce-17.03.2.ce-1.el7.centos.x86_64 already installed. Checking for update.
Nothing to do

なんか入っている扱いっぽい
なのでダウングレードを実行してみるも

$ yum downgrade docker-ce-17.03.2.ce
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 0:17.03.2.ce-1.el7.centos will be a downgrade
--> Processing Dependency: docker-ce-selinux >= 17.03.2.ce-1.el7.centos for package: docker-ce-17.03.2.ce-1.el7.centos.x86_64
Package docker-ce-selinux-17.03.3.ce-1.el7.noarch is obsoleted by docker-ce-18.06.1.ce-3.el7.x86_64 which is already installed
---> Package docker-ce.x86_64 0:18.06.1.ce-3.el7 will be erased
--> Finished Dependency Resolution
Error: Package: docker-ce-17.03.2.ce-1.el7.centos.x86_64 (docker-ce-stable)
           Requires: docker-ce-selinux >= 17.03.2.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.0.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.0.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.1.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.1.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.2.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.3.ce-1.el7.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.3.ce-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

$ yum install --setopt=obsoletes=0    docker-ce-17.03.2.ce-1.el7.centos.x86_64    docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
Package matching docker-ce-17.03.2.ce-1.el7.centos.x86_64 already installed. Checking for update.
Resolving Dependencies
--> Running transaction check
---> Package docker-ce-selinux.noarch 0:17.03.2.ce-1.el7.centos will be installed
--> Processing Conflict: docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch conflicts docker-selinux
Removing docker-ce-selinux.noarch 0:17.03.2.ce-1.el7.centos - u due to obsoletes from installed docker-ce-18.06.1.ce-3.el7.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package docker-ce-selinux.noarch 0:17.03.2.ce-1.el7.centos will be installed
--> Finished Dependency Resolution
Error: docker-ce-selinux conflicts with 2:container-selinux-2.68-1.el7.noarch
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

aokabin · 2018-10-10T12:53:58Z

どうやら docker-ce-selinuxとcontainer-selinuxがconflictしているみたいなので
container-selinuxを削除してみる

$ yum erase container-selinux

そしたら一緒にdockerも消えちゃったので、再インストールの流れに

aokabin · 2018-10-10T12:59:42Z

$ yum install -y docker-ce-17.03.3.ce-1.el7
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.iij.ad.jp
 * extras: ftp.iij.ad.jp
 * updates: ftp.iij.ad.jp
Resolving Dependencies
--> Running transaction check
---> Package docker-ce.x86_64 0:17.03.3.ce-1.el7 will be installed
--> Processing Dependency: docker-ce-selinux >= 17.03.3.ce-1.el7 for package: docker-ce-17.03.3.ce-1.el7.x86_64
Package docker-ce-selinux is obsoleted by docker-ce, but obsoleting package does not provide for requirements
--> Finished Dependency Resolution
Error: Package: docker-ce-17.03.3.ce-1.el7.x86_64 (docker-ce-edge)
           Requires: docker-ce-selinux >= 17.03.3.ce-1.el7
           Available: docker-ce-selinux-17.03.0.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.0.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.1.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.1.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch (docker-ce-stable)
               docker-ce-selinux = 17.03.2.ce-1.el7.centos
           Available: docker-ce-selinux-17.03.3.ce-1.el7.noarch (docker-ce-edge)
               docker-ce-selinux = 17.03.3.ce-1.el7
           Available: docker-ce-selinux-17.04.0.ce-1.el7.centos.noarch (docker-ce-edge)
               docker-ce-selinux = 17.04.0.ce-1.el7.centos
           Available: docker-ce-selinux-17.05.0.ce-1.el7.centos.noarch (docker-ce-edge)
               docker-ce-selinux = 17.05.0.ce-1.el7.centos
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

とまぁ、普通にインストールしようとすると失敗

$ yum install --setopt=obsoletes=0 -y docker-ce-17.03.3.ce-1.el7

--setopt=obsoletes=0をつけて実行するとうまくいった！

ちなみに、yum-config-manager --enable Extraとかも実行はしていたけど、関係あったかは不明

参考

aokabin · 2018-10-11T04:09:30Z

そこで再度クラスタの追加処理を行う

$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24
[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
I1010 16:38:26.702753   20935 kernel_validator.go:81] Validating kernel version
I1010 16:38:26.702844   20935 kernel_validator.go:96] Validating kernel config
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 3.10.0-862.2.3.el7.x86_64
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled (as module)
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled (as module)
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
CONFIG_OVERLAY_FS: enabled (as module)
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled (as module)
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
[preflight] Some fatal errors occurred:
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
	[ERROR Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
	[ERROR SystemVerification]: failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
	[ERROR Port-10250]: Port 10250 is in use
	[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

エラー、なぜかdockerが立ち上がっていないっぽい？
startさせてみる

$ service docker start
Redirecting to /bin/systemctl start docker.service
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

立ち上げに失敗するので、ログを見てみる

$ journalctl -xeu docker

-- Logs begin at Wed 2018-10-10 15:55:45 UTC, end at Wed 2018-10-10 16:47:44 UTC. --
Oct 10 16:38:07 401.pc.ie.u-ryukyu.ac.jp systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has begun starting up.
Oct 10 16:38:08 401.pc.ie.u-ryukyu.ac.jp dockerd[20920]: time="2018-10-10T16:38:08.133209174Z" level=info msg="libcontainerd: new containerd process, pid: 20923"
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp dockerd[20920]: time="2018-10-10T16:38:09.535518145Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp dockerd[20920]: Error starting daemon: error initializing graphdriver: driver not supported
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp systemd[1]: Unit docker.service entered failed state.
Oct 10 16:38:09 401.pc.ie.u-ryukyu.ac.jp systemd[1]: docker.service failed.

docker立ち上げに失敗しているっぽい

aokabin · 2018-10-11T04:22:37Z

どうも、ストレージドライバの問題らしく、overlay2を使うように設定する必要があるみたい。

// vi /etc/docker/daemon.json
{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

そしてスタートさせる

$ service docker start

動いた！

参考

aokabin · 2018-10-16T14:23:34Z

動いたけど、そのまま調子が悪くなった、rebootもすぐは実行できないくらいに
なので、結局kubernetesも一度アンインストールして再インストールした

$ yum erase kubelet kubeadm kubectl
$ yum install -y kubelet kubeadm kubectl
$ sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
$ less /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24
I1011 05:16:08.786153    4038 version.go:89] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://storage.googleapis.com/kubernetes-release/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1011 05:16:10.738325    4038 version.go:94] falling back to the local client version: v1.12.1
[init] using Kubernetes version: v1.12.1
[preflight] running pre-flight checks
[preflight] Some fatal errors occurred:
	[ERROR Port-6443]: Port 6443 is in use
	[ERROR Port-10251]: Port 10251 is in use
	[ERROR Port-10252]: Port 10252 is in use
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
	[ERROR Port-10250]: Port 10250 is in use
	[ERROR Port-2379]: Port 2379 is in use
	[ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...

失敗ですね...

そういえばCgroup設定してない、それかな？

aokabin · 2018-10-16T15:10:27Z

Cgroup関係なさそうだった、topでみたら、めちゃくちゃCPUとメモリ消費してた、足りないんだな
と思って一応積んで見たけど、エラーはかわらねぇ
ポート使ってるって言ってるので、lsofで見る

$ yum -y install lsof
lsof -i:6443
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kubelet   3120 root   20u  IPv4  37048      0t0  TCP kubemaster:36612->kubemaster:sun-sr-https (ESTABLISHED)
kube-sche 3353 root    5u  IPv4  36321      0t0  TCP kubemaster:36618->kubemaster:sun-sr-https (ESTABLISHED)
kube-apis 3398 root    3u  IPv6  37032      0t0  TCP *:sun-sr-https (LISTEN)
kube-apis 3398 root   70u  IPv6  36575      0t0  TCP kubemaster:sun-sr-https->kubemaster:36826 (ESTABLISHED)
kube-apis 3398 root   71u  IPv6  37261      0t0  TCP kubemaster:sun-sr-https->kubemaster:36610 (ESTABLISHED)
kube-apis 3398 root   72u  IPv6  37262      0t0  TCP kubemaster:sun-sr-https->kubemaster:36612 (ESTABLISHED)
kube-apis 3398 root   75u  IPv6  37265      0t0  TCP kubemaster:sun-sr-https->kubemaster:36618 (ESTABLISHED)
kube-apis 3398 root   92u  IPv4  36531      0t0  TCP localhost:48222->localhost:sun-sr-https (ESTABLISHED)
kube-apis 3398 root  112u  IPv6  36551      0t0  TCP localhost:sun-sr-https->localhost:48222 (ESTABLISHED)
kube-cont 3438 root    5u  IPv4  36316      0t0  TCP kubemaster:36610->kubemaster:sun-sr-https (ESTABLISHED)
kube-cont 3438 root    6u  IPv4  36574      0t0  TCP kubemaster:36826->kubemaster:sun-sr-https (ESTABLISHED)

なんかめっちゃ使ってた

この記事に当たったので挑戦
port 10251 and 10252 are in use · Issue #339 · kubernetes/kubeadm

$ kubeadm reset

aokabin · 2018-10-16T15:13:45Z

そして再挑戦

$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24
[init] using Kubernetes version: v1.12.1
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [localhost.localdomain localhost] and IPs [192.168.50.2 127.0.0.1 ::1]
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost.localdomain localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [localhost.localdomain kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.50.2]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 23.508569 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node localhost.localdomain as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node localhost.localdomain as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "localhost.localdomain" as an annotation
[bootstraptoken] using token: ipt4ha.vd85zft8s8hj27cp
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.50.2:6443 --token xxxxxxxxx --discovery-token-ca-cert-hash shaxxxxxxxxxxxxxxxx

おおー、うまくいったっぽい！
メモリもCPUも大事だなぁ、少なくとも2コア2GBくらいは必要かも
(って書いてあったや→Creating a single master cluster with kubeadm - Kubernetes #Before you begin)

aokabin · 2018-10-17T08:30:02Z

kubeadm initコマンドで出てきたtokenを元にnodeを作るのかな？ 🤔 💭
まぁいいや

Kubernetesの設定

Kubernetesを使う前に、configurationをする必要があるらしい。
以下のコマンドで、設定をコピーし扱えるようにする。

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

なお、configの中身はこんな感じだった

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: xxx
    server: https://192.168.50.2:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: xxx
    client-key-data: xxx

aokabin · 2018-10-17T08:33:01Z

flannel networkをデプロイ

flannelネットワークとやらをデプロイする。
調べてみて感じ、異なるDockerホスト間で通信するネットワークの仕組みっぽい。

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

参考

わかった気になるflannelの仕組み - Qiita

aokabin · 2018-10-17T08:38:15Z

そしたら、Kubeadminにjoinさせよう
さっきのjoinをKube2とかで実行する

$ kubeadm join 192.168.50.2:6443 --token xxxxxxxxx --discovery-token-ca-cert-hash shaxxxxxxxxxxxxxxxx

2回失敗した！
1回は、sudo権限が必要というエラーで、2回目は以下のようなエラー

[preflight] running pre-flight checks
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I1017 08:35:53.646232    3156 kernel_validator.go:81] Validating kernel version
I1017 08:35:53.646298    3156 kernel_validator.go:96] Validating kernel config
[preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

aokabin · 2018-10-17T09:05:53Z

nodeでkubeadm等を再インストールしてないことを思い出したので、再挑戦

$ yum erase -y kubelet kubeadm kubectl
$ yum install -y kubelet kubeadm kubectl

ダメやった...
エラー読むと、RequiredIPVSKernelModulesAvailable的なwarningが出てる
(が、それはほぼ関係ないことがわかった)

エラー内容は

[preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

なので、調べてみると、以下のstackoverflowに行き着いた
docker - Can't install Kubernetes on Vagrant - Stack Overflow

というわけで、以下を実行

$ vi /etc/sysctl.conf
net.bridge.bridge-nf-call-iptables = 1 # これを追記
$ sudo sysctl -p

そして実行したら行けた！
エラー文、よくみたら

/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
→ /proc/sys/net/bridge/bridge-nf-call-iptablesを1にしてね

と書いてある、詳しくはここにあるっぽいよ
Net.bridge.bridge-nf-call and sysctl.conf - Libvirt Wiki

aokabin · 2018-10-17T09:11:24Z

joinができてると思ったので、adminの方で

$ kubectl get nodes
NAME                       STATUS     ROLES    AGE   VERSION
401.pc.domaindayo   Ready      <none>   14m   v1.12.1
localhost.localdomain      NotReady   master   17h   v1.12.1

なるほど、うまくいってないのはわかる

どうやらhostnameがNAMEになってるっぽいことがわかった

$ hostname
401.pc.domaindayo

というわけで、hostnameを変更する

$ sudo hostnamectl set-hostname kubemaster

これでセッションを繋ぎ直せばhostnameは変わる

参考

CentOS 7 の Hostname を変更する - Qiita

aokabin · 2018-10-17T09:27:30Z

もう一つ、kubectlのnodeのローカルホストの名前が変わってなかった(adminになるべき)ので
もう一回kubeadmを設定し直す

$ kubeadm reset
$ kubeadm init --apiserver-advertise-address=192.168.50.2 --pod-network-cidr=192.168.50.0/24

resetはうまくいったけど、initは失敗した。。。
でもみたことあるやつ

[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

多分これっしょ

$ vi /etc/sysctl.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
$ sudo sysctl -p

正しいっぽい！動いた！

aokabin · 2018-10-17T09:28:17Z

と、ここでnodeをチェック

$ kubectl get nodes
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

またエラー...

aokabin · 2018-10-17T09:35:34Z

initした後に、configをコピーする必要があるらしい？

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

一応観れた、けどNotReadyだなー？

$ kubectl get nodes
NAME         STATUS     ROLES    AGE     VERSION
kube1        NotReady   <none>   8m38s   v1.12.1
kube2        NotReady   <none>   8m31s   v1.12.1
kubemaster   NotReady   master   10m     v1.12.1

statusとか見てみるけど、分からないなぁ

$ kubectl describe node kubemaster
Name:               kubemaster
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=kubemaster
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 17 Oct 2018 09:24:10 +0000
Taints:             node-role.kubernetes.io/master:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Wed, 17 Oct 2018 10:07:10 +0000   Wed, 17 Oct 2018 09:24:03 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Wed, 17 Oct 2018 10:07:10 +0000   Wed, 17 Oct 2018 09:24:03 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 17 Oct 2018 10:07:10 +0000   Wed, 17 Oct 2018 09:24:03 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 17 Oct 2018 10:07:10 +0000   Wed, 17 Oct 2018 09:24:03 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 17 Oct 2018 10:07:10 +0000   Wed, 17 Oct 2018 09:24:03 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.50.2
  Hostname:    kubemaster
Capacity:
 attachable-volumes-azure-disk:  16
 cpu:                            2
 ephemeral-storage:              39269648Ki
 hugepages-2Mi:                  0
 memory:                         1882696Ki
 pods:                           110
Allocatable:
 attachable-volumes-azure-disk:  16
 cpu:                            2
 ephemeral-storage:              36190907537
 hugepages-2Mi:                  0
 memory:                         1780296Ki
 pods:                           110
System Info:
 Machine ID:                 d332fe5f44af422cb0e2d378d5c9401e
 System UUID:                D332FE5F-44AF-422C-B0E2-D378D5C9401E
 Boot ID:                    2c2974be-bc9f-464f-8a43-39983475f026
 Kernel Version:             3.10.0-862.2.3.el7.x86_64
 OS Image:                   CentOS Linux 7 (Core)
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://17.3.3
 Kubelet Version:            v1.12.1
 Kube-Proxy Version:         v1.12.1
PodCIDR:                     192.168.50.0/24
Non-terminated Pods:         (5 in total)
  Namespace                  Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                  ------------  ----------  ---------------  -------------
  kube-system                etcd-kubemaster                       0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-apiserver-kubemaster             250m (12%)    0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-controller-manager-kubemaster    200m (10%)    0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-proxy-lfvvw                      0 (0%)        0 (0%)      0 (0%)           0 (0%)
  kube-system                kube-scheduler-kubemaster             100m (5%)     0 (0%)      0 (0%)           0 (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests    Limits
  --------                       --------    ------
  cpu                            550m (27%)  0 (0%)
  memory                         0 (0%)      0 (0%)
  attachable-volumes-azure-disk  0           0
Events:
  Type    Reason                   Age                From                    Message
  ----    ------                   ----               ----                    -------
  Normal  Starting                 43m                kubelet, kubemaster     Starting kubelet.
  Normal  NodeHasSufficientDisk    43m (x6 over 43m)  kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientDisk
  Normal  NodeHasSufficientMemory  43m (x6 over 43m)  kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    43m (x6 over 43m)  kubelet, kubemaster     Node kubemaster status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     43m (x5 over 43m)  kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  43m                kubelet, kubemaster     Updated Node Allocatable limit across pods
  Normal  Starting                 42m                kube-proxy, kubemaster  Starting kube-proxy.
  Normal  Starting                 93s                kubelet, kubemaster     Starting kubelet.
  Normal  NodeHasSufficientDisk    93s                kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientDisk
  Normal  NodeHasSufficientMemory  93s                kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    93s                kubelet, kubemaster     Node kubemaster status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     93s                kubelet, kubemaster     Node kubemaster status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  93s                kubelet, kubemaster     Updated Node Allocatable limit across pods

参考

aokabin · 2018-10-17T10:09:27Z

どうやら、flannelを入れるタイミングが早すぎたっぽく、再度入れてみたら動いた！

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
kube1        Ready    <none>   41m   v1.12.1
kube2        Ready    <none>   41m   v1.12.1
kubemaster   Ready    master   43m   v1.12.1

やったぜ！！！

参考

Raspberry Piでkubernetesクラスタを組んでみる③ - Qiita

aokabin · 2018-10-17T11:19:05Z

再起動処理

再起動しても動いてて欲しいので、再起動処理してみた

案の定落ちてたよね、ぐぬぬ。。。

$ kubectl get nodes
The connection to the server 192.168.50.2:6443 was refused - did you specify the right host or port?

調べてみたら、swapoff -aをしろとのことだったので、してみたら動いた！

前にもやってたんだけどoffになったのか...？

$ swapoff -a
$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
kube1        Ready    <none>   87m   v1.12.1
kube2        Ready    <none>   87m   v1.12.1
kubemaster   Ready    master   90m   v1.12.1

一応別の記事も確認して、swapoff -aの後にmount -aを実行したら再起動処理しても大丈夫だった。

参考

aokabin · 2018-11-09T14:24:48Z

こっちですね

実は接続できていないということがわかりました...
というのも、Node同士が通信できていない！
ネットワークが悪そうということはわかるので、こちらを参考に進めていく

Kubernetes v1.10 クラスタをVagrantで構築したメモ - Qiita

ただ、CentOSでやりたい気持ちではある

aokabin · 2018-11-10T04:53:17Z

ネットワークも変更した

kubemaster : 192.168.50.11
kube1      : 192.168.50.12
kube2      : 192.168.50.13

そして、kubemasterはforwarded_portを利用することにした

config.vm.network :forwarded_port, host: 6443, guest: 6443

普通にkubeadm initしただけではダメだったので、一度kubeadmを再インストールする

$ yum erase -y kubelet kubeadm kubectl
$ yum install -y kubelet kubeadm kubectl

aokabin · 2018-11-10T04:56:55Z

設定をちゃんと描いてあげないといけないらしいのでnode-ipを追記

# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# 元々↓
# Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
# 変更↓
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.50.11"

リロード

$ systemctl daemon-reload
$ systemctl restart kubelet

そしたらkubeadm initする

$ kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.50.11 --service-cidr=10.244.0.0/16
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kubemaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.244.0.1 192.168.50.11]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [kubemaster localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [kubemaster localhost] and IPs [192.168.50.11 127.0.0.1 ::1]
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 23.007157 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node kubemaster as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node kubemaster as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubemaster" as an annotation
[bootstraptoken] using token: xxylzo.p8034umioekqkqk7
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.50.11:6443 --token xxylzo.p8034umioekqkqk7 --discovery-token-ca-cert-hash sha256:d6ee35fada8c77af01a46221541222e851d53c8f06465239fd95221504c25403

そして設定ファイルの更新、元々のファイルがあったので、上書きになる

$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

aokabin · 2018-11-10T04:58:42Z

一応kubeletとdockerが動いていることを確認

$ service kubelet status
Redirecting to /bin/systemctl status kubelet.service
* kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           `-10-kubeadm.conf
   Active: active (running) since Fri 2018-11-09 16:36:03 UTC; 54s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 3422 (kubelet)
    Tasks: 16
   Memory: 33.3M
   CGroup: /system.slice/kubelet.service
           `-3422 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --...

Nov 09 16:36:35 kubemaster kubelet[3422]: E1109 16:36:35.854064    3422 kubelet.go:2167] Container runtime network not ready: Network...tialized
Nov 09 16:36:38 kubemaster kubelet[3422]: W1109 16:36:38.618982    3422 cni.go:188] Unable to update cni config: No networks found in...ni/net.d
Nov 09 16:36:38 kubemaster kubelet[3422]: E1109 16:36:38.619276    3422 kubelet.go:2167] Container runtime network not ready: Network...tialized
Nov 09 16:36:43 kubemaster kubelet[3422]: I1109 16:36:43.420389    3422 transport.go:132] certificate rotation detected, shutting dow...dentials
Nov 09 16:36:43 kubemaster kubelet[3422]: W1109 16:36:43.626766    3422 cni.go:188] Unable to update cni config: No networks found in...ni/net.d
Nov 09 16:36:43 kubemaster kubelet[3422]: E1109 16:36:43.626944    3422 kubelet.go:2167] Container runtime network not ready: Network...tialized
Nov 09 16:36:48 kubemaster kubelet[3422]: W1109 16:36:48.628408    3422 cni.go:188] Unable to update cni config: No networks found in...ni/net.d
Nov 09 16:36:48 kubemaster kubelet[3422]: E1109 16:36:48.628591    3422 kubelet.go:2167] Container runtime network not ready: Network...tialized
Nov 09 16:36:53 kubemaster kubelet[3422]: W1109 16:36:53.630802    3422 cni.go:188] Unable to update cni config: No networks found in...ni/net.d
Nov 09 16:36:53 kubemaster kubelet[3422]: E1109 16:36:53.631069    3422 kubelet.go:2167] Container runtime network not ready: Network...tialized
Hint: Some lines were ellipsized, use -l to show in full.


$ service docker status
Redirecting to /bin/systemctl status docker.service
* docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-11-09 15:16:51 UTC; 1h 20min ago
     Docs: https://docs.docker.com
 Main PID: 936 (dockerd)
    Tasks: 153
   Memory: 102.2M
   CGroup: /system.slice/docker.service
           |- 936 /usr/bin/dockerd
           |-1073 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --...
           |-3500 docker-containerd-shim 73269b813b6b656aad79ed19d531c44d386e433402c93d26febd27d0c4823315 /var/run/docker/libcontainerd/73269...
           |-3521 docker-containerd-shim c1c2cbf3d831105ad201d48ebad08d32cf76244d48ffa32b8610178c2ec11b3b /var/run/docker/libcontainerd/c1c2c...
           |-3548 docker-containerd-shim 6eb76602a066061832b6737da248c61e392c89958979635f458b5b824c633d2f /var/run/docker/libcontainerd/6eb76...
           |-3551 docker-containerd-shim 15ebb110a6bf8f7df0b49d80fc88cc5d242c853edccfc245f8c5008719a1c56f /var/run/docker/libcontainerd/15ebb...
           |-3636 docker-containerd-shim 3519e7fd19273c43285f1a16531a70537817ea4bc475227d98bfdf2b55002c7e /var/run/docker/libcontainerd/3519e...
           |-3650 docker-containerd-shim e2a735ce7d15af78948a5a32cc49557bb4eebd7e0424e755b603583b13af6c2b /var/run/docker/libcontainerd/e2a73...
           |-3667 docker-containerd-shim d4aa03e56aca5809b811ca43473e866862854b87dc35ed08b88563646602000c /var/run/docker/libcontainerd/d4aa0...
           |-3697 docker-containerd-shim e5eed98972db7c34c2bf2f4b95b37d3358290dc82102cc6f1b595dced124004f /var/run/docker/libcontainerd/e5eed...
           |-3864 docker-containerd-shim 244532765d47846f11a28c2ba5159a2b723b582e784df52a7e92e197870f6818 /var/run/docker/libcontainerd/24453...
           `-3903 docker-containerd-shim ae88eae135f6987a5c7847805e55346d67be5d3ad5cfb7527005e8069fc55b48 /var/run/docker/libcontainerd/ae88e...

Nov 09 16:35:52 kubemaster dockerd[936]: time="2018-11-09T16:35:52.505500347Z" level=error msg="Handler for GET /v1.27/containers/k8s...v1.12.2"
Nov 09 16:35:52 kubemaster dockerd[936]: time="2018-11-09T16:35:52.521849285Z" level=error msg="Handler for GET /v1.27/containers/k8s...v1.12.2"
Nov 09 16:35:52 kubemaster dockerd[936]: time="2018-11-09T16:35:52.536337367Z" level=error msg="Handler for GET /v1.27/containers/k8s...use:3.1"
Nov 09 16:35:52 kubemaster dockerd[936]: time="2018-11-09T16:35:52.549450018Z" level=error msg="Handler for GET /v1.27/containers/k8s...:3.2.24"
Nov 09 16:35:52 kubemaster dockerd[936]: time="2018-11-09T16:35:52.563145917Z" level=error msg="Handler for GET /v1.27/containers/k8s...s:1.2.2"
Nov 09 16:36:05 kubemaster dockerd[936]: time="2018-11-09T16:36:05.263083879Z" level=warning msg="Unknown healthcheck type 'NONE' (ex...5002c7e"
Nov 09 16:36:05 kubemaster dockerd[936]: time="2018-11-09T16:36:05.455579587Z" level=warning msg="Unknown healthcheck type 'NONE' (ex...602000c"
Nov 09 16:36:05 kubemaster dockerd[936]: time="2018-11-09T16:36:05.463108397Z" level=warning msg="Unknown healthcheck type 'NONE' (ex...124004f"
Nov 09 16:36:05 kubemaster dockerd[936]: time="2018-11-09T16:36:05.531169019Z" level=warning msg="Unknown healthcheck type 'NONE' (ex...3af6c2b"
Nov 09 16:36:36 kubemaster dockerd[936]: time="2018-11-09T16:36:36.518603805Z" level=warning msg="Unknown healthcheck type 'NONE' (ex...fc55b48"
Hint: Some lines were ellipsized, use -l to show in full.


$ lsof -i:6443
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kubelet   3422 root   31u  IPv4  51321      0t0  TCP kubemaster:51358->kubemaster:sun-sr-https (ESTABLISHED)
kube-cont 3665 root    6u  IPv4  48331      0t0  TCP kubemaster:51176->kubemaster:sun-sr-https (ESTABLISHED)
kube-cont 3665 root    7u  IPv4  48453      0t0  TCP kubemaster:51332->kubemaster:sun-sr-https (ESTABLISHED)
kube-apis 3692 root    3u  IPv6  47481      0t0  TCP *:sun-sr-https (LISTEN)
kube-apis 3692 root   73u  IPv6  48418      0t0  TCP kubemaster:sun-sr-https->kubemaster:51108 (ESTABLISHED)
kube-apis 3692 root   74u  IPv6  48454      0t0  TCP kubemaster:sun-sr-https->kubemaster:51332 (ESTABLISHED)
kube-apis 3692 root   75u  IPv6  48990      0t0  TCP kubemaster:sun-sr-https->kubemaster:51350 (ESTABLISHED)
kube-apis 3692 root   78u  IPv6  52233      0t0  TCP kubemaster:sun-sr-https->kubemaster:51358 (ESTABLISHED)
kube-apis 3692 root   87u  IPv6  48432      0t0  TCP kubemaster:sun-sr-https->kubemaster:51176 (ESTABLISHED)
kube-apis 3692 root   92u  IPv6  47679      0t0  TCP localhost:50084->localhost:sun-sr-https (ESTABLISHED)
kube-apis 3692 root  113u  IPv6  47699      0t0  TCP localhost:sun-sr-https->localhost:50084 (ESTABLISHED)
kube-sche 3715 root    5u  IPv4  48297      0t0  TCP kubemaster:51108->kubemaster:sun-sr-https (ESTABLISHED)
kube-prox 3919 root    3u  IPv4  51013      0t0  TCP kubemaster:51350->kubemaster:sun-sr-https (ESTABLISHED)

aokabin · 2018-11-10T05:09:33Z

そしたら一応kubectlが使えるようになってるはず

$ kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
kubemaster   NotReady   master   90s   v1.12.2

not readyだけど動いてるね！
そしたらFlannelあたりをやる

Kubernetes v1.10 クラスタをVagrantで構築したメモ - Qiita # ポッド・ネットワークの設定
今回masterから取ってきた(↑はバージョン指定だけど)

$ curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ vi kube-flannel.yaml

こんな感じで書き換え

...
containers:
- name: kube-flannel
  image: quay.io/coreos/flannel:v0.10.0-amd64
  command:
  - /opt/bin/flanneld
  args:
  - --ip-masq
  - --kube-subnet-mgr
  - --iface=eth1 # 127行目くらいにこれを追加
...

どうや！

$ kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
$ kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
kubemaster   NotReady   master   20m   v1.12.2
$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
kubemaster   Ready    master   21m   v1.12.2

一瞬できてなくて焦ったけど、できてそう

aokabin · 2018-11-10T05:55:57Z

そういえば全部のhosts変更しないといけない、と思ってkubeclu_3を立ち上げようとしたらエラーがでた
どうやら、ネットワークが競合してるとダメらしい

というのも、ipを192.168.50.13とかにしてたんだけど、たまたま今繋げてるWi-Fiが同じセグメントだった！
だからNGなので、ipをあんまり被らないやつにしないといけなさそう

kubemaster : 192.233.133.11
kube1      : 192.233.133.12
kube2      : 192.233.133.13

これで起動はしたけど、もう一回設定し直さないといけないや

設定の変更
↑を参考に以下を進める

$ vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
$ systemctl daemon-reload
$ systemctl restart kubelet
$ kubeadm reset
[reset] WARNING: changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] are you sure you want to proceed? [y/N]: y
[preflight] running pre-flight checks


$ kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.233.133.11 --service-cidr=10.244.0.0/16
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [kubemaster localhost] and IPs [192.233.133.11 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [kubemaster localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kubemaster kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.244.0.1 192.233.133.11]
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 20.003715 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node kubemaster as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node kubemaster as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kubemaster" as an annotation
[bootstraptoken] using token: add54o.7fcs2so6fyrq2s5y
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.233.133.11:6443 --token add54o.7fcs2so6fyrq2s5y --discovery-token-ca-cert-hash sha256:9133407db5a1eb5c4ec521230ba6f8e65a663328f0be8e4424678995f9389aaf


$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp: overwrite '/root/.kube/config'? y
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

そしたらそのままさっきのkube-flannel.yamlを動かす

$ kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

flannelはどのネットワークインターフェースを使うかを指定しただけなので、そのまま打っても問題なし
逆に言えば、ネットワークインターフェースが変わるなら変更必要ですが...

これでクラスタができているので、前の設定が引き継がれてたりするのか確認してみる

# $ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.233.133.11:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

ないですね！

一応再起動試験も実施してみた、動いてそうでした
念のため確認

$ systemctl list-unit-files | grep kube
kubelet.service                               enabled

大丈夫そうですね！

aokabin · 2018-11-10T05:59:03Z

Node側は再起動するとちゃんとReadyになるのかな？
→ なりました

そしてネットワークの問題で接続できなかった問題

$ ping 192.233.133.11
PING 192.233.133.11 (192.233.133.11): 56 data bytes
64 bytes from 192.233.133.11: icmp_seq=0 ttl=64 time=5.999 ms

なぜ通るんだ...
もうipが悪悪だった可能性がめっちゃ浮上している

forwardedがあんまり意味ないなこれ。。。
一旦おいといて、設定を作っていきましょう。

これはローカルからkubemasterに接続する · Issue #6 · aokabin/kubernetes-practiceを参考にやっていった

aokabin changed the title ~~Kubernetes ClusterをVMe~~ Kubernetes ClusterをVMde Sep 20, 2018

aokabin changed the title ~~Kubernetes ClusterをVMde~~ Kubernetes ClusterをVMで作ってみる Sep 20, 2018

aokabin mentioned this issue Oct 17, 2018

CentOSでKubernetesを組み上げるまとめを書く #5

Open

aokabin closed this as completed Oct 18, 2018

aokabin mentioned this issue Oct 24, 2018

ローカルからkubemasterに接続する #6

Closed

aokabin reopened this Nov 9, 2018

aokabin closed this as completed Nov 10, 2018

Kubernetes ClusterをVMで作ってみる #1

Kubernetes ClusterをVMで作ってみる #1

Comments

aokabin commented Sep 20, 2018 • edited

How to install a Kubernetes cluster on CentOS 7 - TechRepublic

aokabin commented Sep 27, 2018 • edited

MacにVagrantでCentOS7環境を作成 - Qiita

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

How to install a Kubernetes cluster on CentOS 7 - TechRepublic

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018 • edited

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

aokabin commented Sep 27, 2018

Docker-ceのインストール

aokabin commented Sep 27, 2018

Docker-ceとは？

aokabin commented Sep 27, 2018

Kubernetesのインストール

aokabin commented Oct 10, 2018

Cgroupの変更

aokabin commented Oct 10, 2018

aokabin commented Oct 10, 2018

aokabin commented Oct 10, 2018

aokabin commented Oct 10, 2018 • edited

aokabin commented Oct 10, 2018

aokabin commented Oct 10, 2018

参考

aokabin commented Oct 11, 2018 • edited

aokabin commented Oct 11, 2018 • edited

参考

aokabin commented Oct 16, 2018

aokabin commented Oct 16, 2018

aokabin commented Oct 16, 2018 • edited

aokabin commented Oct 17, 2018

Kubernetesの設定

aokabin commented Oct 17, 2018 • edited

flannel networkをデプロイ

参考

aokabin commented Oct 17, 2018

aokabin commented Oct 17, 2018

aokabin commented Oct 17, 2018 • edited

参考

aokabin commented Oct 17, 2018 • edited

aokabin commented Oct 17, 2018

aokabin commented Oct 17, 2018 • edited

参考

aokabin commented Oct 17, 2018 • edited

参考

aokabin commented Oct 17, 2018

再起動処理

参考

aokabin commented Nov 9, 2018

aokabin commented Nov 10, 2018

aokabin commented Nov 10, 2018 • edited

aokabin commented Nov 10, 2018

aokabin commented Nov 10, 2018 • edited

aokabin commented Nov 10, 2018

aokabin commented Nov 10, 2018 • edited

aokabin commented Sep 20, 2018 •

edited

aokabin commented Sep 27, 2018 •

edited

aokabin commented Sep 27, 2018 •

edited

aokabin commented Oct 10, 2018 •

edited

aokabin commented Oct 11, 2018 •

edited

aokabin commented Oct 11, 2018 •

edited

aokabin commented Oct 16, 2018 •

edited

aokabin commented Oct 17, 2018 •

edited

aokabin commented Oct 17, 2018 •

edited

aokabin commented Oct 17, 2018 •

edited

aokabin commented Oct 17, 2018 •

edited

aokabin commented Oct 17, 2018 •

edited

aokabin commented Nov 10, 2018 •

edited

aokabin commented Nov 10, 2018 •

edited

aokabin commented Nov 10, 2018 •

edited