Kubernetes 环境搭建

kubernetes

指引:https://kubernetes.io/docs/tasks/tools/install-kubectl/

安装 VM 驱动

腾讯云 CVM 未能启用驱动启动 minikube,安装驱动的过程可忽略。

- VirtualBox

CentOS:虚拟机不支持内部 VM,未在物理机上尝试

1
2
3
4
5
6
7
8
9
cd /etc/yum.repos.d/
wget http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
yum update

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install -y binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms
yum install -y VirtualBox-5.2

service vboxdrv start

MAC 上可以直接下载安装:https://www.virtualbox.org/wiki/Downloads

- KVM2

安装成功,但 minikube start 时未能尝试成功

https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver

1
2
3
4
5
6
yum -y install libvirt-daemon-kvm qemu-kvm

curl -Lo docker-machine-driver-kvm2 https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 \
&& chmod +x docker-machine-driver-kvm2 \
&& sudo cp docker-machine-driver-kvm2 /usr/local/bin/ \
&& rm docker-machine-driver-kvm2

附:查看虚拟机网络以及使用不同的网络启动 minikube

1
2
3
4
5
yum -y install libvirt virt-install bridge-utils
# 查看虚拟机网络
virsh net-list --all
# 使用不同的虚拟机网络(minikube-net)
minikube start --vm-driver kvm2 --kvm-network minikube-net

- 其他驱动

指引:https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm-driver

安装 kubectl

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# CentOS
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl

# MAC: 方式一
brew install kubernetes-cli
# MAC:方式二
curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.5.1/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/

# 安装校验
kubectl version

# 确保服务已启动
systemctl enable kubelet.service
systemctl start kubelet.service

安装 minikube

指引:https://github.com/kubernetes/minikube/releases

1
2
3
4
5
6
7
8
# CentOS
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.29.0/minikube-linux-amd64 \
&& chmod +x minikube \
&& sudo cp minikube /usr/local/bin/ \
&& rm minikube

# MAC
brew cask install minikube

运行

(使用 KVM2 驱动)运行之前

1
2
3
4
5
6
7
8
9
10
11
# 启动服务
systemctl enable libvirtd && systemctl start libvirtd

# 查看进程文件
ls -l /var/run/libvirt/libvirt-sock

# 若进程文件不存在,则修改配置:
vi /etc/libvirt/libvirtd.conf

# 取消行前注释,重新启动服务
#unix_sock_dir = "/var/run/libvirt"

拉取 k8s 相关镜像

原本在 minikube start 命令中会自动拉取镜像,但是因为众所周知的原因,我们无法成功拉取到镜像。这里可以使用阿里 docker 容器镜像服务:https://dev.aliyun.com/search.html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 进入虚拟机拉取,而非本机
minikube ssh

# 拉取镜像
docker pull registry.cn-hangzhou.aliyuncs.com/k8sth/kube-apiserver-amd64:v1.10.0
docker pull registry.cn-hangzhou.aliyuncs.com/k8sth/kube-controller-manager-amd64:v1.10.0
docker pull registry.cn-hangzhou.aliyuncs.com/k8sth/kube-scheduler-amd64:v1.10.0
docker pull registry.cn-hangzhou.aliyuncs.com/k8sth/etcd-amd64:3.1.12
docker pull registry.cn-shenzhen.aliyuncs.com/kubernetes_google/kubernetes-dashboard-amd64:v1.10.0

# 修改 tag
docker tag registry.cn-hangzhou.aliyuncs.com/k8sth/kube-apiserver-amd64:v1.10.0 k8s.gcr.io/kube-apiserver-amd64:v1.10.0
docker tag registry.cn-hangzhou.aliyuncs.com/k8sth/kube-controller-manager-amd64:v1.10.0 k8s.gcr.io/kube-controller-manager-amd64:v1.10.0
docker tag registry.cn-hangzhou.aliyuncs.com/k8sth/kube-scheduler-amd64:v1.10.0 k8s.gcr.io/kube-scheduler-amd64:v1.10.0
docker tag registry.cn-hangzhou.aliyuncs.com/k8sth/etcd-amd64:3.1.12 k8s.gcr.io/etcd-amd64:3.1.12
docker tag registry.cn-shenzhen.aliyuncs.com/kubernetes_google/kubernetes-dashboard-amd64:v1.10.0 k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0

Kubenetes

指引:https://kubernetes.io/docs/setup/minikube/#quickstart

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# 在代理下运行时,需要指定正确的代理,否则 minikube 在内部访问网络会有问题
# 注意,代理地址应当是在虚拟机中能够访问的地址,不能使用 127.0.0.1、localhost 这种
https_proxy=http://web-proxy.oa.com:8080
minikube start \
--docker-env http_proxy=http://web-proxy.oa.com:8080 \
--docker-env https_proxy=http://web-proxy.oa.com:8080 \
--docker-env no_proxy=192.168.99.0/24
--vm-driver=none

# 安装和启动过程中可能产生错误,通过该命令查看日志
minikube logs -f

# 启动 k8s( CentOS 上不使用任何 vm 驱动,MAC 上默认可使用 VirtualBox)
minikube start \
--network-plugin=cni \
--container-runtime=containerd \
--bootstrapper=kubeadm
--vm-driver=none

# Dashboard
# https://github.com/kubernetes/dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl proxy

# 部署及运行镜像
kubectl run hello-minikube --image=registry.cn-hangzhou.aliyuncs.com/acs/echoserver:1.4 --port=8080

# 导出服务
kubectl expose deployment hello-minikube --type=NodePort

# 查询 Pod
kubectl get pod
kubectl get pods --all-namespaces

# 查看状态
kubectl describe --namespace=kube-system po kubernetes-dashboard-6f4cfc5d87-d647l
kubectl logs kubernetes-dashboard-6f4cfc5d87-x976v --namespace=kube-system
kubectl cluster-info

# 调用服务
curl $(minikube service hello-minikube --url)

# 删除服务
kubectl delete services hello-minikube

# 删除部署
kubectl delete deployment hello-minikube

# 停止 k8s
minikube stop

向 kubernetes 添加账号

https://github.com/kubernetes/dashboard/wiki/Creating-sample-user

FAQ

failed to create kubelet: misconfiguration: kubelet cgroup driver: “cgroupfs” is different from docker cgr…driver: “systemd”

该问题因为 docker 的配置和 kubelet 的配置不一致导致。

使用 docker info 打印 docker 信息:

1
2
3
4
5
6
[root@VM_0_16_centos kubelet.service.d]# docker info | grep Driver
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
WARNING: You're not using the default seccomp profile
Storage Driver: devicemapper
Logging Driver: journald
Cgroup Driver: systemd

而查看 kubelet 服务的启动参数(--cgroup-driver),其设置为 cgroupfs

1
2
3
4
5
6
7
8
9
10
11
[root@VM_0_16_centos kubelet.service.d]# more /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[Unit]
Wants=docker.socket

[Service]
ExecStart=
ExecStart=/usr/bin/kubelet --hostname-override=minikube --cluster-domain=cluster.local --cgroup-driver=cgroupfs --authorization-mode=Webhook --client-ca-file=/var/lib/minikube/certs/ca.crt --fail-swap-on=false --kubeconfig=/etc/kubernetes/ku
belet.conf --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cluster-dns=10.96.0.10 --cadvisor-port=0

[Install]

此时,修改 docker 的服务参数(vi /usr/lib/systemd/system/docker.service),将其中的 --exec-opt native.cgroupdriver 参数值改为 cgroupfs

然后,重启 docker,重启 kubelet

1
systemctl daemon-reload && systemctl restart docker && systemctl restart kubelet

[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

解决方案:

1
2
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

Unable to update cni config: No networks found in /etc/cni/net.d

yum install -y kubernetes-cni

参考

https://www.jianshu.com/p/a42eeb66a19c