十、Kubernetes实践篇
10.1) Kubernetes安装三种方式(官方提供)
10.1.1)minikube
Minikube是一个工具,可以在本地快速运行一个单点的Kubernetes,尝试Kubernetes或日常开发的用户使用。不能用于生产环境。
官方文档:
10.1.2)kubeadm
kubeadm可帮助你快速部署一套kubernetes集群。kubeadm设计目的为新用户开始尝试kubernetes提供一种简单的方法。目前是Beta版。
官方文档:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
https://kubernetes.io/docs/setup/independent/install-kubeadm/
10.1.3)二进制包
从官方下载发行版的二进制包,手动部署每个组件,组成Kubernetes集群。目前企业生产环境中主要使用该方式。
下载地址:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113
10.2) Kubernetes安装规划
10.2.1)基本资源
操作系统
Ubuntu 16.04+
Debian 9
CentOS 7
RHEL 7
Fedora 25/26 (best-effort)
其它要求:
内存2GB + ,2核CPU +(生产具体规划)
集群节点之间可以通信
每个节点唯一主机名,MAC地址和product_uuid
检查MAC地址:使用ip link或者ifconfig -a
检查product_uuid:cat /sys/class/dmi/id/product_uuid
禁止swap分区,这样才能使kubelet正常工作
10.2.2)节点规划
192.168.111.134 node7 --node1
192.168.111.135 node8 --node2
192.168.111.136 node9 --master
10.3) Kubernetes准备环境
10.3.1)关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
10.3.2)关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
10.3.3)关闭swap
swapoff -a # 临时
swapoff -a && sysctl -w vm.swappiness=0
vim /etc/fstab # 永久
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
更改swap限制
cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--fail-swap-on=false
10.3.4)hosts配置
192.168.111.134 node7
192.168.111.135 node8
192.168.111.136 node9
10.3.5)时间同步
ntpdate 1.cn.pool.ntp.org
yum install ntpdate –y
#配置ntp
client
Server:
10.3.6)添加ssh互信
ssh-keygen -t rsa
for i in node7 node8 node9;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
yum install -y bridge-utils.x86_64
10.4) Kubernetes集群安装(kubeadm)
10.4.1)系统资源参数
* hardnofile 65536
* softnofile 65536
* hardnproc 65536
* softnproc 65536
编辑配置文件/etc/sysctl.conf,添加以下内容:
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_keepalive_time = 1200
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 16384
保存退出后执行sysctp -p生效
10.4.2)docker安装
2.6)Docker 安装管理
10.4.3)kubeadm相关工具安装
kubeadm: 引导集群的命令
kubelet:集群中运行任务的代理程序
kubectl:命令行管理工具
添加阿里云YUM软件源
#cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
或者:
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl kubernetes-cni
yum install -y kubelet-1.13.5-0.x86_64 kubeadm-1.13.5-0.x86_64 kubectl-1.13.5-0.x86_64 kubernetes-cni
systemctl enable kubelet && systemctl start kubelet
注意:使用Docker时,kubeadm会自动检查kubelet的cgroup驱动程序,并/var/lib/kubelet/kubeadm-flags.env在运行时将其设置在文件中。如果使用的其他CRI,则必须在/etc/default/kubelet中cgroup-driver值修改为cgroupfs:
cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
systemctl daemon-reload
systemctl restart kubelet
kubeadm 常用的命令
helpHelp about any command
initRun this command in order to set up the Kubernetes control plane. # master上执行,初始化所有的master组件
joinRun this on any machine you wish to join an existing cluster # node上执行,加入master
reset Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join'. # 清理 init,join的环境
token Manage bootstrap tokens. # token的增删查
upgrade Upgrade your cluster smoothly to a newer version with this command. # 更新集群
version Print the version of kubeadm
10.4.4)下载kubernetes的相关镜像
K8S_VERSION=v1.13.5
ETCD_VERSION=3.2.24
DASHBOARD_VERSION=v1.8.3
FLANNEL_VERSION=v0.10.0-amd64
DNS_VERSION=1.2.6
PAUSE_VERSION=3.1
基本组件
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION
网络组件
docker pull quay.io/coreos/flannel:$FLANNEL_VERSION
修改tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION k8s.gcr.io/kube-apiserver:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION k8s.gcr.io/kube-controller-manager:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION k8s.gcr.io/kube-scheduler:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION k8s.gcr.io/kube-proxy:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION k8s.gcr.io/etcd:$ETCD_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION k8s.gcr.io/coredns:$DNS_VERSION
10.4.5)运行kubeadm init 安装master
配置国内镜像加速
cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com" ]
}
kubeadm config
kubeadm config upload from-file 由配置文件上传到集群中生成ConfigMap
kubeadm config upload from-flags 由配置参数生成ConfigMap
kubeadm config view 查看当前集群中的配置值
kubeadm config print init-defaults 输出init-defaults默认参数文件内容
kubeadm config print join-defaults 输出join-defaults默认参数文件内容
kubeadm config migrate 在新旧版本之间进行配置转换
kubeadm config images list 列出所需镜像列表
kubeadm config images pull 拉去镜像到本地
新建init-config.yaml文件定制镜像仓库地址和Pod地址段
cat init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
imageRepository: docker.io/dustise
kubernetesVersion: v1.14.0
networking:
podSubnet: "172.16.0.0/16 "
下载所需镜像
kubeadm config images pull --config=init-config.yaml
[config/images] Pulled docker.io/dustise/kube-apiserver:v1.14.0
[config/images] Pulled docker.io/dustise/kube-controller-manager:v1.14.0
[config/images] Pulled docker.io/dustise/kube-scheduler:v1.14.0
[config/images] Pulled docker.io/dustise/kube-proxy:v1.14.0
[config/images] Pulled docker.io/dustise/pause:3.1
[config/images] Pulled docker.io/dustise/etcd:3.3.10
[config/images] Pulled docker.io/dustise/coredns:1.3.1
查看默认参数文件
kubeadm config print init-defaults
出现[WARNING IsDockerSystemdCheck],是由于docker的Cgroup Driver和kubelet的Cgroup Driver不一致导致的,此处选择修改docker的和kubelet一致
docker info | grep Cgroup
Cgroup Driver: cgroupfs
编辑文件/usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd
systemctl daemon-reload
systemctl restart docker
docker info | grep Cgroup
Cgroup Driver: system
操作
修改配置
sed -e 's/KUBELET_CGROUP_ARGS=--cgroup-driver=systemd/KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs/' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
echo $DOCKER_CGROUPS
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
kubeadm init --kubernetes-version=1.13.5 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.111.136
kebeadm init需要加上参数
详细的参数介绍可看:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
10.4.5)运行kubeadm init 安装master
配置国内镜像加速
cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.docker-cn.com" ]
}
kubeadm config
kubeadm config upload from-file 由配置文件上传到集群中生成ConfigMap
kubeadm config upload from-flags 由配置参数生成ConfigMap
kubeadm config view 查看当前集群中的配置值
kubeadm config print init-defaults 输出init-defaults默认参数文件内容
kubeadm config print join-defaults 输出join-defaults默认参数文件内容
kubeadm config migrate 在新旧版本之间进行配置转换
kubeadm config images list 列出所需镜像列表
kubeadm config images pull 拉去镜像到本地
新建init-config.yaml文件定制镜像仓库地址和Pod地址段
#cat init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
imageRepository: docker.io/dustise
kubernetesVersion: v1.14.0
networking:
podSubnet: "172.16.0.0/16 "
下载所需镜像
kubeadm config images pull --config=init-config.yaml
[config/images] Pulled docker.io/dustise/kube-apiserver:v1.14.0
[config/images] Pulled docker.io/dustise/kube-controller-manager:v1.14.0
[config/images] Pulled docker.io/dustise/kube-scheduler:v1.14.0
[config/images] Pulled docker.io/dustise/kube-proxy:v1.14.0
[config/images] Pulled docker.io/dustise/pause:3.1
[config/images] Pulled docker.io/dustise/etcd:3.3.10
[config/images] Pulled docker.io/dustise/coredns:1.3.1
#
DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
echo $DOCKER_CGROUPS
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
kubeadm init --kubernetes-version=1.13.5 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.111.136
kebeadm init需要加上参数 详细的参数介绍可看:
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
使用kubeadm reset重置主机状态然后重新初始化
10.4.6)常规用户使用kubectl访问集群
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
10.4.7)安装pod网络插件Flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
安装网络插件weave
kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')
10.4.8)查看所有pod和节点
10.4.9)添加工作节点
在Node节点切换到root账号执行:
kubeadm join 192.168.111.136:6443 --token gf25fd.xntkm8qy5klmhrv6 --discovery-token-ca-cert-hash sha256:f409b76900e0bf4e334f1bc2b629a89f4e031744489c6bfe8d8233f9af7ecdd7
#格式:kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
10.4.10)安装配置访问dashboardb
安装dashboard
https://github.com/kubernetes/dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml
修改Dashboard Service 为NodePort类型
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard
进行部署
kubectl create -f kubernetes-dashboard.yaml
kubectl delete -f kubernetes-dashboard.yaml
查验
kubectl get svc --all-namespaces
创建管理员
cat k8s-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin
namespace: kube-system
使用token登录
https://192.168.111.136:30001/#!/login