引言
k8s风靡的时代已经到来,小猿决定从今天开始逐步学习k8s,首先在实战之前得说说小猿在实践k8s安装的两个实践问题。
第一个是版本问题,因为k8s是最简单最粗暴的理解就是管理docker容器的一个工具,因此docker和k8s的版本一致是非常重要的,不要忽视此问题的重要性,否则可能会让你焦头烂额,至于版本问题去哪里查,小猿也没有找到比较官方的说法,有人说去github kubernetes的Changelog上找,可是有的版本能找到,有的版本找不到。
第二个问题是部署方法,小猿在此处使用的k8s 不使用二进制方法部署,如果童鞋想看需用二进制文件的方式部署,则请看这个大佬的文章。
kubernetes基本概念
What is Kubernetes?
Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
Kubernetes Components
A Kubernetes cluster consists of the components that represent the control plane and a set of machines called nodes.
The Kubernetes API
The Kubernetes API lets you query and manipulate the state of objects in Kubernetes. The core of Kubernetes’ control plane is the API server and the HTTP API that it exposes. Users, the different parts of your cluster, and external components all communicate with one another through the API server.
节点架构
Container Runtime
The container runtime is responsible for managing the life cycle of each container running in the node. After a pod is scheduled on the node, the runtime pulls the images specified by the pod from the registry. When a pod is terminated, the runtime kills the containers that belong to the pod. Kubernetes may communicate with any Open Container Initiative (OCI)-compliant container runtime, including Docker and CRI-O.
The OCI is a standard that defines the runtime specification and image specification, with a goal to drive standardization of container runtimes and image formats.
Kubelet
The Kubelet is the Kubernetes agent whose responsibility is to interact with the container runtime to perform operations such as starting, stopping and maintaining containers.
Each kubelet also monitors the state of the pods. When a pod does not meet the desired state as defined by the deployment, it may be restarted on the same node. The node’s status is transmitted to the head every few seconds via heartbeat messages. If the head detects a node failure, the replication controller observes this state change and schedules the pods on other healthy nodes.
Kube-Proxy
The kube-proxy component is implemented as a network proxy and a load balancer that orchestrates the network to route requests to the appropriate pods. It routes traffic to the appropriate pod based on the associated service name and the port number of an incoming request. It also takes advantage of OS-specific networking capabilities by manipulating the policies and rules defined through iptables. Each kube-proxy component may be integrated with network layers such as Calico and Flannel.
Logging Layer
The orchestrator makes frequent use of logging as a means for gathering resource usage and performance metrics for containers on each node, such as CPU, memory, file and network usage. The Cloud Native Computing Foundation hosts a software component that provides a unified logging layer for use with Kubernetes or other orchestrators, called Fluentd. This component generates metrics that the Kubernetes head controller needs in order to keep track of available cluster resources, as well as the health of the overall infrastructure.
k8s集群架构简要示意图
Pods
Pods are the smallest deployable units of computing that you can create and manage in Kubernetes.
A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod’s contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific “logical host”: it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host.
As well as application containers, a Pod can contain init containers that run during Pod startup. You can also inject ephemeral containers for debugging if your cluster offers this
Deployments
A Deployment provides declarative updates for Pods and ReplicaSets.
You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.
DNS
While the other addons are not strictly required, all Kubernetes clusters should have cluster DNS, as many examples rely on it.
Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.
Containers started by Kubernetes automatically include this DNS server in their DNS searches.
service
While the control plane and the worker nodes form the core cluster infrastructure, the workloads are the containerized applications deployed in Kubernetes.
After developing and testing a microservice, the developers package it as a container, which is the smallest unit of deployment packaged as a pod. A set of containers belonging to the same application is grouped, packaged, deployed and managed within Kubernetes.
Kubernetes exposes primitives for deployment, while constantly scaling, discovering, and monitoring the health of these microservices. Namespaces are typically used to logically separate one application from the other. They act as a logical cluster by providing a well-defined boundary and scope for all resources and services belonging to an application.
Within a namespace, the following Kubernetes primitives are deployed
controller
In Kubernetes, controllers augment pods by adding additional capabilities, such as desired configuration state and runtime characteristics.
A deployment brings declarative updates to pods. It guarantees that the desired state is always maintained by tracking the health of the pods participating in the deployment. Each deployment manages a ReplicaSet, which maintains a stable set of replica pods running at any given time, as defined by the desired state.
Deployments bring PaaS-like capabilities to pods through scaling, deployment history and rollback features. When a deployment is configured with a minimum replica count of two, Kubernetes ensures that at least two pods are always running, which brings fault tolerance. Even when deploying the pod with just one replica, it is highly recommended to use a deployment controller instead of a plain vanilla pod specification.
A statefulset is similar to a deployment, but is meant for pods that need persistence and a well-defined identifier and guaranteed order of creation. For workloads such as database clusters, a statefulset controller will create a highly available set of pods in a given order that have a predictable naming convention. Stateful workloads that need to be highly available, such as Cassandra, Kafka, ZooKeeper, and SQL Server, are deployed as statefulsets in Kubernetes.
To force a pod to run on every node of the cluster, a DaemonSet controller can be used. Since Kubernetes automatically schedules a DaemonSet in newly provisioned worker nodes, it becomes an ideal candidate to configure and prepare the nodes for the workload. For example, if an existing network file system (NFS) or Gluster file share has to be mounted on the nodes before deploying the workload, it is recommended to package and deploy the pod as a DaemonSet. Monitoring agents are good candidates to be used as a DaemonSet, to ensure that each node runs the monitoring agent.
For batch processing and scheduling jobs, pods can be packaged for a run-to-completion job or a cron job. A job creates one or more pods and ensures that a specified number of them successfully terminate. Pods configured for run to completion execute the job and exit, while a cron job will run a job based on the schedule defined in the crontab format.
Controllers define the life cycle of pods based on the workload characteristics and their execution context.
Now that we understand the basics of the Kubernetes control plane and how applications run on Kubernetes, it’s time to talk about service discovery to better understand how production workloads run in Kubernetes.
准备工作
三个centos7同一版本的虚拟机
网络设置
三台虚拟机的网络模式为Network Address Tranlation。
网络配置文件既需要修改ip addr 也需要修改mac,典型的配置文件如下所示
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
#BOOTPROTO=dhcp
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=a32c82c0-4b1a-489c-bc08-1ffbed671f3f
DEVICE=ens33
ONBOOT=yes
GATEWAY=192.168.177.2
IPADDR=192.168.177.180
NETMASK=255.255.255.0
MACADDR=00:07:E9:05:E8:B4
#set dns server or the docker can not get the network resources
DNS1=8.8.8.8
DNS2=114.114.114.114
PEERDNS=no
ZONE=public
PREFIX=24
CONNECTION_METERED=no
其他的两台照此配置即可,需要高边MAC 和ipaddr 其他的uuid 也最好变改一下。
防火墙设置
systemctl stop firewalld
systemctl disable firewalld
关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
关闭swap
#临时
swapoff -a
#永久
sed -ri 's/.*swap.*/#&/' /etc/fstab
#验证
free -g
##swap 必须为 0;
添加主机名与ip对应的关系
192.168.177.180 k8smaster
192.168.177.181 k8snode1
192.168.177.182 k8snode2
上图中小猿以master节点为例。
hostnamectl set-hostname k8smaster
将桥接的 IPv4 流量传递到 iptables 的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
同步时间
###查看时间
date
yum install -y ntpdate
ntpdate time.windows.com
k8s安装
据说k8s 在1.21 完全抛弃 Docker了,届时将实现OCI (Container Runtime Interface),用containerd(CNCF毕业项目)作为替代品,但小猿这任然以docker为例子做讲解。首先要确定版本
docker 版本
yum list docker-ce --showduplicates | sort -r
docker 版本采用19.03.6
kubernetes版本 小猿的案例采用kubernetes 1.7.3
docker 安装
移除以前的docker 版本
sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
安装必须的依赖
sudo yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
设置 docker repo 的 yum 位置
sudo yum-config-manager \
--add-repo \
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安装docker的时候需要指定版本
yum list docker-ce --showduplicates | sort -r
# docker 指定版本安装
#sudo yum install docker-ce-3:19.03.6-3.el7 docker-ce-cli-3:19.03.6-3.el7 containerd.io
sudo yum install -y --setopt=obsoletes=0 docker-ce-19.03.6 docker-ce-selinux-19.03.6 docker-ce-cli-19.03.6
配置docker 加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://a61nkhlq.mirror.aliyuncs.com"]
}
EOF
//重启docker的伴随线程
sudo systemctl daemon-reload
//重启docker
sudo systemctl restart docker
sudo systemctl enable docke
kubernetes安装
卸载kubernetes历史版本
卸载
kubeadm reset -f
modprobe -r ipip
lsmod
rm -rf ~/.kube/
rm -rf /etc/kubernetes/
rm -rf /etc/systemd/system/kubelet.service.d
rm -rf /etc/systemd/system/kubelet.service
rm -rf /usr/bin/kube*
rm -rf /etc/cni
rm -rf /opt/cni
rm -rf /var/lib/etcd
rm -rf /var/etcd
yum clean all
yum remove kube*
查看yum源中的版本
yum list|grep kube
添加阿里云 yum 源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
开始安装
yum install -y kubelet-1.17.3 kubeadm-1.17.3 kubectl-1.17.3
systemctl enable kubelet
systemctl start kubelet
部署 k8smaster
初始化之前
k8s初始化时需要下载一些依赖,因此需要执行拉去命令,下面的命令直接做成shell 可执行文件,注意下面的脚本需要在master节点上执行即可。
images=(
kube-apiserver:v1.17.3
kube-proxy:v1.17.3
kube-controller-manager:v1.17.3
kube-scheduler:v1.17.3
coredns:1.6.5
etcd:3.4.3-0
pause:3.1
)
for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done
所有需要的镜像下载完毕
master节点初始化
kubeadm init \
--apiserver-advertise-address=192.168.177.180 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=192.96.0.0/16 \
--pod-network-cidr=192.244.0.0/16
按照提示来执行。
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
发布pod网络
网络种类很多,我们采用Flannel网络插件,其配置最好先下载,再使用,
配置文件
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "192.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni-plugin
#image: flannelcni/flannel-cni-plugin:v1.0.1 for ppc64le and mips64le (dockerhub limitations may apply)
image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.1
command:
- cp
args:
- -f
- /flannel
- /opt/cni/bin/flannel
volumeMounts:
- name: cni-plugin
mountPath: /opt/cni/bin
- name: install-cni
#image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)
image: rancher/mirrored-flannelcni-flannel:v0.17.0
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
#image: flannelcni/flannel:v0.17.0 for ppc64le and mips64le (dockerhub limitations may apply)
image: rancher/mirrored-flannelcni-flannel:v0.17.0
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: EVENT_QUEUE_DEPTH
value: "5000"
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: xtables-lock
mountPath: /run/xtables.lock
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
配置文件准备好后可以放到主节点中,主节点需要用配置文件建立pod网络
应用配置文件
##应用flannel
kubectl apply -f kube-flannel.yml
## 卸载应用
kubectl delete -f kube-flannel.yml
如果遇到下面的警告或者错误,请运行
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> /etc/profile
source /etc/profile
应用成功
至此pod配置基本完成
获取所有名称空间
kubectl get ns
kubectl get pods --all-namespaces
确保所有的名称空间及名称处于running状态
查看节点状态。
加入节点
主要是另个从节点执行下面的命令。
kubeadm join 192.168.177.180:6443 --token q5esd5.ysl9kvp55hbedai5 \
--discovery-token-ca-cert-hash sha256:0c2ba2e9024dfbef1a221750c5527a9ef2c664f19797aba65f113b9512605e4f
再次查看节点数量
加入成功。
修改接口角色属性名
#增加角色属性名
kubectl label nodes k8snode1 node-role.kubernetes.io/node1=node1
#删除角色属性名
kubectl label nodes k8snode1 node-role.kubernetes.io/node1-
监控 pod 进度
watch kubectl get pod -n kube-system -o wide
至此小猿的kubernetes就安装成功了,接下来就是部署相关项目,期待下节分享。