通过ansible-playbook，以Kubeadm方式部署K8S集群（一主多从）。
kubernetes安装目录： /etc/kubernetes/
KubeConfig： ~/.kube/config
Version： v1.18.3
主机说明：
| 系统 | ip | 角色 | cpu | 内存 | hostname |
|---|---|---|---|---|---|
| CentOS 7.8 | 192.168.30.128 | master | >=2 | >=2G | master |
| CentOS 7.8 | 192.168.30.129 | node | >=2 | >=2G | node1 |
| CentOS 7.8 | 192.168.30.130 | node | >=2 | >=2G | node2 |
| CentOS 7.8 | 192.168.30.131 | node | >=2 | >=2G | node3 |
准备
- 将所有部署k8s集群的主机分组：
# vim /etc/ansible/hosts
[master]
192.168.30.128 hostname=master
[node]
192.168.30.129 hostname=node1
192.168.30.130 hostname=node2
192.168.30.131 hostname=node3
- 创建管理目录：
mkdir -p k8s/roles/{
docker_install,master_install,node_install,addons_install}/{
files,handlers,meta,tasks,templates,vars}
cd k8s/
说明：
files：存放需要同步到异地服务器的源码文件及配置文件；
handlers：当资源发生变化时需要进行的操作，若没有此目录可以不建或为空；
meta：存放说明信息、说明角色依赖等信息，可留空；
tasks：K8S 安装过程中需要进行执行的任务；
templates：用于执行 K8S 安装的模板文件，一般为脚本；
vars：本次安装定义的变量
tree .
.
├── k8s.yml
└── roles
├── addons_install
│ ├── files
│ ├── handlers
│ ├── meta
│ ├── tasks
│ │ ├── calico.yml
│ │ ├── ingress.yml
│ │ └── main.yml
│ ├── templates
│ │ ├── calico-rbac.yaml
│ │ ├── calico.yaml
│ │ └── ingress-nginx.yaml
│ └── vars
│ └── main.yml
├── docker_install
│ ├── files
│ ├── handlers
│ ├── meta
│ ├── tasks
│ │ ├── install.yml
│ │ ├── main.yml
│ │ └── prepare.yml
│ ├── templates
│ │ ├── daemon.json
│ │ ├── install.sh
│ │ ├── kubernetes.conf
│ │ └── kubernetes.repo
│ └── vars
│ └── main.yml
├── master_install
│ ├── files
│ ├── handlers
│ ├── meta
│ ├── tasks
│ │ ├── install.yml
│ │ └── main.yml
│ ├── templates
│ │ └── kubeadm-config.yaml
│ └── vars
│ └── main.yml
└── node_install
├── files
├── handlers
├── meta
├── tasks
│ ├── install.yml
│ └── main.yml
├── templates
└── vars
└── main.yml
29 directories, 23 files
- 创建安装入口文件，用来调用roles：
vim k8s.yml
---
- hosts: all
remote_user: root
gather_facts: True
roles:
- docker_install
- hosts: master
remote_user: root
gather_facts: True
roles:
- master_install
- hosts: node
remote_user: root
gather_facts: True
roles:
- node_install
- hosts: master
remote_user: root
gather_facts: True
roles:
- addons_install
docker部分
- 创建docker入口文件，用来调用docker_install：
vim docker.yml
- hosts: all
remote_user: root
gather_facts: True
roles:
- docker_install
- 创建变量：
vim roles/docker_install/vars/main.yml
SOURCE_DIR: /software
VERSION: 1.18.3
- 创建模板文件：
docker配置daemon.json
vim roles/docker_install/templates/daemon.json
{
"registry-mirrors": ["http://f1361db2.m.daocloud.io"],
"exec-opts":["native.cgroupdriver=systemd"]
}
系统环境kubernetes.conf
vim roles/docker_install/templates/kubernetes.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
repo文件kubernetes.repo
vim roles/docker_install/templates/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
docker-py安装脚本install.sh
vim roles/docker_install/templates/install.sh
#!/bin/bash
loop_exec() {
CMD=$1
while :; do
${CMD}
if [ $? -eq 0 ] ; then
break;
fi
done
}
main() {
loop_exec "yum install -y python python-pip"
loop_exec "pip install --upgrade pip"
loop_exec "pip install docker-py"
}
main
- 环境准备prepare.yml：
vim roles/docker_install/tasks/prepare.yml
- name: 关闭firewalld
service: name=firewalld state=stopped enabled=no
- name: 临时关闭 selinux
shell: "setenforce 0"
failed_when: false
- name: 永久关闭 selinux
lineinfile:
dest: /etc/selinux/config
regexp: "^SELINUX="
line: "SELINUX=disabled"
- name: 添加EPEL仓库
yum: name=epel-release state=latest
- name: 安装常用软件包
yum:
name:
- vim
- lrzsz
- net-tools
- wget
- curl
- bash-completion
- rsync
- gcc
- unzip
- git
- iptables
- conntrack
- ipvsadm
- ipset
- jq
- sysstat
- libseccomp
state: latest
- name: 更新系统
shell: "yum update -y --exclude kubeadm,kubelet,kubectl"
ignore_errors: yes
args:
warn: False
- name: 配置iptables
shell: "iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT"
- name: 关闭swap
shell: "swapoff -a && sed -i '/swap/s/^\\(.*\\)$/#\\1/g' /etc/fstab"
- name: 系统配置
template: src=kubernetes.conf dest=/etc/sysctl.d/kubernetes.conf
- name: 加载br_netfilter
shell: "modprobe br_netfilter"
- name: 生效配置
shell: "sysctl -p /etc/sysctl.d/kubernetes.conf"
- docker安装install.yml：
vim roles/docker_install/tasks/install.yml
- name: 创建software目录
file: name={
{
SOURCE_DIR }} state=directory
- name: 更改hostname
raw: "echo {
{ hostname }} > /etc/hostname"
- name: 更改生效
shell: "hostname {
{ hostname }}"
- name: 设置本地dns
shell: "if [ `grep '{
{ ansible_ssh_host }} {
{ hostname }}' /etc/hosts |wc -l` -eq 0 ]; then echo {
{ ansible_ssh_host }} {
{ hostname }} >> /etc/hosts; fi"
- name: 下载repo文件
shell: "if [ ! -f /etc/yum.repos.d/docker.repo ]; then curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo; fi"
- name: 生成缓存
shell: "yum makecache fast"
args:
warn: False
- name: 安装docker-ce
yum:
name: docker-ce
state: latest
- name: 启动docker并开机启动
service:
name: docker
state: started
enabled: yes
- name: 配置docker
template: src=daemon.json dest=/etc/docker/daemon.json
- name: 重启docker
service:
name: docker
state: restarted
- name: 配置kubernetes源
template: src=kubernetes.repo dest=/etc/yum.repos.d/kubernetes.repo
- name: 安装kubernetes-cni
yum:
name: kubernetes-cni
state: latest
- name: 安装kubeadm、kubelet、kubectl
shell: "yum install -y kubeadm-{
{ VERSION }} kubelet-{
{ VERSION }} kubectl-{
{ VERSION }} --disableexcludes=kubernetes"
args:
warn: False
- name: 启动kubelet并开机启动
service:
name: kubelet
state: started
enabled: yes
- name: 拷贝脚本
template: src=install.sh dest={
{
SOURCE_DIR }} mode=0755
- name: 安装docker-py
script: "{
{ SOURCE_DIR }}/install.sh"
- 引用文件main.yml：
vim roles/docker_install/tasks/main.yml
- include: prepare.yml
- include: install.yml
master部分
- 创建master入口文件，用来调用master_install：
vim master.yml
- hosts: master
remote_user: root
gather_facts: True
roles:
- master_install
- 创建变量：
vim roles/master_install/vars/main.yml
SOURCE_DIR: /software
VERSION: v1.18.3
POD_CIDR: 172.10.0.0/16
MASTER_IP: "{
{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}"
- 创建模板文件：
kubeadm配置文件 kubeadm-config.yaml
vim roles/master_install/templates/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: "{
{ VERSION }}"
controlPlaneEndpoint: "{
{ MASTER_IP }}:6443"
networking:
podSubnet: "{
{ POD_CIDR }}"
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
- 集群初始化install.yml：
vim roles/master_install/tasks/install.yml
- name: 拷贝kubeadm配置文件
template: src=kubeadm-config.yaml dest={
{
SOURCE_DIR }}
- name: 集群初始化准备1
shell: "swapoff -a && kubeadm reset -f"
- name: 集群初始化准备2
shell: "systemctl daemon-reload && systemctl restart kubelet"
- name: 集群初始化准备3
shell: "iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X"
- name: 拉取镜像
shell: "kubeadm config images pull --kubernetes-version={
{ VERSION }} --image-repository=registry.aliyuncs.com/google_containers"
- name: 集群初始化
shell: "kubeadm init --config={
{ SOURCE_DIR }}/kubeadm-config.yaml --upload-certs &>{
{ SOURCE_DIR }}/token"
- name: 获取master的token
shell: "grep -B2 'control-plane --certificate-key' {
{ SOURCE_DIR }}/token > {
{ SOURCE_DIR }}/master.sh"
- name: 获取node的token
shell: "grep -A1 'kubeadm join' {
{ SOURCE_DIR }}/token |tail -2 > {
{ SOURCE_DIR }}/node.sh"
- name: 分发master.sh
shell: "ansible master -m copy -a 'src={
{ SOURCE_DIR }}/master.sh dest={
{ SOURCE_DIR }} mode=0755'"
args:
warn: False
- name: 分发node.sh
shell: "ansible node -m copy -a 'src={
{ SOURCE_DIR }}/node.sh dest={
{ SOURCE_DIR }} mode=0755'"
args:
warn: False
- name: 创建 $HOME/.kube 目录
file: name=$HOME/.kube state=directory
- name: 拷贝KubeConfig
copy: src=/etc/kubernetes/admin.conf dest=$HOME/.kube/config owner=root group=root
- name: kubectl命令补全1
shell: "kubectl completion bash > $HOME/.kube/completion.bash.inc"
- name: kubectl命令补全2
shell: "if [ `grep 'source $HOME/.kube/completion.bash.inc' $HOME/.bash_profile |wc -l` -eq 0 ]; then echo 'source $HOME/.kube/completion.bash.inc' >> $HOME/.bash_profile; fi"
- name: 生效配置
shell: "source $HOME/.bash_profile"
ignore_errors: yes
- 引用文件main.yml：
vim roles/master_install/tasks/main.yml
- include: install.yml
node部分
- 创建node入口文件，用来调用node_install：
vim node.yml
- hosts: node
remote_user: root
gather_facts: True
roles:
- node_install
- 创建变量：
vim roles/node_install/vars/main.yml
SOURCE_DIR: /software
- 添加node到集群install.yml：
vim roles/node_install/tasks/install.yml
- name: 集群初始化准备1
shell: "swapoff -a && kubeadm reset -f"
- name: 集群初始化准备2
shell: "systemctl daemon-reload && systemctl restart kubelet"
- name: 集群初始化准备3
shell: "iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X"
- name: 集群增加node
script: "{
{ SOURCE_DIR }}/node.sh"
- name: 删除node的token
file: name={
{
SOURCE_DIR }}/node.sh state=absent
- 引用文件main.yml：
vim roles/node_install/tasks/main.yml
- include: install.yml
addons部分
- 创建addons入口文件，用来调用addons_install：
vim addons.yml
- hosts: master
remote_user: root
gather_facts: True
roles:
- addons_install
- 创建变量：
vim roles/addons_install/vars/main.yml
SOURCE_DIR: /software
POD_CIDR: 172.10.0.0/16
CALICO_VER: v3.15.1
BACKEND_VER: 1.5
INGRESS_VER: 0.19.0
- 创建模板文件：
calico rbac配置文件 calico-rbac.yaml
vim roles/addons_install/templates/calico-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-kube-controllers
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
rules:
- apiGroups: [""]
resources:
- nodes
verbs:
- watch
- list
- get
- apiGroups: [""]
resources:
- pods
verbs:
- get
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
verbs:
- list
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
verbs:
- get
- list
- create
- update
- delete
- apiGroups: ["crd.projectcalico.org"]
resources:
- hostendpoints
verbs:
- get
- list
- create
- update
- delete
- apiGroups: ["crd.projectcalico.org"]
resources:
- clusterinformations
verbs:
- get
- create
- update
- apiGroups: ["crd.projectcalico.org"]
resources:
- kubecontrollersconfigurations
verbs:
- get
- create
- update
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-kube-controllers
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-kube-controllers
subjects:
- kind: ServiceAccount
name: calico-kube-controllers
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-node
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: calico-node
rules:
- apiGroups: [""]
resources:
- pods
- nodes
- namespaces
verbs:
- get
- apiGroups: [""]
resources:
- endpoints
- services
verbs:
- watch
- list
- get
- apiGroups: [""]
resources:
- configmaps
verbs:
- get
- apiGroups: [""]
resources:
- nodes/status
verbs:
- patch
- update
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- watch
- list
- apiGroups: [""]
resources:
- pods
- namespaces
- serviceaccounts
verbs:
- list
- watch
- apiGroups: [""]
resources:
- pods/status
verbs:
- patch
- apiGroups: ["crd.projectcalico.org"]
resources:
- globalfelixconfigs
- felixconfigurations
- bgppeers
- globalbgpconfigs
- bgpconfigurations
- ippools
- ipamblocks
- globalnetworkpolicies
- globalnetworksets
- networkpolicies
- networksets
- clusterinformations
- hostendpoints
- blockaffinities
verbs:
- get
- list
- watch
- apiGroups: ["crd.projectcalico.org"]
resources:
- ippools
- felixconfigurations
- clusterinformations
verbs:
- create
- update
- apiGroups: [""]
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups: ["crd.projectcalico.org"]
resources:
- bgpconfigurations
- bgppeers
verbs:
- create
- update
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
- ipamblocks
- ipamhandles
verbs:
- get
- list
- create
- update
- delete
- apiGroups: ["crd.projectcalico.org"]
resources:
- ipamconfigs
verbs:
- get
- apiGroups: ["crd.projectcalico.org"]
resources:
- blockaffinities
verbs:
- watch
- apiGroups: ["apps"]
resources:
- daemonsets
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: calico-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-node
subjects:
- kind: ServiceAccount
name: calico-node
namespace: kube-system
calico配置文件 calico.yaml
vim roles/addons_install/templates/calico.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: calico-config
namespace: kube-system
data:
typha_service_name: "none"
calico_backend: "bird"
veth_mtu: "1440"
cni_network_config: |-
{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "calico",
"log_level": "info",
"datastore_type": "kubernetes",
"nodename": "__KUBERNETES_NODE_NAME__",
"mtu": __CNI_MTU__,
"ipam": {
"type": "calico-ipam"
},
"policy": {
"type": "k8s"
},
"kubernetes": {
"kubeconfig": "__KUBECONFIG_FILEPATH__"
}
},
{
"type": "portmap",
"snat": true,
"capabilities": {
"portMappings": true}
},
{
"type": "bandwidth",
"capabilities": {
"bandwidth": true}
}
]
}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: bgpconfigurations.crd.projectcalico.org
spec:
group: crd.projectcalico.org
names:
kind: BGPConfiguration
listKind: BGPConfigurationList
plural: bgpconfigurations
singular: bgpconfiguration
scope: Cluster
versions:
- name: v1
schema:
openAPIV3Schema:
最低0.47元/天 解锁文章
970
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
