Kubernetes服务器集群部署

最新推荐文章于 2024-05-15 17:06:32 发布

问渠科技

最新推荐文章于 2024-05-15 17:06:32 发布

阅读量231

点赞数

分类专栏： kubernetes 文章标签： kubernetes

本文链接：https://blog.csdn.net/chinaxsw/article/details/118080018

版权

kubernetes 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

文章目录

- Kubernetes服务器集群部署

Kubernetes服务器集群部署

1、节点配置

主机名	IP	角色	系统	CPU/内存	磁盘
kubernetes-master	192.168.3.100	Master	Ubuntu Server 20.04	4 核 8G	1T
kubernetes-node-01	192.168.3.101	Node	Ubuntu Server 20.04	4核 8G	1T
kubernetes-node-02	192.168.3.102	Node	Ubuntu Server 20.04	4 核 8G	1T
kubernetes-node-03	192.168.3.103	Node	Ubuntu Server 20.04	4 核 8G	1T
kubernetes-volumes	192.168.3.110	Volumes	Ubuntu Server 20.04	4核 8G	1T
kubernetes-volumes-mount	192.168.3.111	Volumes	Ubuntu Server 20.04	4 核 8G	1T

2、环境配置

root权限

sudo passwd root
# 设置root远程登录
vi /etc/ssh/sshd_config
# 将此行PermitRootLogin注释去掉，并设置为yes

单独节点配置

注意： 为 Master 和 Node 节点单独配置对应的 IP 和 主机名

配置 IP

编辑 vi /etc/netplan/00-installer-config.yaml 配置文件，修改内容如下

# vi /etc/netplan/00-installer-config.yaml
network:
    ethernets:
        ens33:
          addresses: [192.168.3.100/24]
          gateway4: 192.168.3.1
          nameservers:
            addresses: [114.114.114.114，8.8.8.8]
    version: 2

使用 netplan apply 命令让配置生效

配置主机名

# 修改主机名
hostnamectl set-hostname kubernetes-master
# 配置 hosts
vi /etc/hosts
#添加
192.168.3.100 kubernetes-master

配置DNS

# 取消 DNS 行注释，并增加 DNS 配置如：114.114.114.114，修改后重启下计算机
vi /etc/systemd/resolved.conf

DNS= 114.114.114.114 8.8.8.8

关闭防火墙

ufw disable

关闭交换空间

# 临时
swapoff -a  
# 永久
sed -ri 's/.*swap.*/#&/' /etc/fstab

设置主机名

# master主机修改192.168.3.100
hostnamectl set-hostname kubernetes-master
# node-01主机修改192.168.3.101
hostnamectl set-hostname kubernetes-node-01
# node-02主机修改192.168.3.102
hostnamectl set-hostname kubernetes-node-02
# node-03主机修改192.168.3.103
hostnamectl set-hostname kubernetes-node-03

修改 cloud.cfg

主要作用是防止重启后主机名还原

vi /etc/cloud/cloud.cfg
# 该配置默认为 false，修改为 true 即可
preserve_hostname: true

在Master主机中修改hosts

vi /etc/hosts
# 添加
192.168.3.100 kubernetes-master
192.168.3.101 kubernetes-node-01
192.168.3.102 kubernetes-node-02
192.168.3.103 kubernetes-node-03

安装Docker

# 卸载旧版本
sudo apt remove docker docker-engine docker.io containerd runc
sudo apt-get autoremove docker-ce-*
sudo rm -rf /etc/systemd/system/docker.service.d
sudo rm -rf /var/lib/docker

# 更新软件源
sudo apt-get update
# 安装所需依赖
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
# 安装 GPG 证书
curl -fsSL http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
# 新增软件源信息
sudo add-apt-repository "deb [arch=amd64] http://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
# 再次更新软件源
sudo apt-get -y update

# 查看Docker CE 版本
apt-cache madison docker-ce

# 指定版本安装 Docker CE 版
sudo apt-get install docker-ce=5:19.03.15~3-0~ubuntu-focal

# 开启 Docker Service
systemctl enable docker.service

配置阿里云Docker加速器

注意： 国内镜像加速器可能会很卡，请替换成你自己阿里云镜像加速器，地址如：https://yourself.mirror.aliyuncs.com，在阿里云控制台的 容器镜像服务 -> 镜像加速器 菜单中可以找到

在 /etc/docker/daemon.json 中写入如下内容（以下配置修改 cgroup 驱动为 systemd，满足 K8S 建议）

sudo mkdir -p /etc/docker

vi /etc/docker/daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "registry-mirrors": [
    "https://iqgum0ag.mirror.aliyuncs.com",
    "https://dockerhub.azk8s.cn",
    "https://registry.docker-cn.com"
  ],
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "data-root": "/data1/docker"
}


sudo systemctl daemon-reload
sudo systemctl restart docker

安装docker-compose

# 安装其他版本请更换1.29.0
curl -L https://github.com/docker/compose/releases/download/1.29.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
# 如果慢，换个源
curl -L https://get.daocloud.io/docker/compose/releases/download/1.29.0/docker-compose`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

安装三个 Kubernetes 必备工具，分别为 kubeadm，kubelet，kubectl

# 安装系统工具
apt-get update && apt-get install -y apt-transport-https
# 安装 GPG 证书
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
# 写入软件源；注意：我们用系统代号为 bionic，但目前阿里云不支持，所以沿用 16.04 的 xenial
vi /etc/apt/sources.list.d/kubernetes.list

deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main

# 跟新软件源
sudo apt update

# 指定版本安装
apt-get install kubeadm=1.20.0-00 kubectl=1.20.0-00 kubelet=1.20.0-00

同步时间

设置时区

dpkg-reconfigure tzdata

选择 Asia（亚洲）
选择 Shanghai（上海）
时间同步

# 安装 ntpdate
apt-get install ntpdate
# 设置系统时间与网络时间同步（cn.pool.ntp.org 位于中国的公共 NTP 服务器）
ntpdate cn.pool.ntp.org
# 将系统时间写入硬件时间
hwclock --systohc

确认时间

date
# 输出如下（自行对照与系统时间是否一致）
Sun Jun  2 22:02:35 CST 2020

3、主节点配置

kubernetes-master(192.168.3.100)

# 导出配置文件
cd /etc/kubernetes
kubeadm config print init-defaults --kubeconfig ClusterConfiguration > kubeadm.yml
# 修改配置文件  

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  # 修改为主节点 IP
  advertiseAddress: 192.168.3.100
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  # 修改主节点名称
  name: kubernetes-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
# 国内不能访问 Google，修改为阿里云
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
# 修改版本号 与kubeadm版本相同
kubernetesVersion: v1.20.0
networking:
  dnsDomain: cluster.local
  # 配置 POD 所在网段为我们虚拟机不重叠的网段（这里用的是 Flannel 默认网段）
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}


# 拉取镜像
# 我遇到coredns拉取不下来的
#用 docker pull coredns/coredns:1.8.0 拉取
#再更改标签 docker tag coredns/coredns:1.8.0 registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0
kubeadm config images pull --config kubeadm.yml


# 安装主节点
kubeadm init --config=kubeadm.yml --upload-certs | tee kubeadm-init.log

# 成功后显示从节点添加命令
kubeadm join 192.168.3.100:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:6878af7b312e1cdaec119c3a0ee86cdef625b7063905c931f7b943c7ca113954

安装失败重新安装需要执行

kubeadm reset

配置用户运行kubectl

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# 非 ROOT 用户执行
chown $(id -u):$(id -g) $HOME/.kube/config

验证是否成功

kubectl get node
# 输出如下
NAME                STATUS     ROLES    AGE     VERSION
kubernetes-master   NotReady   master   4m38s   v1.15.0

4、安装从节点

将 Node 节点加入到集群中很简单，只需要在 Node 服务器上安装 kubeadm，kubectl，kubelet 三个工具，然后使用 kubeadm join 命令加入即可

kubeadm join 192.168.3.100:6443 --token abcdef.0123456789abcdef \	--discovery-token-ca-cert-hash sha256:2c478d704de9b7e570a068c4b8f854b98dc177a3156570095b13cf9edf59c326

默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，操作如下：（主节点）

kubeadm token create --print-join-command

验证是否成功(主节点)

kubectl get node

安装Kubernetes 中的 CNI 插件（主节点）

Kubernetes 中可选的 CNI 插件如下：

Flannel(我选择Flannel)
Calico
Canal
Weave

下载 Flannel配置文件并修改

mkdir flannel
cd flannel
# 下载配置文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# 部署
kubectl apply -f kube-flannel.yml

验证是否成功(主节点)

kubectl get node
# STATUS都是Ready

5、Ingress nginx配置

配置

# 下载 Nginx Ingress Controller 配置文件
cd /etc/kubernetes
mkdir ingress
cd ingress
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml

修改配置文件

# 将authorization.k8s.io/v1beta1 修改成authorization.k8s.io/v1
//以上省略
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      # 增加 hostNetwork: true，意思是开启主机网络模式，暴露 Nginx 服务端口 80
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
// 以下代码省略...

通过命令 kubectl apply -f mandatory.yaml 部署

通过命令 kubectl get pods -n ingress-nginx -o wide 查看

创建一个名为 ingress.yml 的资源配置文件

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-web
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: k8s.wenqu.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-service
          servicePort: 80

nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.9
        ports:
        - containerPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-service 
  labels:
    app: nginx
spec:
  # ClusterIP, NodePort, LoadBalancer
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

部署nginx

kubectl apply -f nginx.yml

# 查看部署
kubectl get deployment
# 查看服务
kubectl get service
# 查看ingress-nginx
# 注意下面的 IP 地址，就是我们实际访问地址
kubectl get pods -n ingress-nginx -o wide
# 查看ingress
kubectl get ingress

测试

# 不设置 Hosts 的方式请求地址，下面的 IP 和 HOST 均在上面有配置
curl -v http://192.168.3.102 -H 'host: k8s.wenqu.com'

修改本机hosts

# 添加
192.168.3.102   k8s.wenqu.com

用浏览器打开

http://k8s.wenqu.com

生产发布需要连接上层nginx

按照我们正常使用，nginx-ingress所在服务器集群式不开通外网，但是为了方便使用，我们通过externalIPs将服务使用固定IP的方式对内网开放，其他服务/应用可以通过http//externalIPs/xxx的方式即可访问nginx-ingress.

修改 ingress.yml 的资源配置文件

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-web
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /api
        backend:
          serviceName: nginx-service
          servicePort: 80

修改nginx.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.19.9
        ports:
        - containerPort: 80

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-service 
  labels:
    app: nginx
spec:
  # ClusterIP, NodePort, LoadBalancer
  type: ClusterIP
  externalIPs:
  - 192.168.3.100
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

用浏览器打开

http://192.168.3.100/api

6、资源NFS服务部署

主机名	IP	角色	系统	CPU/内存	磁盘
kubernetes-volumes	192.168.3.110	Volumes	Ubuntu Server 20.04	2 核 8G	1T
kubernetes-volumes-mount	192.168.3.111	Volumes	Ubuntu Server 20.04	2 核 8G	1T

服务端kubernetes-volumes

# 创建一个目录作为共享文件目录
mkdir -p /usr/local/kubernetes/volumes
# 给目录增加读写权限
chmod a+rw /usr/local/kubernetes/volumes
# 安装 NFS 服务端
apt-get update
apt-get install -y nfs-kernel-server

配置 NFS 服务目录，打开文件

vi /etc/exports
# 在尾部新增一行，内容如下
/usr/local/kubernetes/volumes *(rw,sync,no_subtree_check,no_root_squash)
# 重启服务，使配置生效
/etc/init.d/nfs-kernel-server restart

客户端kubernetes-volumes-mount

apt-get update
apt-get install -y nfs-common
# 创建 NFS 客户端挂载目录
mkdir -p /usr/local/kubernetes/volumes-mount
chmod a+rw /usr/local/kubernetes/volumes-mount
# 将 NFS 服务器的 `/usr/local/kubernetes/volumes` 目录挂载到 NFS 客户端的 `/usr/local/kubernetes/volumes-mount` 目录
mount 192.168.3.110:/usr/local/kubernetes/volumes /usr/local/kubernetes/volumes-mount

取消 NFS 客户端挂载

注意： 不要直接在挂载目录下执行，否则会报错

umount /usr/local/kubernetes/volumes-mount

7、使用 Kuboard 图形化管理kubernetes

# 安装 master上
kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
kubectl apply -f https://addons.kuboard.cn/metrics-server/0.3.7/metrics-server.yaml


# 查看是否启动
kubectl get pods -l k8s.kuboard.cn/name=kuboard -n kube-system

然后访问您集群中任意节点的 32567 端口（http://any-of-your-node-ip:32567），即可打开 Kuboard 界面，比如我的 Node 节点 IP 为：http://192.168.3.100:32567

建议使用谷歌浏览器

获取 Token

echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d)

使用此token即可登录kuboard

8、安装helm

# master 主节点
cd /usr/local
mkdir kubernetes
cd kubernetes
mkdir helm
cd helm
# 下载
wget https://get.helm.sh/helm-v3.5.4-linux-amd64.tar.gz
# 解压
tar -zxvf helm-v3.5.4-linux-amd64.tar.gz
#复制客户端执行文件到bin目录下
cp linux-amd64/helm /usr/local/bin
# 添加公用的仓库
helm repo add stable http://mirror.azure.cn/kubernetes/charts
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts 
helm repo update
# 指定对应的k8s集群
vi /etc/profile
# 末尾添加
export KUBECONFIG=/root/.kube/config
# 测试
helm version

9、部署Tidb

部署本地卷

所有node节点执行

# 查看硬盘挂载
fdisk -l
# 如果是单块硬盘，需要分区
# 分区工具
apt-get install gparted
gparted
# 格式化 挂载磁盘
mkfs.ext4 /dev/sdb 
DISK_UUID=$(blkid -s UUID -o value /dev/sdb)
mkdir /mnt/$DISK_UUID 
mount -t ext4 /dev/sdb /mnt/$DISK_UUID
# 持久化 mnt
echo UUID=`sudo blkid -s UUID -o value /dev/sdb` /mnt/$DISK_UUID ext4 defaults 0 2 | sudo tee -a /etc/fstab
# 创建多个目录并 `mount` 到 `discovery directory`
for i in $(seq 1 10); do
  sudo mkdir -p /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}
  sudo mount --bind /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i}
done

# `/etc/fstab` 自动挂载
for i in $(seq 1 10); do
  echo /mnt/${DISK_UUID}/vol${i} /mnt/disks/${DISK_UUID}_vol${i} none bind 0 0 | sudo tee -a /etc/fstab
done

取消挂载

# 取消挂载
#  删除全部 PV
kubectl delete pv --all

# 卸载挂载目录

DISK_UUID=$(blkid -s UUID -o value /dev/sdb)
for i in $(seq 1 10); do  
sudo umount /mnt/disks/${DISK_UUID}_vol${i}donerm -fr /mnt
done
# 删除 `/etc/fstab` 配置中挂载的目录
UUID=062815c7-b202-41ef-a5fb-77c783792737 / ext4 defaults 0 0
UUID=e8717c59-6d9b-4709-9303-b2161a57912b /boot ext4 defaults 0 0
#/swap.img      none    swap    sw      0       0


# 卸载挂载目录后需要删除如下内容
UUID=58759186-ffab-42a3-96ce-f9d3c355d4d1 /mnt/58759186-ffab-42a3-96ce-f9d3c355d4d1 ext4 defaults 0 2
/mnt/58759186-ffab-42a3-96ce-f9d3c355d4d1/vol1 /mnt/disks/58759186-ffab-42a3-96ce-f9d3c355d4d1_vol1 none bind 0 0
/mnt/58759186-ffab-42a3-96ce-f9d3c355d4d1/vol2 /mnt/disks/58759186-ffab-42a3-96ce-f9d3c355d4d1_vol2 none bind 0 0

安装tidb

# 克隆tidb-oprator到本地
cd /usr/local/kubernetes
mkdir tidb
cd tidb
git clone https://github.com/pingcap/tidb-operator.git
cd tidb-operator

# 部署本地卷local-volume-provisioner
kubectl apply -f manifests/local-dind/local-volume-provisioner.yaml
#查看POD和PV
kubectl get po -n kube-system -l app=local-volume-provisioner
kubectl get pv | grep local-storage

# 安装 TiDB Operator CRDs
kubectl apply -f manifests/crd.yaml

# Add the PingCAP repository
helm repo add pingcap https://charts.pingcap.org/
helm search repo tidb-cluster
helm search repo tidb-operator
# Create a namespace for TiDB Operator
kubectl create namespace tidb-admin
# 安装 TiDB Operator


helm install tidb-operator charts/tidb-operator --namespace=tidb-admin --version=v1.1.12 \
    --set tidbBackupManagerImage=registry.cn-beijing.aliyuncs.com/tidb/tidb-backup-manager:v1.1.12 \
    --set scheduler.kubeSchedulerImageName=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler

# 卸载
helm delete tidb-operator charts/tidb-operator --namespace=tidb-admin

# 验证
kubectl get pods --namespace tidb-admin -l app.kubernetes.io/instance=tidb-operator

安装 TiDB Cluster

# Deploy a TiDB cluster
kubectl create namespace tidb-cluster 

# 编辑
vi /usr/local/kubernetes/tidb/tidb-operator/charts/tidb-cluster/Chart.yaml


apiVersion: v1
description: A Helm chart for TiDB Cluster
name: tidb-cluster
# 将version改为1.0.0
version: 1.0.0
home: https://github.com/pingcap/tidb-operator
sources:
  - https://github.com/pingcap/tidb-operator
keywords:
  - newsql
  - htap
  - database
  - mysql
  - raft



helm install tidb-cluster charts/tidb-cluster --namespace=tidb-cluster --version=v1.1.12


#安装结束出现
NOTES:
Cluster Startup
1. Watch tidb-cluster up and running
     watch kubectl get pods --namespace tidb-cluster -l app.kubernetes.io/instance=tidb-cluster -o wide
2. List services in the tidb-cluster
     kubectl get services --namespace tidb-cluster -l app.kubernetes.io/instance=tidb-cluster

Cluster access
* Access tidb-cluster using the MySQL client
    kubectl port-forward -n tidb-cluster svc/tidb-cluster-tidb 4000:4000 &
    mysql -h 127.0.0.1 -P 4000 -u root -D test
  Set a password for your user
    SET PASSWORD FOR 'root'@'%' = '9rv4BiIqRd'; FLUSH PRIVILEGES;
* View monitor dashboard for TiDB cluster
   kubectl port-forward -n tidb-cluster svc/tidb-cluster-grafana 3000:3000
   Open browser at http://localhost:3000. The default username and password is admin/admin.
   If you are running this from a remote machine, you must specify the server's external IP address.



# 临时暴露端口
kubectl port-forward -n tidb-cluster svc/tidb-cluster-tidb 4000:4000 --address 0.0.0.0
# 临时暴露监控端口
kubectl port-forward -n tidb-cluster svc/tidb-cluster-grafana 3000:3000 --address 0.0.0.0

# 验证
watch kubectl get po -n tidb-cluster

销毁 TiDB 集群

完成测试后，您可能希望销毁 TiDB 集群。

销毁 Kubernetes 集群的方法取决于其创建方式，您可参考前面 Kubernetes 创建文档说明。以下是销毁 TiDB 集群的步骤，并不会影响 Kubernetes 集群本身。

删除 TiDB Cluster

kubectl delete tc tidb-cluster -n tidb-cluster

此命令中，tc 为 tidbclusters 的简称。

删除 TiDB Monitor

kubectl delete tidbmonitor tidb-cluster -n tidb-cluster

删除 PV 数据

如果您的部署使用持久性数据存储，则删除 TiDB 集群将不会删除集群的数据。如果不再需要数据，可以运行以下命令来清理数据：

kubectl delete pvc -n tidb-cluster -l app.kubernetes.io/instance=tidb-cluster,app.kubernetes.io/managed-by=tidb-operator && \kubectl get pv -l app.kubernetes.io/namespace=tidb-cluster,app.kubernetes.io/managed-by=tidb-operator,app.kubernetes.io/instance=tidb-cluster -o name | xargs -I {} kubectl patch {} -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'

删除命名空间

为确保没有残余资源，您可以删除用于 TiDB 集群的命名空间。

kubectl delete namespace tidb-cluster

停止 kubectl 的端口转发

如果您仍在运行正在转发端口的 kubectl 进程，请终止它们：

pgrep -lfa kubectl

10、部署Nacos

准备Nacos需要使用的NFS

注意：kubernetes1.20版本 nfs-provisioner报错问题:“selfLink was empty” PVC状态一直为pending状态

selfLink在1.16版本以后已经弃用，在1.20版本停用。

修改/etc/kubernetes/manifests/kube-apiserver.yaml文件，找到如下内容后，在最后添加一项参数

# vi /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.3.100
    - --.......　　#省略多行内容
    - --feature-gates=RemoveSelfLink=false　　#添加此行

所有Node节点上安装nfs-common

apt-get install nfs-common

在kubernetes-volumes 192.168.3.110 上执行

# 创建 Nacos 所需共享目录

mkdir -p /data/nfs-share
mkdir -p /data/mysql

# 给目录增加读写权限 (因为volumes目录已经添加权限，可以省略)
chmod a+rw /data/nfs-share
chmod a+rw /data/mysql

managed-nfs-storage
# 配置 NFS 服务目录 (因为volumes目录已经添加，可以省略)
vi /etc/exports
# 底部增加
/usr/local/kubernetes/volumes *(rw,sync,no_subtree_check,no_root_squash)
/data/nfs-share *(rw,sync,no_subtree_check,no_root_squash)
/data/mysql *(rw,sync,no_subtree_check,no_root_squash)

# 重启服务，使配置生效
/etc/init.d/nfs-kernel-server restart

回到kubernetes-master主节点

cd /usr/local/kubernetes

# 克隆nacos-k8s
git clone https://github.com/nacos-group/nacos-k8s.git
cd nacos-k8s

# 创建角色
kubectl create -f deploy/nfs/rbac.yaml

# 修改deployment.yaml
vi deploy/nfs/deployment.yaml

# 进行如下修改
...
          # 修改为 NFS 服务端 IP
          value: 192.168.3.110
        - name: NFS_PATH
          # 修改为 NFS 服务端共享目录
          value: /data/nfs-share
      volumes:
      - name: nfs-client-root
        nfs:
          # 修改为 NFS 服务端 IP
          server: 192.168.3.110
          # 修改为 NFS 服务端共享目录
          path: /data/nfs-share
 ...         
          
# 执行deployment.yaml           
kubectl create -f deploy/nfs/deployment.yaml   

# 创建 `NFS StorageClass`
kubectl create -f deploy/nfs/class.yaml

# 验证 NFS 部署成功
kubectl get pod -l app=nfs-client-provisioner
# 输出如下
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-5f9f96c4dc-mrzr6   1/1     Running   0          177m

# 部署数据库
vi deploy/mysql/mysql-nfs.yaml

# 修改 NFS 服务端 IP
...
volumes:
      - name: mysql-data
        nfs:
          server: 192.168.3.110
          path: /data/mysql
...

# 开始部署数据库
kubectl create -f deploy/mysql/mysql-nfs.yaml

kubectl get pod
# 显示
NAME                                     READY   STATUS    RESTARTS   AGE
mysql-2nwdt                              1/1     Running   9          113m


# 部署Nacos
vi deploy/nacos/nacos-pvc-nfs.yaml

kubectl create -f deploy/nacos/nacos-pvc-nfs.yaml

暴露地址ingress

Master

/usr/local/kubernetes

mkdir ingress
cd ingress
vi nacos.yml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: nacos-web
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: nacos.wenqu.com
    http:
      paths:
      - path:
        backend:
          serviceName: nacos-headless
          servicePort: 8848
          
          
          
kubectl apply -f nacos.yml

查看ingress
kubectl get pods -n ingress-nginx -o wide

验证

# 查看ingress部署的node
kubectl get pods -n ingress-nginx -o wide

NAME                                        READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
nginx-ingress-controller-7d4544b644-hznrt   1/1     Running   0          14m   192.168.3.101   kubernetes-node-01   <none>           <none>


# 修改本机hosts
192.168.3.101   nacos.wenqu.com

# 浏览器打开
http://nacos.wenqu.com/nacos/

11、部署Redis

添加helm仓库

helm repo add stable http://mirror.azure.cn/kubernetes/charts/
helm repo update

拉取redis

cd /usr/local/kubernetes/helm
# 拉取
helm pull stable/redis-ha
# 解压
tar zxf redis-ha-4.4.6.tgz

cd redis-ha

修改values.yaml

修改 “hardAntiAffinity: true” 为 “hardAntiAffinity: false” (仅限当replicas > worker node 节点数时修改)
修改 “auth: false” 为 “auth: true”，打开 “# redisPassword:” 的注释并设置密码
打开 “ # storageClass: “-” ” 的注释，并修改 “-” 为 集群中的自动供给卷 “managed-nfs-storage”， 配置中 “size: 10Gi” 的大小为默认设置，可根据需要进行调整

部署redis

# 建立namespace
kubectl create namespace redis
# 部署
helm install redis   -f values.yaml --namespace=redis .
# 查看 等待Running
kubectl get pod -n redis

临时访问

在实际生产环境中我们不会将 Redis 端口暴露到公网，故咱们依然采取 kubectl port-forward 的方式暴露服务到主机

# 临时暴露端口
kubectl port-forward svc/redis-redis-ha 6379:6379 --address 0.0.0.0 -n redis

使用 RedisDesktopManager 等 Redis 客户端工具访问即可

12、部署RocketMQ

克隆rocketmq-operator

# 克隆
git clone https://github.com/apache/rocketmq-operator.git
# 部署
cd rocketmq-operator
./install-operator.sh
# 检验
kubectl get pods

部署数据卷

kubernetes-volumes 192.168.3.110

mkdir -p /data/rocketmq
chmod a+rw /data/rocketmq

vi /etc/exports
# 底部增加
/data/rocketmq *(rw,sync,no_subtree_check,no_root_squash)
# 重启服务，使配置生效
/etc/init.d/nfs-kernel-server restart

kubernetes-master 192.168.3.100

修改配置文件nfs-client.yaml

vi deploy/storage/nfs-client.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: rocketmq/nfs
            - name: NFS_SERVER
              # 修改你自己的 NFS 服务器地址
              value: 192.168.3.110
            - name: NFS_PATH
              # 修改你自己的 NFS 共享目录
              value: /data/rocketmq
      volumes:
        - name: nfs-client-root
          nfs:
            # 修改你自己的 NFS 服务器地址
            server: 192.168.3.110
            # 修改你自己的 NFS 共享目录
            path: /data/rocketmq

创建 StorageClass

cd deploy/storage/
./deploy-storage-class.sh

# 检验
kubectl get pods

部署Cluster

部署NameServer

修改配置文件 rocketmq_v1alpha1_nameservice_cr.yaml

vi example/rocketmq_v1alpha1_nameservice_cr.yaml


apiVersion: rocketmq.apache.org/v1alpha1
kind: NameService
metadata:
  name: name-service
spec:
  # size is the the name service instance number of the name service cluster
  size: 1
  # nameServiceImage is the customized docker image repo of the RocketMQ name service
  nameServiceImage: apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # hostNetwork can be true or false
  hostNetwork: true
  #  Set DNS policy for the pod.
  #  Defaults to "ClusterFirst".
  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
  #  explicitly to 'ClusterFirstWithHostNet'.
  dnsPolicy: ClusterFirstWithHostNet
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1024Mi"
      cpu: "500m"
  # storageMode can be EmptyDir, HostPath, StorageClass
  # 修改成StorageClass
  storageMode: StorageClass
  hostPath: /data/rocketmq/nameserver
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: namesrv-storage
        annotations:
          volume.beta.kubernetes.io/storage-class: rocketmq-storage
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi

部署rocketmq_v1alpha1_nameservice_cr.yaml

kubectl apply -f example/rocketmq_v1alpha1_nameservice_cr.yaml
# 输出如下
nameservice.rocketmq.apache.org/name-service created
#检验
kubectl get pods -owide

部署Broker

vi example/rocketmq_v1alpha1_broker_cr.yaml



apiVersion: v1
kind: ConfigMap
metadata:
  name: broker-config
data:
  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
  broker-common.conf: |
    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
    deleteWhen=04
    fileReservedTime=48
    flushDiskType=ASYNC_FLUSH
    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
    brokerRole=ASYNC_MASTER

---
apiVersion: rocketmq.apache.org/v1alpha1
kind: Broker
metadata:
  # name of broker cluster
  name: broker
spec:
  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
  size: 1
  # nameServers is the [ip:port] list of name service
  # 修改为你部署的 name-service-x 的地址
  nameServers: "192.168.3.102:9876"
  # replicaPerGroup is the number of each broker cluster
  replicaPerGroup: 1
  # brokerImage is the customized docker image repo of the RocketMQ broker
  brokerImage: apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0
  # imagePullPolicy is the image pull policy
  imagePullPolicy: Always
  # resources describes the compute resource requirements and limits
  resources:
    requests:
      memory: "2048Mi"
      cpu: "250m"
    limits:
      memory: "12288Mi"
      cpu: "500m"
  # allowRestart defines whether allow pod restart
  allowRestart: true
  # storageMode can be EmptyDir, HostPath, StorageClass
  # 默认为 HostPath 修改为 StorageClass
  storageMode: StorageClass
  # hostPath is the local path to store data
  hostPath: /data/rocketmq/broker
  # scalePodName is [Broker name]-[broker group number]-master-0
  scalePodName: broker-0-master-0
  # env defines custom env, e.g. BROKER_MEM
  env:
    - name: BROKER_MEM
      valueFrom:
        configMapKeyRef:
          name: broker-config
          key: BROKER_MEM
  # volumes defines the broker.conf
  volumes:
    - name: broker-config
      configMap:
        name: broker-config
        items:
          - key: broker-common.conf
            path: broker-common.conf
  # volumeClaimTemplates defines the storageClass
  volumeClaimTemplates:
    - metadata:
        name: broker-storage
        annotations:
          volume.beta.kubernetes.io/storage-class: rocketmq-storage
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 8Gi

部署rocketmq_v1alpha1_broker_cr.yaml

kubectl apply -f example/rocketmq_v1alpha1_broker_cr.yaml
# 输出如下
broker.rocketmq.apache.org/broker created
# 检验
kubectl get pods -owide

部署控制台

创建rocketmq-console.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: rocketmq-console
  labels:
    app: rocketmq-console
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rocketmq-console
  template:
    metadata:
      labels:
        app: rocketmq-console
    spec:
      containers:
      - name: rocketmq-console
        image: styletang/rocketmq-console-ng
        env:
        - name: JAVA_OPTS
          # 修改为你自己的 NameServer 地址
          value: -Drocketmq.namesrv.addr=192.168.3.102:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false
---
apiVersion: v1
kind: Service
metadata:
  name: rocketmq-service
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 32568
  selector:
    app: rocketmq-console

部署rocketmq-console.yaml

kubectl apply -f rocketmq-console.yaml

访问您集群中任意节点的 32568 端口

http://192.168.3.100:32568

删除

# 删除 Console
kubectl delete -f rocketmq-console.yaml
# 删除 Broker
kubectl delete -f example/rocketmq_v1alpha1_broker_cr.yaml
# 删除 NameServer
kubectl delete -f example/rocketmq_v1alpha1_nameservice_cr.yaml
# 删除 Operator
./purge-operator.sh
# 删除 Storage
cd deploy/storage./remove-storage-class.sh

13、部署SpringBoot应用

# 该镜像需要依赖的基础镜像
FROM mdsol/java11-jre
# 将targer目录下的jar包复制到docker容器/usr/local/springboot/目录下面目录下面
ADD k8s-app-0.0.1-SNAPSHOT.jar /usr/local/springboot/app.jar
# 声明服务运行在9000端口
EXPOSE 9000
# 执行命令
CMD ["java","-jar","/usr/local/springboot/app.jar"]
# 指定维护者名称
MAINTAINER wenqu china.xsw@163.com

# 制作镜像
docker build -t k8s-springboot-master .

# 查看镜像
docker images

# 注册阿里云，后台创建个人镜像仓库
# 登录阿里镜像仓库
sudo docker login --username=china.xsw@163.com registry.cn-hangzhou.aliyuncs.com

# 添加版本号  要用到镜像ID
# 命令: sudo docker tag [ImageId] [仓库地址]/[命名空间名字]/[仓库名字]:[镜像版本号]
sudo docker tag d3f33402dd59 registry.cn-hangzhou.aliyuncs.com/wenqu/wenqu:1.0

# 上传镜像
# 命令:sudo docker push [仓库地址]/[命名空间名字]/[仓库名字]:[镜像版本号]
sudo docker push registry.cn-hangzhou.aliyuncs.com/wenqu/wenqu:1.0

# 拉取镜像
# 命令: sudo docker pull [仓库地址]/[命名空间名字]/[仓库名字]:[镜像版本号]
sudo docker pull registry.cn-hangzhou.aliyuncs.com/wenqu/wenqu:1.0


#deployment部署镜像创建出pod
# 命令: kubectl create deployment [deployment名字]–image=[镜像地址] --dry-run -o yaml >[yaml文件名].yaml
kubectl create  deployment wenqu --image=registry.cn-hangzhou.aliyuncs.com/wenqu/wenqu:1.0 --dry-run -o yaml >wenqu.yaml

# 部署镜像
kubectl apply -f wenqu.yaml

# NodePort 方式测试
kubectl  expose deployment wenqu --port=9000  --target-port=9000 --type=NodePort

#查验
kubectl  get svc

# 访问
http://192.168.3.100/k8s/hello

问渠科技

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
打赏
0
评论
Kubernetes服务器集群部署

文章目录Kubernetes服务器集群部署1、节点配置2、环境配置3、主节点配置4、安装从节点5、Ingress nginx配置6、资源NFS服务部署7、使用 Kuboard 图形化管理kubernetes8、安装helm9、部署Tidb10、部署Nacos11、部署Redis12、部署RocketMQ13、部署SpringBoot应用Kubernetes服务器集群部署1、节点配置主机名IP角色系统CPU/内存磁盘kubernetes-master192.168.3.10
复制链接

扫一扫