k3s安装过程

月夜楓

已于 2022-09-01 18:25:43 修改

阅读量1.6k

点赞数

分类专栏： kubernetes 文章标签： linux 服务器运维

于 2022-07-14 15:47:33 首次发布

本文链接：https://blog.csdn.net/cyxinda/article/details/125780762

版权

kubernetes 专栏收录该内容

16 篇文章 1 订阅

订阅专栏

参考：https://blog.csdn.net/qq_42889280/article/details/123994851

环境
| | |
|–|–|
| | |
安装containerd

1.安装依赖
 
yum install -y yum-utils device-mapper-persistent-data lvm2
 
2.添加yum源
 
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
 
3.安装containerd
 
yum install containerd -y
 
4.生成配置文件
 
containerd config default > /etc/containerd/config.toml
 
5.替换 containerd 默认的 sand_box 镜像，编辑 /etc/containerd/config.toml
 
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2"
 
6.启动服务
 
systemctl restart containerd && systemctl enable containerd
————————————————
版权声明：本文为CSDN博主「筑梦之路」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/qq_34777982/article/details/124906096

关闭防火墙

systemctl stop firewalld && systemctl disable firewalld

为两个机器命名
hostnamectl set-hostname k3s-master
hostnamectl set-hostname k3s-node1
并修改hosts

[root@k3s-master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.70.10.194 k3s-master
172.60.10.82 node1

安装k3s
准备工作，在两台机器上面分别执行：
使用离线安装，官方叫做Air-Gap Install，参考官网：https://rancher.com/docs/k3s/latest/en/installation/airgap/
下载k3s安装程序：https://github.com/k3s-io/k3s/releases
1. k3s安装文件
2. k3s所需镜像
3. curl -sfL https://get.k3s.io | tee install.sh

如果你使用的容器运行时为containerd，在启动 K3s 时，它会检查/var/lib/rancher/k3s/agent/images/是否存在可用的镜像压缩包，如果存在，就将该镜像导入到containerd 镜像列表中。所以我们只需要下载 K3s 依赖的镜像到/var/lib/rancher/k3s/agent/images/目录，然后启动 K3s 即可

sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp ./k3s-airgap-images-$ARCH.tar /var/lib/rancher/k3s/agent/images/

启动server节点：

[root@k3s-master ~]# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --node-external-ip 172.70.10.194 --advertise-address 172.70.10.194 --node-ip 172.70.10.194 --flannel-iface eth0 --write-kubeconfig /root/.kube/config
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
[root@k3s-master ~]# systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since 四 2022-07-14 11:04:53 CST; 3min 45s ago
     Docs: https://k3s.io
  Process: 23223 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 23221 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 23218 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 23225 (k3s-server)
    Tasks: 151
   Memory: 867.9M
   CGroup: /system.slice/k3s.service
           ├─ 4229 /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 66ccf8f6aa37199f8794c021af7...
           ├─ 4367 /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7ef8b11af4bddeeaf236ea3257c...
           ├─ 4455 /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f/bin/containerd-shim-runc-v2 -namespace k8s.io -id bf64c964fe91a1033e33f728d94...
           ├─ 5820 /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 57a505a90c6dc69fdd51ded0b0f...
           ├─ 5902 /var/lib/rancher/k3s/data/f4431393712ffbc38bd97607e1f813faef14e378c15e53ea07e11c6bbe9a634f/bin/containerd-shim-runc-v2 -namespace k8s.io -id ee1a5d1cb60a1441d077da8945e...
           ├─23225 /usr/local/bin/k3s server
           └─23260 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent...

	7月 14 11:05:20 k3s-master k3s[23225]: I0714 11:05:20.087822   23225 reconciler.go:270] operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbsc\" (Un...
	7月 14 11:05:20 k3s-master k3s[23225]: I0714 11:05:20.087858   23225 reconciler.go:157] "Reconciler: start to sync state"
	7月 14 11:05:21 k3s-master k3s[23225]: I0714 11:05:21.817357   23225 controller.go:611] quota admission added evaluator for: endpoints
	7月 14 11:05:21 k3s-master k3s[23225]: I0714 11:05:21.819915   23225 controller.go:611] quota admission added evaluator for: endpointslices.discovery.k8s.io
	7月 14 11:05:27 k3s-master k3s[23225]: time="2022-07-14T11:05:27+08:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"bfc7...
	7月 14 11:05:27 k3s-master k3s[23225]: time="2022-07-14T11:05:27+08:00" level=info msg="Event(v1.ObjectReference{Kind:\"Service\", Namespace:\"kube-system\", Name:\"traefik\", UID:\"bfc7...
	7月 14 11:05:35 k3s-master k3s[23225]: E0714 11:05:35.394946   23225 resource_quota_controller.go:413] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1b... the request
	7月 14 11:05:35 k3s-master k3s[23225]: W0714 11:05:35.808493   23225 garbagecollector.go:747] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is current...the request]
	7月 14 11:06:18 k3s-master k3s[23225]: I0714 11:06:18.989418   23225 scope.go:110] "RemoveContainer" containerID="452f28e60eab3341b936403d9845a4b37fb0e8b956075fd1666a9f2e34e56aa4"
	7月 14 11:06:19 k3s-master k3s[23225]: I0714 11:06:19.078841   23225 scope.go:110] "RemoveContainer" containerID="759388a3a7aa54bf7eeba12f76bc9dc9669dc8353e90ced50596698eb136d858
	Hint: Some lines were ellipsized, use -l to show in full.

[root@k3s-master ~]# kubectl get nodes
NAME         STATUS   ROLES                  AGE     VERSION
k3s-master   Ready    control-plane,master   25h     v1.24.2+k3s2

在node节点上

##从service节点上面，获取token
[root@k3s-master ~]# cat /var/lib/rancher/k3s/server/node-token
K103fed60e2b8e6e2f5148bf7b0d48f94ec121f803ad64458e1153ad71181578e31::server:7abaf1f8c247780f803998ec84dfbb4e
#在node1节点上面，执行：
[root@ds3 k3s_install]# cp k3s /usr/local/bin/k3s
[root@ds3 k3s_install]# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL="https://172.70.10.194:6443" sh -s - --node-external-ip 172.60.10.82 --node-ip 172.60.10.82 --flannel-iface eth0 --token "K103fed60e2b8e6e2f5148bf7b0d48f94ec121f803ad64458e1153ad71181578e31::server:7abaf1f8c247780f803998ec84dfbb4e" 
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  systemd: Starting k3s-agent
root@node1:~# systemctl status k3s-agent.service
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-14 10:38:40 CST; 22s ago
       Docs: https://k3s.io
    Process: 47152 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 47154 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 47155 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 47156 (k3s-agent)
      Tasks: 28
     Memory: 17.7M
     CGroup: /system.slice/k3s-agent.service
             └─47156 /usr/local/bin/k3s agent

7月 14 10:38:40 node1 k3s[47156]: time="2022-07-14T10:38:40+08:00" level=info msg="Starting k3s agent v1.24.2+k3s2 (a2372602)"
7月 14 10:38:40 node1 k3s[47156]: time="2022-07-14T10:38:40+08:00" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [172.70.10.194:6443]"
7月 14 10:38:43 node1 k3s[47156]: time="2022-07-14T10:38:43+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39520->127.0.0.1:6444:>
7月 14 10:38:45 node1 k3s[47156]: time="2022-07-14T10:38:45+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39524->127.0.0.1:6444:>
7月 14 10:38:48 node1 k3s[47156]: time="2022-07-14T10:38:48+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39530->127.0.0.1:6444:>
7月 14 10:38:51 node1 k3s[47156]: time="2022-07-14T10:38:51+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39534->127.0.0.1:6444:>
7月 14 10:38:53 node1 k3s[47156]: time="2022-07-14T10:38:53+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39540->127.0.0.1:6444:>
7月 14 10:38:56 node1 k3s[47156]: time="2022-07-14T10:38:56+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39544->127.0.0.1:6444:>
7月 14 10:38:59 node1 k3s[47156]: time="2022-07-14T10:38:59+08:00" level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:39548->127.0.0.1:6444:>
7月 14 10:39:01 node1 k3s[47156]: time="2022-07-1

root@node1:~/.kube# kubectl get nodes
NAME         STATUS   ROLES                  AGE     VERSION
node1        Ready    <none>                 6h39m   v1.24.2+k3s2
k3s-master   Ready    control-plane,master   26h     v1.24.2+k3s2

k3s卸载

参照：官网
如果使用安装脚本安装了 K3，则在安装过程中会生成卸载 K3 的脚本。

卸载 K3s 会删除集群数据和所有脚本。若要使用不同的安装选项重新启动群集，请使用不同的标志重新运行安装脚本。

要从服务器节点卸载 K3，请运行：

/usr/local/bin/k3s-uninstall.sh
要从代理节点卸载 K3，请运行：

/usr/local/bin/k3s-agent-uninstall.sh

k3s\k8s集群-node节点设置不可调度或者删除node节点
k3s集群：

在master节点执行：

确认k3s节点信息

#k3s kubectl get node -o wide
或者
#k3s kubectl get nodes
1、不可调度
#k3s kubectl cordon k3s-agent-node3
取消不可调度
#k3s kubectl uncordon k3s-agent-node3
2、驱逐已经运行的业务容器
#k3s kubectl drain --ignore-daemonsets --delete-local-data k3s-agent-node3
3、删除node 节点
#k3s kubectl delete node k3s-agent-node3
k8s集群只需要把前边的k3s命令去掉即可。

k3s创建ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  name: client-h5-ingress
  namespace: drimoxiaole-xiaoledev
spec:
  rules:
  - host: drimoxiaole-xiaoledev-client-h5.kd.com
    http:
      paths:
      - backend:
          service:
            name: client-h5
            port:
              number: 80
        path: /
        pathType: Prefix

k3s自带traefik的ingress-controller，因此无须安装nginx-ingress，于是乎，画蛇添足的安装了nginx-ingress-controller后，
遇到了如下问题：

[root@k3s-master xiaole]# kubectl apply -f ingress.yaml 
Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": service "ingress-nginx-controller-admission" not found

[root@k3s-master xiaole]# kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted

原因是，有些资源，是与namespace无关的，因此，在删除nginx-ingress的时候，并不能一并删除
如下命令，可以看到与命名空间无关的对象：

kubectl api-resources --namespaced=false

参见：https://pet2cattle.com/2021/02/service-ingress-nginx-controller-admission-not-found

k3s上面搭建longhorn存储

##需要先在每个机器上面安装如下工具
# centos 
yum install iscsi-initiator-utils
# ubuntu
apt-get install open-iscsi

#获取 Longhorn 部署清单文件
wget https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/deploy/longhorn.yaml

#创建 Longhorn 命名空间
kubectl create namespace longhorn-system

然后即可访问longhorn的UI：
在这里插入图片描述

k3s环境中使用ctr

如果想在k3s环境中，使用ctr导入镜像，需要调用k3s ctr添加相关的参数；

[root@master workspace]# k3s ctr i pull docker.io/library/redis:latest
docker.io/library/redis:latest:                                                   resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:495732ba570db6a3626370a1fb949e98273a13d41eb3e26f7ecb1f6e31ad4041:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:e96c03a6dda7d0f28e2de632048a3d34bb1636d0858b65ef9a554441c70f6633: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:05b1f5f3b2c07d2482748fc898e813c372c0124d3929cbe64fca300681f86f01:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:2a9998409df955c541c7130e339eaa0686fdbba1046fbd113c8fd1ac59f520e3:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:dc7b40a0b05d1975384eb9d3e001fe152344d4c1a4563cd39cfe152d0bb0d73b:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:cd7ddcecb993c8ca31f5b0911959ade673454bd0a051d92b7937f6c95af51017:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8cfc9a467ed7a9a37ee62871f67fcb821bae5aaf9409452da3449432df3923e8:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f0036f71a6fe33df2a839d1507119ea1f385a73316b080dd1ccc3245d5188e23:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:7a6db449b51b92eac5c81cdbd82917785343f1664b2be57b22337b0a40c5b29d:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 44.4s                                                                    total:  40.4 M (931.8 KiB/s)                                     
unpacking linux/amd64 sha256:495732ba570db6a3626370a1fb949e98273a13d41eb3e26f7ecb1f6e31ad4041...
done: 1.162280995s
[root@master workspace]# crictl images
IMAGE                                        TAG                    IMAGE ID            SIZE
docker.io/library/redis                      latest                 dc7b40a0b05d1       42.4MB