k8s经常使用命令记录

k8s搭建

我的本地服务器
INSTALL_K3S_SKIP_DOWNLOAD=true \
K3S_URL=https://10.211.55.4:6443 \
K3S_TOKEN=K1047fd86f774821f115665b59ffa67fd0695b0e7e1f42767c0968362281c83cdfc::server:317e2d1e92bda510675f09b55a8775ea \
./install.sh

查看启动情况

watch kubectl get node 

查看配置列表

ps -ef | grep containerd

在每个节点上创建 每个节点上新建/etc/rancher/k3s/registries.yaml文件

写下如下内容

mirrors:
  docker.io:
    endpoint:
      - "https://fsp2sfpr.mirror.aliyuncs.com/"

然后主节点重启

systemctl restart k3s

从节点重启

systemctl restart k3s-agent

每个节点查看镜像是否生效了,底部有镜像信息

cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

---

查看有哪些镜像

kubectl get pods --all-namespaces -o jsonpath="{..image}" | tr -s '[[:space:]]' '\n' | sort | uniq

尝试装一个pod

 kubectl run mynginx --image=nginx:1.22
 watch kubectl get pod
 查看日志
 kubectl logs -f mynginx
 查看具体信息
 kubectl describe pod mynginx
 删除
 kubectl delete pod mynginx

查看详细信息

kubectl get node -owide


NAME      READY   STATUS    RESTARTS   AGE   IP          NODE         NOMINATED NODE   READINESS GATES
mynginx   1/1     Running   0          25s   10.42.0.4   k8s-master   <none>           <none>

---

创建副本

kubectl create deployment nginx-deploy --image=nginx:1.22 --replicas=3

查看deploy

kubectl get deploy
删除
kubectl delete deploy nginx-deploy

NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deploy 3/3 3 3 2m39s

查看replicaSet

kubectl get replicaSet
[root@k8s-master ~]# kubectl get replicaSet
NAME                      DESIRED   CURRENT   READY   AGE
nginx-deploy-855866bb46   3         3         3       4m9s
里面的副本集是多少 对应的name ，滚动更新的时候可以看这个name变化

动态扩容和缩容

将副本集调整为2
kubectl scale deploy nginx-deploy --replicas=2

监听 
kubectl get deploy --watch

自动扩容和缩容

kubectl autoscale deployment/nginx-deploy --min=1 --max=10 --cpu-percent=75


并设置它的最小 Pod 数量为 3，最大 Pod 数量为 10，CPU 利用率的目标百分比为 75%。在 CPU 利用率超过 80% 时，将增加 Pod 数量以满足负载要求

监听节点变化
kubectl get deploy --watch


查看自动auto扩容和缩容
kubectl get hpa

获取自动扩容和缩容的详细信息

[root@k8s-master ~]# kubectl get hpa
NAME           REFERENCE                 TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx-deploy   Deployment/nginx-deploy   <unknown>/75%   3         10        3          116



NAME: HPA资源的名称
REFERENCE: HPA控制的目标资源，通常是Deployment或ReplicaSet名称和kind
TARGETS: 水平自动扩展的资源指标，包括CPU利用率和内存利用率
MINPODS: HPA自动缩放时，最小允许的Pod数量
MAXPODS: HPA自动缩放时，最大允许的Pod数量
REPLICAS: 当前正在运行的Pod数量
AGE: HPA资源创建的时间


删除自动扩容和缩容
kubectl delete hpa nginx-deploy

滚动更新

kubectl get deploy -owide

1.2.2更新成 1.2.3版本
kubectl set image deployment/nginx-deploy nginx=nginx:1.2.3

同时另外一个窗口监听副本name变化
kubectl set image deployment/nginx-deploy nginx=nginx:1.2.3

然后在查看副本信息
kubectl get deploy -owide

回滚操作

查看当前版本
kubectl get deploy -owide

kubectl rollout history deploy/nginx-deploy

输出:
deployment.apps/nginx-deploy
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

查看具体信息

kubectl rollout history deploy/nginx-deploy revision=1
会输出具体信息

回滚具体操作
kubectl rollout undo deploy/nginx-deploy --to-revision=1

---

service

kubectl expose deploy/nginx-deploy --name=nginx-service --port=9091 --target-port=80
--port 宿主机端口
--target-port 容器端口

查看 
kubectl get service
输出
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes        ClusterIP   10.43.0.1       <none>        443/TCP    26d
ngnginx-service   ClusterIP   10.43.129.212   <none>        9091/TCP   7s
其中 clusterIP 为容器里面的ip,集群内部可以测试
curl 10.43.129.212:9091 输出nginx

验证service pod里面访问

kubectl run test_nginx -it --image=nginx:1.22 --rm -- bash

创建一个临时的 pod 
kubectl get pod 
会显示一个临时的pod

 kubectl get service
 输出
ngnginx-service   ClusterIP   10.43.129.212   <none>        9091/TCP
 
 curl测试
 curl ngnginx-service:9091  
 返回响应的结果

service详解

kubectl describe service ngnginx-service

endpoints 有三个地址

kubectl get pod
选择其中一个进入
操作命令如下
[root@k8s-master ~]# kubectl exec -it nginx-deploy-855866bb46-lwq4n -- bash
root@nginx-deploy-855866bb46-lwq4n:/# cd /usr/share/nginx/html/
root@nginx-deploy-855866bb46-lwq4n:/usr/share/nginx/html# ls
50x.html  index.html
root@nginx-deploy-855866bb46-lwq4n:/usr/share/nginx/html# echo gitxuzan > index.html
root@nginx-deploy-855866bb46-lwq4n:/usr/share/nginx/html#

 curl 10.43.129.212:9091
 会有三分之1 输出gitxuzan 负载均衡

网络类型有

ClusterIp 默认网络
NodePort 集群外部主机访问到内部
ExternalName: 将集群外部网络引入集群内部
LoadBalancer: 使用云提供商的负载均衡器向外部暴露服务

给service 创建一个 NodePort

kubectl expose deploy/nginx-deploy --name=nginx-outside --type=NodePort --port=9091 --target-port=80

然后查看
kubectl get service

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes        ClusterIP   10.43.0.1       <none>        443/TCP          26d
ngnginx-service   ClusterIP   10.43.129.212   <none>        9091/TCP         45m
nginx-outside     NodePort    10.43.26.27     <none>        9091:32131/TCP   36s



再浏览器访问
宿主机ip:32131 能访问到数据
例如 curl 10.211.55.4:32131
例如 curl 10.211.55.5:32131
例如 curl 10.211.55.6:32131

---

命名空间 Namespace

用于deploy,service,pod

[root@k8s-master ~]# kubectl get namespace
NAME              STATUS   AGE
default           Active   26d
kube-system       Active   26d   控制平面和node
kube-public       Active   26d   自动创建公共命名空间,整个集群公用可见可读资源放到这个空间
kube-node-lease   Active   26d   kubectl get lease -A  检测每个节点是否发生故障

查看命名空间运行的pod

kubectl get pod -A

创建一个命名空间

创建一个命名空间
kubectl create ns mydevelop

在mydevelop下运行一个nginx
kubectl run nginx --image=nginx:1.22 -n=mydevelop

查看pod
[root@k8s-master ~]# kubectl get pod -n=mydevelop
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          3m9s

把创建好的命名空间设置为默认的 mydevelop

ktbectl config set-context $(kubectl config current-context) --namespace=mydevelop

---

声明式的配置 yaml

namespaces	缩写	Kind
namespaces	ns	Namespace
nodes	no	Node
pods	po	Pod
services	svc	Service
deveployments	deploy	Deveployment
replicasets	rs	ReplicaSet
statefulsets	sts	StatefulSet

kubernetes yaml模板
https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/

例如创建简单pod模板 pods/simple-pod.yaml

apiVersion: v1
kind: Pod 
metadata:  # 元数据
  name: nginx-pod # 元数据pod名称
spec:
  containers:
  - name: nginx
    image: nginx:1.22
    ports:
    - containerPort: 80

执行命令创建

kubectl apply -f pods/simple-pod.yaml

kubectl get pod 

执行命令删除

kubectl delete -f pods/simple-pod.yaml 
kubectl get pod 

yaml 标签使用查看容器

https://kubernetes.io/zh-cn/docs/concepts/overview/working-with-objects/labels/

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod-label
  labels:
    environment: production
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.22
    ports:
    - containerPort: 80

展示标签

kubectl get pod --show-labels=true

根据标签查询
kubectl get pod -l "app=nginx,environment=production"

service yaml 模板

https://kubernetes.io/zh-cn/docs/concepts/services-networking/service/

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  type: NodePort
  selector:
    app: nginx # 上面的pod 的labels
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30007

执行 kubectl apply -f pods/service-pod.yaml

kubectl get service  查看创建的 nginx-service

查看service 详情

kubectl describe svc/nginx-service  等同于 kubectl describe service nginx-service
查看endpoints 

 IP
kubectl get pod -l "app=nginx" -owide  

service 的 endpoints 和 pod的ip 一致，说明通过label绑定起来了

---

容器运行时的接口CRI

k8s在 v1.2.4版本开始，把dockershim移除，crictl 指令和docker一样

crictl 命令除了没有导入导出，其他基本都支持和docker一样的命令

导入导出需要指令 ctr
演示ctr 导入导出

docker pull alpine:3.15
 docker save alpine:3.15 > alpine-3.15.tar
 
 本地传到服务器
 scp alpine-3.15.tar root@10.211.55.4:~
 
 在宿主机导入  -n k8s.io 是k8s默认空间镜像namespace 
 ctr -n k8s.io images import alpine-3.15.tar --platform linux/arm64
 crictl images 可以查看到了 alpine:3.15的导入
 
 
 导出在宿主机上
 ctr -n k8s.io images export alpine.tar docker.io/library/alpine:3.15 --platform=linux/arm64

---

金丝雀发布

apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-v1
  namespace: dev # 依赖上面的namespace
  labels:
    app: nginx-deployment-v1
spec:
  replicas: 3
  selector:
    matchLabels: #需要和template.metadata.labels一致
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.22
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: dev
spec:
  type: NodePort
  selector:
    app: nginx # 通过labels 绑定pod
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30008

然后通过

kubectl get all -n=dev 可以查看详细信息
kubectl get service -n dev
kubectl describe service xxxxx 查看endpoints

发布一个v2 版本，灰度发布

spec:
  replicas: 1
  selector:
    matchLabels: #需要和template.metadata.labels一致
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        track: v2v2v2v2  # 测试区分用的
    spec:
      containers:
      - name: nginx-v2
        image: docker/getting-started  # 可以本地导入过来
        ports:
        - containerPort: 80

通过

1. kubectl get all -n=dev 可以查看详细信息
2. kubectl get service -n dev
3. kubectl describe service xxxxx 查看endpoints
4. 最后 curl xxxxxx:30008 查看是否负载均衡了
5. 当灰度发布稳定后，升级新版本

新版本副本调整为3
 kubectl scale deploy nginx-deployment-v2 --replicas=3 -n=dev
 观察变化
 kubectl get all -n dev
 
 下线nginx-deployment-v1版本
 kubectl scale deploy nginx-deployment-v1 --replicas=0 -n=dev

6. 缺点：不能控制用户和网络去灰度区分
7. 清除所有，kubectl delete all --all -n=dev

---

有状态的pod

• hostPath 几个类型

hostPath ->type	缩写
DirectoryOrCreate	目录不存在就创建
Directory	挂载已存在的目录,不存在挂载会报错
FileOrCreate	文件不存在则创建
File	文件类型，不存在挂载会报错
Socket	挂载UNIX套接字。例如挂载/var/run/docker.sock进程

• 写一个mysql-pod.yaml文件

apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    app: mysql
spec:
  containers:
    - name: mysql
      image: mysql:8.0
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
      volumeMounts:
        # mysql 镜像目录
        - mountPath: /var/lib/mysql
          name: data-volume
  volumes:
    - name: data-volume
      hostPath:
        # 宿主机的目录
        path: /home/mysql/data
        # 存储的是目录类型
        type: DirectoryOrCreate

存储卷相关文档:
https://kubernetes.io/zh-cn/docs/concepts/storage/volumes/

1. 执行 kubectl apply -f mysql-pod.yaml
2. 执行 kubectl get pod -owide

ConfigMaps

通过例子把my.cnf 通过configmaps 存储到etcd，然后落到容器上

apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    app: mysql
spec:
  containers:
    - name: mysql
      image: mysql:8.0
      env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
      volumeMounts:
        # mysql 镜像目录
        - mountPath: /var/lib/mysql
          name: data-volume
        - mountPath: /etc/mysql/conf.d  # https://hub.docker.com/_/mysql
          name: configmap-volume
          readOnly: true # 只读
  volumes:
    - name: data-volume
      hostPath:
        # 宿主机的目录
        path: /home/mysql/data
        # 存储的是目录类型
        type: DirectoryOrCreate
    # configmap卷
    - name: configmap-volume
      configMap:
        name: configmap-mysql  # 匹配下面metatata
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-mysql
data:
  # 类文件键文件名my.cnf
  my.cnf: |
    [mysqld]
    character-set-server=utf8mb4
    collation-server=utf8mb4_general_ci
    init-connect='SET NAMES utf8mb4'
    [client]
    default-character-set=utf8mb4
    [mysql]
    default-character-set=utf8mb4

然后测试字符集是否生效

kubectl apply -f mysql-pod.yaml
kubectl exec -it mysql-pod -- bash
查看到 cat /etc/mysql/conf.d/my.cnf
登录mysql查看字符集是否生效
mysql -uroot -p 
执行
mysql> SHOW VARIABLES LIKE 'character_set_client';
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| character_set_client | utf8mb4 |
+----------------------+---------+
1 row in set (0.00 sec)

mysql> SHOW VARIABLES LIKE 'character_set_client';
+----------------------+---------+
| Variable_name        | Value   |
+----------------------+---------+
| character_set_client | utf8mb4 |
+----------------------+---------+

说明设置生效了

---

secret 使用

文档: https://kubernetes.io/zh-cn/docs/tasks/configmap-secret/managing-secret-using-config-file/
文档: https://kubernetes.io/zh-cn/docs/concepts/configuration/secret/#using-secrets-as-environment-variables

apiVersion: v1
kind: Secret
metadata:
  name: mysql-password
type: Opaque
data:
  password: MTIzNDU2
---
apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    app: mysql
spec:
  containers:
    - name: mysql
      image: mysql:8.0
      env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              # 对应   name: mysql-password
              name: mysql-password
              # 对应 password: MTIzNDU2
              key: password
              optional: false
      volumeMounts:
        # mysql 镜像目录
        - mountPath: /var/lib/mysql
          name: data-volume
        - mountPath: /etc/mysql/conf.d
          name: configmap-volume
          readOnly: true
  volumes:
    - name: data-volume
      hostPath:
        # 宿主机的目录
        path: /home/mysql/data
        # 存储的是目录类型
        type: DirectoryOrCreate
    # configmap卷
    - name: configmap-volume
      configMap:
        name: configmap-mysql
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-mysql
data:
  # 类文件键
  my.cnf: |
    [mysqld]
    character-set-server=utf8mb4
    collation-server=utf8mb4_general_ci
    init-connect='SET NAMES utf8mb4'
    [client]
    default-character-set=utf8mb4
    [mysql]
    default-character-set=utf8mb4

然后执行验证密码

kubectl apply -f mysql-pod-secret.yaml

创建了secret 查看
 kubectl describe secret  mysql-password

 kubectl exec -it mysql-pod -- bash 进入到pod
 mysql -uroot -p

---

卷

• 临时卷， 跟随pod一起删除

1. emptyDir 做缓存或者存储日志
2. configMap secret ， downwardAPI

kubectl get cm // config-maps
kubectl get pod -owide 查看部署到那台机器，然后查看/var/lib/kubelet/pods/093d358f-ae75-428b-8374-af85466ab63a/volumes/kubernetes.io~configmap/configmap-volume/my.cnf 实际临时券存储的位置，删除后临时券就对应删除了
查看到了config-map 对象，如果删除 kubectl delete pod mysql-pod  ,config-map 对象 还是存在

• 持久卷: 删除pod后， 持久券不被删除

1. 本地存储 - hostPath 、local
2. 网络存储 - NFS
3. 分布式存储 - Ceph(cephfs文件存储,rdb快存储)

https://kubernetes.io/zh-cn/docs/concepts/storage/storage-classes/#local

https://kubernetes.io/zh-cn/docs/concepts/storage/volumes/

apiVersion: storage.k8s.io/v1
# 存储类
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
---
apiVersion: v1
# 持久卷
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 2Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - k8s-worker1

声明方式持久卷
https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#create-a-pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: local-storage
  accessModes:
  # 卷可以被一个节点读写方式挂载，并允许多个节点多个pod访问
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

kubectl get pvc 查看bound的storageClassName

查看持久券

• 投射券：projected 卷可以将多个券映射到同一个目录

动态创建持久卷

apiVersion: v1
kind: Secret
metadata:
  name: mysql-password
type: Opaque
data:
  password: MTIzNDU2
---
apiVersion: v1
kind: Pod
metadata:
  name: mysql-pod
  labels:
    app: mysql
spec:
  containers:
    - name: mysql
      image: mysql:8.0
      env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              # 对应   name: mysql-password
              name: mysql-password
              # 对应 password: MTIzNDU2
              key: password
              optional: false
      volumeMounts:
        # mysql 镜像目录
        - mountPath: /var/lib/mysql
          name: data-volume
        - mountPath: /etc/mysql/conf.d
          name: configmap-volume
          readOnly: true
  volumes:
    - name: data-volume
      persistentVolumeClaim:
        ## kubectl get pvc 的name
        #claimName: task-pv-claimy
        # 默认存储 WaitForFirstConsumer 模式 删除pvc时候，会删除pc
        claimName: local-pvc-1
#      hostPath:
#        # 宿主机的目录
#        path: /home/mysql/data
#        # 存储的是目录类型
#        type: DirectoryOrCreate
    # configmap卷
    - name: configmap-volume
      configMap:
        name: configmap-mysql
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-mysql
data:
  # 类文件键
  my.cnf: |
    [mysqld]
    character-set-server=utf8mb4
    collation-server=utf8mb4_general_ci
    init-connect='SET NAMES utf8mb4'
    [client]
    default-character-set=utf8mb4
    [mysql]
    default-character-set=utf8mb4

kubectl get sc 查看 VOLUMEBINDINGMODE
kubectl describe pod mysql-pod 查看持久卷
kubectl delete pvc 会自动删除pv