k8s安装MySQL集群问题汇总

@[TOC]k8s安装MySQL集群问题汇总

1、错误0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

kubectl get pod -A

NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
db                     mysql-0                                      0/2     Pending   0          7m48s

kubectl -n db describe pod mysql-0

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  6m55s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  6m53s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

kubectl -n db get pv,pvc

NAME                    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM             STORAGECLASS   REASON   AGE
persistentvolume/pv-a   2Gi        RWX            Retain           Released   db/data-mysql-0   nfs                     77m
persistentvolume/pv-b   2Gi        RWX            Retain           Released   db/data-mysql-1   nfs                     77m
persistentvolume/pv-c   2Gi        RWX            Retain           Released   db/data-mysql-2   nfs                     77m

NAME                                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/data-mysql-0   Pending                                      nfs            2m54s

原因：pv状态Released未释放，创建pvc时找不到合适的pv
解决方法：删除pvc,pv，然后重建

kubectl -n db delete pvc data-mysql-0
kubectl -n db delete pv pv-a pv-b pv-c

kubectl -n db describe pod mysql-0

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  2m39s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  2m37s  default-scheduler  0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

kubectl -n db describe pvc

Name:          data-mysql-0
Namespace:     db
StorageClass:  nfs
Status:        Pending
Volume:        
Labels:        app=mysql
               app.kubernetes.io/name=mysql
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       mysql-0
Events:
  Type     Reason              Age                    From                         Message
  ----     ------              ----                   ----                         -------
  Warning  ProvisioningFailed  3m11s (x5 over 4m10s)  persistentvolume-controller  storageclass.storage.k8s.io "nfs" not found
  Warning  ProvisioningFailed  7s (x7 over 97s)       persistentvolume-controller  storageclass.storage.k8s.io "nfs" not found

pvc创建失败，检查pv、pvc配置
pv配置：

metadata:
  name: pv-a
spec:
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    server: 192.168.216.4
    path: /nfsdata/mysql-01

pvc：配置

- metadata:
      name: data
      namespace: db
    spec:
      accessModes:
      accessModes: ["ReadWriteOnce"]
      storageClassName: nfs
      resources:
        requests:
          storage: 1Gi

原因：pv和pvc accessModes配置不一致，造成创建pvc时找不到合适pv，同时pvc大小不能超过pv大小
解决方法：调整参数一致，删除pvc,pv，然后重建

2、init-mysql CrashLoopBackoff.

kubectl -n db describe pod mysql-0

Events:
  Type     Reason     Age                             From               Message
  ----     ------     ----                            ----               -------
  Normal   Scheduled  3m43s                           default-scheduler  Successfully assigned db/mysql-0 to node02
  Normal   Pulled     <invalid> (x5 over <invalid>)   kubelet            Container image "mysql:5.7" already present on machine
  Normal   Created    <invalid> (x5 over <invalid>)   kubelet            Created container init-mysql
  Normal   Started    <invalid> (x5 over <invalid>)   kubelet            Started container init-mysql
  Warning  BackOff    <invalid> (x10 over <invalid>)  kubelet            Back-off restarting failed container

查看pod mysql-0日志

kubectl -n db logs mysql-0

error: a container name must be specified for pod mysql-0, choose one of: [mysql xtrabackup] or one of the init containers: [init-mysql clone-mysql]

kubectl -n db logs mysql-0 init-mysql

++ hostname
bash: line 5: hostname: command not found
+ [[ '' =~ -([0-9]+)$ ]]
+ exit 1

原因：shell 命令hostname不存在，可能时容器bash问题
解决方法：yaml文件中hostname命令改成环境变量$HOSTNAME

   #[ `hostname` =~ -([0-9]+)$ ]] || exit 1
   [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1

删除statefulset后重建

kubectl -n db get pod -o wide

NAME      READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
mysql-0   2/2     Running   0          30s   10.122.140.123   node02   <none>           <none>
mysql-1   2/2     Running   0          20s   10.122.140.89    node02   <none>           <none>
mysql-2   1/2     Running   0          9s    10.122.140.90    node02   <none>           <none>

kubectl -n db describe pod

Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  64s        default-scheduler  Successfully assigned db/mysql-2 to node02
  Normal  Pulled     <invalid>  kubelet            Container image "mysql:5.7" already present on machine
  Normal  Created    <invalid>  kubelet            Created container init-mysql
  Normal  Started    <invalid>  kubelet            Started container init-mysql
  Normal  Pulled     <invalid>  kubelet            Container image "gcr.io/google-samples/xtrabackup:1.0" already present on machine
  Normal  Created    <invalid>  kubelet            Created container clone-mysql
  Normal  Started    <invalid>  kubelet            Started container clone-mysql
  Normal  Pulled     <invalid>  kubelet            Container image "mysql:5.7" already present on machine
  Normal  Created    <invalid>  kubelet            Created container mysql
  Normal  Started    <invalid>  kubelet            Started container mysql
  Normal  Pulled     <invalid>  kubelet            Container image "gcr.io/google-samples/xtrabackup:1.0" already present on machine
  Normal  Created    <invalid>  kubelet            Created container xtrabackup
  Normal  Started    <invalid>  kubelet            Started container xtrabackup

k8s MySQL集群创建完成。