Centos7下Kubernetes配置使用GlusterFS

操作环境

网络拓扑

环境说明：

OS：

CentOS Linux release 7.4.1708 

docker：

Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Mon Dec 11 16:08:42 2017
 OS/Arch:         linux/amd64

kubernetes：

Kubernetes v1.5.2

glusterfs：

glusterfs 3.12.3

操作步骤

    Kubernetes环境已经完成部署完成，部署步骤参照 Centos7 下Kubernetes集群安装部署。

部署Glusterfs

    这里有5台服务器，10.10.200.226、10.10.200.227作为GlusterFS存储服务器，10.10.200.224、

10.10.200.229、10.10.200.230调用GlusterFS，同样也需要安装Gluster。

1.分别在5台服务器上安装Gluster

[root@k8s-master ~]#yum -y install centos-release-gluster
[root@k8s-master ~]#yum -y install glusterfs glusterfs-fuse glusterfs-server

2.启动Glusterfs

[root@k8s-master ~]#systemctl enable glusterd
[root@k8s-master ~]#systemctl start glusterd

3.创建GlusterFS Cluster

编辑/etc/hosts文件

[root@k8s-master ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.200.226   glusterfs-node1
10.10.200.227   glusterfs-node2
10.10.200.224   k8s-master
10.10.200.229   k8s-node1
10.10.200.230   k8s-node2

选择1台服务器添加其他服务器

[root@gluster-node1 ~]#gluster peer probe k8s-master
[root@gluster-node1 ~]#gluster peer probe k8s-node1
[root@gluster-node1 ~]#gluster peer probe k8s-node2
[root@gluster-node1 ~]#gluster peer probe gluster-node2

查看gluster节点状态

[root@k8s-master yaml]# gluster peer status
Number of Peers: 4

Hostname: glusterfs-node2
Uuid: 3c5c5994-0c05-4be0-9ec3-a431f193a1f0
State: Peer in Cluster (Connected)

Hostname: glusterfs-node1
Uuid: 31ac3dff-94a3-4166-991b-bf4727113539
State: Peer in Cluster (Connected)

Hostname: 10.10.200.229
Uuid: 7c2f03fe-e2cb-412e-a91f-2baf873610ef
State: Peer in Cluster (Connected)
Other names:
10.10.200.229

Hostname: 10.10.200.230
Uuid: 314a7676-d1eb-4c58-9617-8c510c93961d
State: Peer in Cluster (Connected)

3.创建volume

这里只选择10.10.200.226/10.10.200.227来进行创建volume

[root@gluster-node1 brick1]# gluster volume create mysql-volume replica 2 glusterfs-node2:/data/brick1/mysql-volume/ glusterfs-node1:/data/brick1/mysql-volume/

启动mysql-volume

[root@gluster-node1 brick1]# gluster volume start mysql-volume
volume start: mysql-volume: success

查看volume状态

[root@gluster-node1 brick1]# gluster volume status
Status of volume: mysql-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick glusterfs-node2:/data/brick1/mysql-vo
lume                                        49152     0          Y       2976 
Brick glusterfs-node1:/data/brick1/mysql-vo
lume                                        49152     0          Y       7835 
Self-heal Daemon on localhost               N/A       N/A        Y       7856 
Self-heal Daemon on k8s-master              N/A       N/A        Y       2025 
Self-heal Daemon on glusterfs-node2         N/A       N/A        Y       2997 
 
Task Status of Volume mysql-volume
------------------------------------------------------------------------------
There are no active volume tasks

到这里就完成了2台Glusterfs存储服务器的配置

配置Kubernetes

    可以在https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/glusterfs上面找

到kuernetes配置gluster的文件glusterfs-endpoint.json/glusterfs-pod.json/glusterfs-services.json。

     这里只需要glusterfs-endpoint.json，glusterfs-service.json。

修改glusterfs-endpoint.json文件如下，主要是ip地址的修改

[root@k8s-master yaml]# vi glusterfs-endpoints.json 
{
  "kind": "Endpoints",
  "apiVersion": "v1",
  "metadata": {
    "name": "glusterfs-cluster"
  },
  "subsets": [
    {
      "addresses": [
        {
          "ip": "10.10.200.226"
        }
      ],
      "ports": [
        {
          "port": 1
        }
      ]
    },
    {
      "addresses": [
        {
          "ip": "10.10.200.227"
        }
      ],
      "ports": [
        {
          "port": 1
        }
      ]
    }
  ]
}

glusterfs-service.json文件如下，不需修改

[root@k8s-master yaml]# vi glusterfs-service.json 
{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "glusterfs-cluster"
  },
  "spec": {
    "ports": [
      {"port": 1}
    ]
  }
}

启用上述两文件

[root@k8s-master yaml]# kubectl apply -f glusterfs-endpoints.json 
endpoints "glusterfs-cluster" created
[root@k8s-master yaml]# kubectl apply -f glusterfs-service.json 
service "glusterfs-cluster" created

[root@k8s-master yaml]# kubectl get ep
NAME                ENDPOINTS                         AGE
glusterfs-cluster   10.10.200.226:1,10.10.200.227:1   31s
kubernetes          10.10.200.224:6443                6d
[root@k8s-master yaml]# kubectl get service
NAME                CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
glusterfs-cluster   172.17.62.135   <none>        1/TCP     33s
kubernetes          10.254.0.1      <none>        443/TCP   6d

创建presistent volume以及presistent volume claim yaml文件

[root@k8s-master yaml]# vi glusterfs-pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: gluster-dev-volume1
  labels:
    name: mysql1

spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: "glusterfs-cluster"
    path: "mysql-volume"
    readOnly: false

[root@k8s-master yaml]# vi glusterfs-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: glusterfs-mysql1
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      name: "mysql1"

启动上述两文件

[root@k8s-master yaml]# kubectl apply -f glusterfs-pv.yaml 
persistentvolume "gluster-dev-volume1" created
[root@k8s-master yaml]# kubectl apply -f glusterfs-pvc.yaml 
persistentvolumeclaim "glusterfs-mysql1" created
[root@k8s-master yaml]# kubectl get pv
NAME                  CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                      REASON    AGE
gluster-dev-volume1   10Gi       RWX           Retain          Bound     default/glusterfs-mysql1             8s
[root@k8s-master yaml]# kubectl get pvc
NAME               STATUS    VOLUME                CAPACITY   ACCESSMODES   AGE
glusterfs-mysql1   Bound     gluster-dev-volume1   10Gi       RWX           6s

以上就完成了kubernetes配置glusterfs的全过程，下面我们通过运行mysql测试一下，编写mysql deployment yaml如下

[root@k8s-master yaml]# vi mysql-deployment.yaml 
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  type: NodePort
  ports:
  - name: mysqlport
    port: 3306
    nodePort: 32006
  selector:
    name: mysql
---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mysql
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: mysql
    spec:
      containers:
        - name: mysqlcontainer
          image: docker.io/mysql
          imagePullPolicy: IfNotPresent
          env:
          - name: MYSQL_ROOT_PASSWORD
            value: root123456
          ports:
            - containerPort : 3306
          volumeMounts:
            - name: gluster-mysql-data
              mountPath: "/var/lib/mysql"
      volumes:
        - name: gluster-mysql-data
          persistentVolumeClaim:
            claimName: glusterfs-mysql1

启动该文件

[root@k8s-master yaml]# kubectl apply -f mysql-deployment.yaml 
service "mysql" created
deployment "mysql" configured

查看状态

[root@k8s-master yaml]# kubectl get service
NAME                CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
glusterfs-cluster   172.17.62.135    <none>        1/TCP            13m
kubernetes          10.254.0.1       <none>        443/TCP          6d
mysql               172.17.231.221   <nodes>       3306:32006/TCP   1m
[root@k8s-master yaml]# kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
mysql-3736381796-5j11t   1/1       Running   0          1m
nginx-controller-1hxkz   1/1       Running   3          3d
nginx-controller-3xcl8   1/1       Running   3          3d
[root@k8s-master yaml]# kubectl get pods -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
mysql-3736381796-5j11t   1/1       Running   0          1m        172.17.99.3   10.10.200.230
nginx-controller-1hxkz   1/1       Running   3          3d        172.17.99.2   10.10.200.230
nginx-controller-3xcl8   1/1       Running   3          3d        172.17.17.2   10.10.200.229

登录mysql

[root@k8s-node2 ~]# mysql -h 10.10.200.230 -P 32006 -uroot -proot123456
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>