基于Ubuntu系统的GlusterFS集群独立部署

一、服务器准备

主机名	IP 地址	操作系统	设备
ksmaster-01	192.168.0.2	Ubuntu 18.04，4 核，4 GB 内存	/dev/sdb 300 GB
ksworker-01	192.168.0.3	Ubuntu 18.04，4 核，4 GB 内存	/dev/sb 300 GB
ksworker-02	192.168.0.4	Ubuntu 18.04，4 核，4 GB 内存	dev/sdb 300 GB

二、集群安装

1. 设置无密码ssh登录

• 登录root账户

  sudo -i
  

• 设置root密码

  passwd
  

• 允许无密码登录

  vi /etc/ssh/sshd_config
  

  确保：

  PermitRootLogin yes
  PasswordAuthentication yes
  

  /etc/init.d/ssh restart
  

• 编辑hosts文件

  vi /etc/hosts
  

  添加：

  192.168.0.2 ksmaster-01
  192.168.0.3 ksworker-01
  192.168.0.4 ksworker-02
  

以上命令，每个主机都需要执行

• 无密码ssh登录
  • 在ksmaster-01主机生成秘钥，回车键跳过提示

  ssh-keygen
  

  • 将秘钥复制到所有主机

  ssh-copy-id root@ksmaster-01
  

  ssh-copy-id root@ksworker-01
  

  ssh-copy-id root@ksworker-02
  

  • 验证ksmaster-01主机是否能登录所有主机

  ssh root@ksmaster-01
  

  ssh root@ksworker-01
  

  ssh root@ksworker-02
  

3. 安装glusterFS

• 安装software-properties-common

  apt-get install software-properties-common
  

• 添加GlusterFS ppa

  add-apt-repository ppa:gluster/glusterfs-7
  apt-get update
  

• 安装GlusterFS服务

  apt-get install glusterfs-server -y
  glusterfs -V
  

• 启动GlusterFS

  systemctl start glusterd
  systemctl enable glusterd
  

在所有主机上执行以上命令

• 加载内核模块
  • 加载三个内核

  echo dm_thin_pool | sudo tee -a /etc/modules
  echo dm_snapshot | sudo tee -a /etc/modules
  echo dm_mirror | sudo tee -a /etc/modules
  

  • 安装

  apt-get -y install thin-provisioning-tools
  

在所有主机上执行以上命令

4. 创建GlusterFS集群

• 在ksmaster-01添加其他节点

  gluster peer probe ksworker-01
  gluster peer probe ksworker-02
  

• 验证

  gluster peer status
  

  Number of Peers: 2
  
  Hostname: ksworker-01
  Uuid: 9d0a63ba-24e2-4e68-9638-643cbc48795a
  State: Peer in Cluster (Connected)
  
  Hostname: ksworker-02
  Uuid: d5c140df-b3ad-49f0-bcaa-709dbd6ca84d
  State: Peer in Cluster (Connected)
  

5. 安装Heketi

• 在ksmaster-01上安装Heketi

  wget https://github.com/heketi/heketi/releases/download/v7.0.0/heketi-v7.0.0.linux.amd64.tar.gz
  

• 文件解压缩

  tar -xf heketi-v7.0.0.linux.amd64.tar.gz
  cd heketi
  cp heketi /usr/bin
  cp heketi-cli /usr/bin
  

• 创建Heketi服务文件

  vi /lib/systemd/system/heketi.service
  

  [Unit]
  Description=Heketi Server
  [Service]
  Type=simple
  WorkingDirectory=/var/lib/heketi
  ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
  Restart=on-failure
  StandardOutput=syslog
  StandardError=syslog
  [Install]
  WantedBy=multi-user.target
  

• 创建Heketi文件夹

  mkdir -p /var/lib/heketi
  mkdir -p /etc/heketi
  

• 创建Heketi配置文件

  vi /etc/heketi/heketi.json
  

  {
    "port": "38080",
    "use_auth": true,
    "jwt": {
      "admin": {
        "key": "uJfBjRSH"
      },
      "user": {
        "key": "uJfBjRSH"
      }
    },
    "glusterfs": {
      "executor": "ssh",
      "sshexec": {
        "keyfile": "/root/.ssh/id_rsa",
        "user": "root",
  	  "port": "22",
        "fstab": "/etc/fstab"
      },
      "db": "/var/lib/heketi/heketi.db",
      "loglevel" : "debug"
    }
  }
  

• 启动Heketi

  systemctl start heketi
  systemctl enable heketi
  

• 验证
  • 查看Heketi状态

  systemctl status heketi
  

  ● heketi.service - Heketi Server
     Loaded: loaded (/lib/systemd/system/heketi.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-01-11 01:33:43 UTC; 4s ago
   Main PID: 11142 (heketi)
      Tasks: 13 (limit: 4915)
     CGroup: /system.slice/heketi.service
             └─11142 /usr/bin/heketi --config=/etc/heketi/heketi.json
  

  • 访问Heketi

  curl 127.0.0.1:38080/hello
  

  Hello from Heketi
  

• Heketi拓扑配置

  vi /etc/heketi/topology.json
  

  {
    "clusters": [
      {
        "nodes": [
          {
            "node": {
              "hostnames": {
                "manage": [
                  "192.168.0.2"
                ],
                "storage": [
                  "192.168.0.2"
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/sdb"
            ]
          },
          {
            "node": {
              "hostnames": {
                "manage": [
                  "192.168.0.3"
                ],
                "storage": [
                  "192.168.0.3"
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/sdb"
            ]
          },
          {
            "node": {
              "hostnames": {
                "manage": [
                  "192.168.0.4"
                ],
                "storage": [
                  "192.168.0.4"
                ]
              },
              "zone": 1
            },
            "devices": [
              "/dev/sdb"
            ]
          }
        ]
      }
    ]
  }
  

  echo "export HEKETI_CLI_SERVER=http://localhost:38080" >> /etc/profile.d/heketi.sh
  echo "alias heketi-cli='heketi-cli --server $HEKETI_CLI_SERVER --user admin --secret uJfBjRSH'" >> ~/.bashrc
  source /etc/profile.d/heketi.sh
  source ~/.bashrc
  heketi-cli topology load --json=/etc/heketi/topology.json
  

  Creating cluster ... ID: 6c9c89de14a83a3746482a8c3030850a
          Allowing file volumes on cluster.
          Allowing block volumes on cluster.
          Creating node 192.168.0.2 ... ID: 6f9b852d767826d42670d3affd2ace9a
                  Adding device /dev/sdb ... OK
          Creating node 192.168.0.3 ... ID: f5349a36fc68c9ee3ecfc25f928cc54b
                  Adding device /dev/sdb ... OK
          Creating node 192.168.0.4 ... ID: e67532705f138694355c88c6667c9199
                  Adding device /dev/sdb ... OK
  

  heketi-cli cluster info 6c9c89de14a83a3746482a8c3030850a
  

  Cluster id: 6c9c89de14a83a3746482a8c3030850a
  Nodes:
  6f9b852d767826d42670d3affd2ace9a
  e67532705f138694355c88c6667c9199
  f5349a36fc68c9ee3ecfc25f928cc54b
  Volumes:
  
  Block: true
  
  File: true
  

• 问题
  在使用openssh高版本生成RSA秘钥与Heketi有版本兼容问题

  root@ksmaster-01:~# systemctl status heketi
  ● heketi.service - Heketi Server
       Loaded: loaded (/lib/systemd/system/heketi.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Fri 2023-01-13 10:23:54 UTC; 288ms ago
      Process: 9227 ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json (code=exited, status=1/FAILURE)
     Main PID: 9227 (code=exited, status=1/FAILURE)
  
  Jan 13 10:23:54 ksmaster-01 systemd[1]: heketi.service: Scheduled restart job, restart counter is at 5.
  Jan 13 10:23:54 ksmaster-01 systemd[1]: Stopped Heketi Server.
  Jan 13 10:23:54 ksmaster-01 systemd[1]: heketi.service: Start request repeated too quickly.
  Jan 13 10:23:54 ksmaster-01 systemd[1]: heketi.service: Failed with result 'exit-code'.
  Jan 13 10:23:54 ksmaster-01 systemd[1]: Failed to start Heketi Server.
  

  root@ksmaster-01:~# /usr/bin/heketi --config=/etc/heketi/heketi.json
  Heketi v7.0.0
  ssh: unhandled key typeUnable to get keyfile
  [cmdexec] ERROR 2023/01/13 10:29:46 /src/github.com/heketi/heketi/executors/sshexec/sshexec.go:124: Unable to read private key file
  [heketi] ERROR 2023/01/13 10:29:46 /src/github.com/heketi/heketi/apps/glusterfs/app.go:114: Unable to read private key file
  ERROR: Unable to start application
  

  解决办法：
  参考：https://github.com/heketi/heketi/issues/1705

  ssh-keygen -m PEM -t rsa -b 4096 -q -f