beegfs环境搭建

beegfs环境搭建

1、模拟生产环境搭建同样的测试环境

规划:

192.168.20.129–>108–>beegfs-mgmtd manager metadata
192.168.20.130–>109–>beegfs-client metadata client helperd nfs-server
192.168.20.198–>110–>beegfs-storage01 beegfs-storage
192.168.20.199–>111–>beegfs-storage02 beegfs-storage

模拟环境IP生产环境IP主机名角色
192.168.20.129108beegfs-mgmtdmanager admon
192.168.20.130109beegfs-clientmetadata client helperd nfs-server
192.168.20.198110beegfs-storage01beegfs-storage
192.168.20.199111beegfs-storage02beegfs-storage

每台主机额外添加一块磁盘,用于搭建beegfs

每台主机额外添加一块网卡,用于模拟IB网络

主机名需要使用两种 :一个业务网络,一个IB网络

[root@beegfs-mgmtd ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.20.129 beegfs-mgmtd     
192.168.20.130 beegfs-client    
192.168.20.198 beegfs-storage01 
192.168.20.199 beegfs-storage02 

192.168.1.10 beegfs-mgmtd-ib    
192.168.1.11 beegfs-client-ib
192.168.1.12 beegfs-storage01-ib 
192.168.1.13 beegfs-storage02-ib
[root@beegfs-mgmtd ~]

2、服务安装

a、安装Management Server(管理服务)

[root@beegfs-mgmtd /]# yum install -y beegfs-mgmtd

[root@beegfs-mgmtd /]# yum install -y beegfs-meta

b、安装Metadata Server(元数据服务)、Client and Command-line Utils(客户端和命令行实用程序)nfs

[root@beegfs-client /]# yum install -y beegfs-meta
[root@beegfs-client /]# yum install -y beegfs-client beegfs-helperd beegfs-utils nfs-utlis

c、安装Storage Server(存储服务)

[root@beegfs-storage01 /]# yum install -y beegfs-storage
[root@beegfs-storage02 /]# yum install -y beegfs-storage

3、基本配置

a、配置Management Service

[root@beegfs-mgmtd /]# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs/
Preparing storage directory: /beegfs/
 * Creating format.conf file...
Updating config file: /etc/beegfs/beegfs-mgmtd.conf
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
All done.

b、配置Metadata Server、Client

[root@beegfs-client /]# /opt/beegfs/sbin/beegfs-setup-meta -p /metadata/beegfs/beegfs_meta -s 1 -m beegfs-mgmtd-ib
Preparing storage directory: /metadata/beegfs/beegfs_meta
 * Creating format.conf file...
 * Creating server numeric ID file: /metadata/beegfs/beegfs_meta/nodeNumID
Updating config file: /etc/beegfs/beegfs-meta.conf
 * Setting management host: beegfs-mgmtd
 * Setting storage directory in config file...
 * Disabling usage of uninitialized storage directory in config file...
 * Setting usage of extended attributes to: true
All done.
[root@beegfs-client /]# /opt/beegfs/sbin/beegfs-setup-client -m beegfs-mgmtd
Updating config file: /etc/beegfs/beegfs-client.conf
 * Setting management host: beegfs-mgmtd
All done.

(BeeGFS默认会挂载到/mnt/beegfs,可以自行在配置文件/etc/beegfs/beegfs-mounts.conf中修改)
我们生产环境是/data目录,需要修改

c、配置Storage Server

注意每个存储节点对应Storage Service/Target ID都不一样。

[root@beegfs-storage01 /]# /opt/beegfs/sbin/beegfs-setup-storage -p /storage1 -s 1 -i 101 -m beegfs-mgmtd-ib
Preparing storage target directory: /storage1/
 * Creating format.conf file...
 * Creating chunks directory...
 * Creating buddymir directory...
 * Creating target numeric ID file: /storage1/targetNumID
 * Creating server numeric ID file: /storage1/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
 * Setting management host: beegfs-mgmtd
 * Appending to target directory list in config file...
 * Disabling usage of uninitialized storage targets in config file...
All done.

[root@beegfs-storage02 /]# /opt/beegfs/sbin/beegfs-setup-storage -p storage2/beegfs_storage -s 2 -i 104 -m beegfs-mgmtd-ib
Preparing storage target directory: /storage2/
 * Creating format.conf file...
 * Creating chunks directory...
 * Creating buddymir directory...
 * Creating target numeric ID file: /storage2/targetNumID
 * Creating server numeric ID file: /storage2/nodeNumID
Updating config file: /etc/beegfs/beegfs-storage.conf
 * Setting management host: beegfs-mgmtd
 * Appending to target directory list in config file...
 * Disabling usage of uninitialized storage targets in config file...
All done.

4、启动访问

Client服务端需要最后启动,需要先把存储节点服务启动,在启动Client服务,否则Client启动失败。

Client服务主要用于挂载Beegfs存储

1、启动Management Service服务
[root@beegfs-mgmtd /]# systemctl start beegfs-mgmtd
[root@beegfs-mgmtd /]# systemctl status beegfs-mgmtd
鈼[0m beegfs-mgmtd.service - BeeGFS Management Server
   Loaded: loaded (/usr/lib/systemd/system/beegfs-mgmtd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-30 12:24:49 CST; 7s ago
     Docs: http://www.beegfs.com/content/documentation/
 Main PID: 7461 (beegfs-mgmtd/Ma)
   CGroup: /system.slice/beegfs-mgmtd.service
           鈹斺攢7461 /opt/beegfs/sbin/beegfs-mgmtd cfgFile=/etc/beegfs/beegfs-mgmtd.conf runDaemonized=false

Nov 30 12:24:49 beegfs-mgmtd systemd[1]: Started BeeGFS Management Server.

2、启动Metadata Service服务
[root@beegfs-client /]# systemctl start beegfs-meta
[root@beegfs-client ~]# systemctl status beegfs-meta
鈼[0m beegfs-meta.service - BeeGFS Metadata Server
   Loaded: loaded (/usr/lib/systemd/system/beegfs-meta.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-30 15:16:33 CST; 2h 42min ago
     Docs: http://www.beegfs.com/content/documentation/
 Main PID: 6602 (beegfs-meta/Mai)
   CGroup: /system.slice/beegfs-meta.service
           鈹斺攢6602 /opt/beegfs/sbin/beegfs-meta cfgFile=/etc/beegfs/beegfs-meta.conf runDaemonized=false

Nov 30 15:16:33 beegfs-client systemd[1]: Started BeeGFS Metadata Server.

3、启动Storage Service服务
[root@beegfs-storage01 storage1]# systemctl start beegfs-storage
[root@beegfs-storage01 storage1]# systemctl status beegfs-storage
鈼[0m beegfs-storage.service - BeeGFS Storage Server
   Loaded: loaded (/usr/lib/systemd/system/beegfs-storage.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-30 13:19:16 CST; 1s ago
     Docs: http://www.beegfs.com/content/documentation/
 Main PID: 7616 (beegfs-storage/)
   CGroup: /system.slice/beegfs-storage.service
           鈹斺攢7616 /opt/beegfs/sbin/beegfs-storage cfgFile=/etc/beegfs/beegfs-storage.conf runDaemonized=false

Nov 30 13:19:16 beegfs-storage01 systemd[1]: Started BeeGFS Storage Server.

[root@beegfs-storage02 storage2]#  systemctl enable beegfs-storage
[root@beegfs-storage02 storage2]# systemctl status beegfs-storage
鈼[0m beegfs-storage.service - BeeGFS Storage Server
   Loaded: loaded (/usr/lib/systemd/system/beegfs-storage.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-30 13:19:57 CST; 33min ago
     Docs: http://www.beegfs.com/content/documentation/
 Main PID: 7611 (beegfs-storage/)
   CGroup: /system.slice/beegfs-storage.service
           鈹斺攢7611 /opt/beegfs/sbin/beegfs-storage cfgFile=/etc/beegfs/beegfs-storage.conf runDaemonized=false

Nov 30 13:19:57 beegfs-storage02 systemd[1]: Started BeeGFS Storage Server.

4、启动Client服务
[root@beegfs-client ~]# systemctl start beegfs-client
[root@beegfs-client ~]# systemctl status beegfs-client
鈼[0m beegfs-client.service - Start BeeGFS Client
   Loaded: loaded (/usr/lib/systemd/system/beegfs-client.service; enabled; vendor preset: disabled)
   Active: active (exited) since Tue 2021-11-30 15:24:36 CST; 2h 33min ago
 Main PID: 41551 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/beegfs-client.service

Nov 30 15:24:13 beegfs-client systemd[1]: Starting Start BeeGFS Client...
Nov 30 15:24:13 beegfs-client beegfs-client[41551]: Starting BeeGFS Client:
Nov 30 15:24:13 beegfs-client beegfs-client[41551]: - Loading BeeGFS modules
Nov 30 15:24:13 beegfs-client beegfs-client[41551]: - BeeGFS module autobuild
Nov 30 15:24:14 beegfs-client beegfs-client[41551]: Building beegfs client module
Nov 30 15:24:36 beegfs-client beegfs-client[41551]: - Mounting directories from /etc/beegfs/beegfs-mounts.conf
Nov 30 15:24:36 beegfs-client systemd[1]: Started Start BeeGFS Client.

5、安装beegfs-admon GUI管理工具(非必须安装)

GUI管理前提条件:

(1)需要安装beegfs-Admon服务并运行Admon服务

(2)GUI管理Client需要安装Java程序,GUI管理是通过java程序调用、下载GUI管理程序并运行(3)通过浏览器访问BeeGFS-Admon服务器http://ip:8000端口即可下载GUI管理程序

(4)点击运行BeeGFS Admon GUI程序,输入BeeGFS Admon ip地址即可管理。

[root@beegfs-mgmtd /]# yum install -y beegfs-admon
[root@beegfs-mgmtd /]# systemctl start beegfs-admon
[root@beegfs-mgmtd /]# systemctl status beegfs-admon
鈼[0m beegfs-admon.service - Start BeeGFS Admon
   Loaded: loaded (/usr/lib/systemd/system/beegfs-admon.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-11-30 14:22:20 CST; 13s ago
     Docs: http://www.beegfs.com/content/documentation/
 Main PID: 7840 (beegfs-admon/Ma)
   CGroup: /system.slice/beegfs-admon.service
           鈹斺攢7840 /opt/beegfs/sbin/beegfs-admon cfgFile=/etc/beegfs/beegfs-admon.conf runDaemonized=false

Nov 30 14:22:20 beegfs-mgmtd systemd[1]: Started Start BeeGFS Admon.
[root@beegfs-mgmtd /]# ss -lunpt |grep 8000
tcp    LISTEN     0      128       *:8000                  *:*                   users:(("beegfs-admon/Ma",pid=7840,fd=8))

6、配置admon

修改配置文件

[root@beegfs-mgmtd ~]# cat /etc/beegfs/beegfs-admon.conf 
sysMgmtdHost                 =beegfs-mgmtd
[root@beegfs-mgmtd ~]# systemctl start beegfs-admon
[root@beegfs-mgmtd ~]# systemctl enable beegfs-admon

(1)、通过浏览器访问BeeGFS-Admon服务器http://ip:8000端口即可下载GUI管理程序

(2)、点击运行BeeGFS Admon GUI程序,输入BeeGFS Admon ip地址即可管理。

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-1x2zAbIy-1666774331323)
在这里插入图片描述

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gW89FC9t-1666774331324)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%201.png)]
在这里插入图片描述

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-OcN2TnOC-1666774331325)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%202.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-aBj4yWKm-1666774331325)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%203.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JmX5sPEV-1666774331326)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%204.png)]

排错:

Configure and startConfiguration fileClient serviceStart beegfs-client

/opt/beegfs/sbin/beegfs-setup-clientFor example# /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs/storage/ -m 10.100.0.XPreparing storage target directory: /beegfs/storage/ * Creating format.conf file… * Creating chunks directory… * Creating buddymir directory…Updating config file: /etc/beegfs/beegfs-storage.conf * Setting management host: 10.100.0.X * Appending to target directory list in config file… * Disabling usage of uninitialized storage targets in config file…All done.# systemctl start beegfs-storage# systemctl status beegfs-storage# file /etc/beegfs/beegfs-storage.conf/etc/beegfs/beegfs-storage.conf: ASCII text# /opt/beegfs/sbin/beegfs-setup-client -m 10.100.0.XUpdating config file: /etc/beegfs/beegfs-client.conf * Setting management host: 10.100.0.XAll done.# systemctl start beegfs-clientJob for beegfs-client.service failed because the control process exited with error code. See “systemctl status beegfs-client.service” and “journalctl -xe” for details.# journalctl --unit beegfs-clientsystemd[1]: Starting Start BeeGFS Client…beegfs-client[16857]: Starting BeeGFS Client:beegfs-client[16857]: - Loading BeeGFS modulesbeegfs-client[16857]: - BeeGFS module autobuildbeegfs-client[16857]: Makefile:157: *** Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers).systemd[1]: beegfs-client.service: main process exited, code=exited, status=1/FAILUREsystemd[1]: Failed to start Start BeeGFS Client.systemd[1]: Unit beegfs-client.service entered failed state.systemd[1]: beegfs-client.service failed.beegfs-client[16857]: make: *** [auto_rebuild] Error 2

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jHOdYouP-1666774331327)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%206.png)]

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-2FZ96Mah-1666774331327)(beegfs%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA%20c483019c7c84451696828fd6b4025d19/Untitled%207.png)]

What’s wrong?

beegfs kernel module is missing

lsmod

/etc/init.d/beegfs-client

Install what’s needed to compile the beegfs kernel module

Configuration files

Beegfs module with In

https://www.itread01.com/content/1560424804.html

https://stackoverflow.com/questions/67177487/getting-following-error-while-executing-beegfs-beeond-start-command-on-centos-8

akefile:124: *** Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers).  Stop.
make: *** [AutoRebuild.mk:34: auto_rebuild] Error 2

需要安装kernel 和gcc 要注意是和uname -r里面是同样的版本

https://blogs.oracle.com/cloud-infrastructure/post/140-gb-per-second-with-beegfs-and-oracle-cloud 编译参数

  • 0
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值