1.Ceph部署
1.1 Ceph 安装前准备,环境与拓扑图如下:
主机名 | 外网IP | 内网IP | 安装组件 | 磁盘块 |
c720181 | 172.22.69.84 | 192.168.20.181 | ceph-deploy osd | sdb |
c720182 | 172.22.69.85 | 192.168.20.182 | mon osd | sdb |
c720183 | 172.22.69.86 | 192.168.20.183 | mon osd | sdb |
c720184 | 172.22.69.95 | 192.168.20.184 | cli |
|
============每台主机都需要配置============
配置主机名,IP地址(略)
关闭防护墙和selinux(略)
配置主机名解析,使用/etc/hosts
cat >>/etc/hosts>>EOF
192.168.20.181 c720181
192.168.20.182 c720182
192.168.20.183 c720183
192.168.20.184 c720184
1.2 部署安装
安装方式如下:
官网安装方式:https://docs.ceph.com/docs/master/start/
Ansilbe安装方式:https://docs.ceph.com/ceph-ansible/master/
使用官方推荐的ceph-deploy部署集群:
============每台主机都需要配置============
配置环境变量:
export username="ceph-admin"
export passwd="ceph-admin"
export node1="c720181"
export node2="c720182"
export node3="c720183"
export node4="c720184"
export node1_ip="192.168.20.181"
export node2_ip="192.168.20.182"
export node3_ip="192.168.20.183"
export node4_ip="192.168.20.184"
配置rpm
wget -O /etc/yum.repos.d/ceph.repo https://raw.githubusercontent.com/aishangwei/ceph-demo/master/ceph-deploy/ceph.repo
或者直接在/etc/yum/repos.d/目录下,创建ceph.repo,添加内容如下:
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/
gpgcheck=0
priority=1
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/
gpgcheck=0
priority=1
[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS
enabled=0
gpgcheck=1
type=rpm-md
gpgkey=http://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
配置NTP
yum -y install ntpdate ntp
ntpdate cn.ntp.org.cn
systemctl restart ntpd ntpdate && systemctl enable ntpd ntpdate
创建部署用户和ssh免密码登陆
useradd ${username}
echo "${passwd}" | passwd --stdin ${username}
echo "${username} ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/${username}
chmod 0440 /etc/sudoers.d/${username}
配置sudo 不需要tty
sed -i 's/Default requiretty/#Default requiretty/' /etc/sudoers
============只在管理节点192.168.20.181上配置============
su - ceph-admin
export username=ceph-admin
ssh-keygen
ssh-copy-id ${username}@c720181
ssh-copy-id ${username}@c720182
ssh-copy-id ${username}@c720183
ssh-copy-id ${username}@c720184
1.2.1 安装 ceph-deploy
su - ceph-admin
sudo yum install -y ceph-deploy python-pip
如果提示没有python-pip安装包,则切换到root用户安装epel-release,然后再尝试安装python-pip
1.2.2 部署节点
su - ceph-admin
mkdir my-cluster
cd my-cluster
ceph-deploy new c720181 c720182 c720183
编辑ceph.conf配置文件
cat ceph.conf
[global]
……
public network = 172.22.69.0/24
cluster network = 192.168.20.0/24
============每台主机都需要配置============
安装ceph包,替代再管理端使用ceph-deploy install node1 node2 node3,因为使用ceph-deploy安装的话通过官网yum源去安装,这样会很慢而且容易报错
yum install -y ceph ceph-radosgw
如果有遇到上述报错的话,是因为yum源没有相关依赖包,则切换到root用户安装epel-release,然后再尝试安装 yum install -y ceph ceph-radosgw
============只在管理节点192.168.20.181上配置============
配置初始化monitor(s)、并收集所有密钥
su - ceph-admin
cd my-cluster
ceph-deploy mon create-initial
如果出现上述报错:
解决办法:在各个节点上执行sudo pkill ceph,然后再在deploy节点执行ceph-deploy mon create-initial
ls -l *.keyring
然后发现ERROR报错消失了,配置初始monitor(s)、并收集到了所有密钥,当前目录下可以看到下面这些密钥环
ceph.bootstrap-mds.keyring
ceph.bootstrap-mgr.keyring
ceph.bootstrap-osd.keyring
ceph.bootstrap-rgw.keyring
ceph.client.admin.keyring
把配置信息拷贝到各节点
ceph-deploy admin c720181 c720182 c720183
配置osd
for dev in /dev/sdb
do
ceph-deploy disk zap c720181 $dev
ceph-deploy osd create c720181 --data $dev
ceph-deploy disk zap c720182 $dev
ceph-deploy osd create c720182 --data $dev
ceph-deploy disk zap c720183 $dev
ceph-deploy osd create c720183 --data $dev
done
如果有多个硬盘,可以将
for dev in /dev/sdb 改为
for dev in /dev/sdb, /dev/sdc, /dev/sdd
部署mgr(新版本监控UI, L版以后才需要部署)
ceph-deploy mgr create c720181 c720182 c720183
开启dashboard模块,用于UI查看
ceph mgr module enable dashboard
上述报错是因为/etc/ceph 目录权限是root,而不是ceph-admin
所以用所属用户权限改成ceph-admin即可:sudo chown -R ceph-admin /etc/ceph/
再执行启用dashboard模块命令:ceph mgr module enable dashboard
访问监控面板地址:http://192.168.20.181:7000/servers
问题处理:
1、 访问速度慢,输入链接访问,退出lab非常慢,通过F12发现访问跳转到fonts.googleapis.com CSS文件特别久。
解决办法:取消访问http://fonts.googleapis.com 站点的CSS静态文件。
路径:/usr/lib64/ceph/mgr/dashboard/static/AdminLTE-2.3.7/dist/css
需改下面两个文件,注释掉访问http://fonts.googleapis.com的行
vim AdminLTE.min.css
# @import url(https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,400,600,700,300italic,400italic,600italic);/*!
* AdminLTE v2.3.7
* Author: Almsaeed Studio
* Website: Almsaeed Studio <http://almsaeedstudio.com>
* License: Open source - MIT
* Please visit http://opensource.org/licenses/MIT for more information
重启服务:
[ceph-admin@c720181 my-cluster]$ ceph mgr module disable dashboard
[ceph-admin@c720181 my-cluster]$ ceph mgr module enable dashboard