centos-7.4_ceph-12.2.4部署:
前言:
基于centos7.4安装ceph-luminous的主要步骤有一下几点:
1、安装centos7.4的系统,并配置网卡
2、安装前的环境配置
3、安装ceph、配置mon、mgr、osd等组件创建pool,即完成了ceph的环境安装和部署
安装系统并配置网卡,确认系统可以上外网,或者添加本地安装源:
安装前的环境配置
修改主机名:
# hostnamectl set-hostname $HOSTNAME(主机名)
# yum -y install vim
添加本地解析:
# vim /etc/hosts
1.1.1.12 $HOSTNAME
添加epel源:
# vim /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
添加ceph-12源:
# vim /etc/yum.repos.d/ceph.repo
[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/
gpgcheck=0
加速ssh:
# vim /etc/ssh/sshd_config
GSSAPIAuthentication no
UseDNS no
# systemctl restart sshd
修改selinux策略:
# vim /etc/selinux/config
SELINUX=disabled
setenforce 0
systemctl stop NetworkManager.service
systemctl disable NetworkManager.service
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl stop postfix.service
systemctl disable postfix.service
安装和部署:
安装ceph:
查询有那些可安装的包
yum --showduplicates list ceph | expand
安装ceph-12.2.4
# vim /etc/yum.conf
exclude=*12.2.7* *12.2.6* *12.2.5*(此处是限制不安装哪些版本,默认安装的是最新版本)
安装:
# yum -y install ceph
添加ceph配置文件:
# cd /etc/ceph
# vim ceph.conf
[global]
fsid = a7f64266-0894-4f1e-a635-d0aeaca0e993
mon initial members = $HOSTNAME(mon进程名)
mon host = 1.1.1.12
public network = 1.1.1.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1
mon osd max split count = 10000
mon max pg per osd = 10000
rbd_default_features = 1
配置mon服务:
# sudo -u ceph ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
# ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd'
# ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
# ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
# monmaptool --create --add $HOSTNAME 1.1.1.12 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /etc/ceph/monmap
# sudo -u ceph mkdir /var/lib/ceph/mon/ceph-$HOSTNAME
# sudo -u ceph ceph-mon --mkfs -i $HOSTNAME --monmap /etc/ceph/monmap --keyring /tmp/ceph.mon.keyring
# ll /var/lib/ceph/mon/ceph-$HOSTNAME/
total 8
-rw------- 1 ceph ceph 77 May 20 05:26 keyring
-rw-r--r-- 1 ceph ceph 8 May 20 05:26 kv_backend
drwxr-xr-x 2 ceph ceph 112 May 20 05:26 store.db
# touch /var/lib/ceph/mon/ceph-$HOSTNAME/{done,upstart}
# systemctl enable ceph.target
# systemctl enable ceph-mon.target
# systemctl enable ceph-mon@$HOSTNAME
# systemctl start ceph-mon@$HOSTNAME
# tail -f /var/log/messages
May 20 05:31:52 ceph systemd: Started Ceph cluster monitor daemon.
May 20 05:31:52 ceph systemd: Starting Ceph cluster monitor daemon... #说明已经成功启动
# ll /var/run/ceph/
total 0
srwxr-xr-x 1 ceph ceph 0 May 20 05:31 ceph-mon.$HOSTNAME.asok
# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.$HOSTNAME.asok mon_status #查看mon的详细信息
配置mgr服务:
# sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-$HOSTNAME
# ceph auth get-or-create "mgr.$HOSTNAME" mon 'allow profile mgr' osd 'allow *' mds 'allow *' -o /var/lib/ceph/mgr/ceph-$HOSTNAME/keyring
# chown -R ceph:ceph /var/lib/ceph/mgr/ceph-$HOSTNAME/keyring
# systemctl enable ceph-mgr.target
# systemctl enable ceph-mgr@$HOSTNAME
# systemctl start ceph-mgr@$HOSTNAME.service
# tail -f /var/log/messages
May 20 05:42:25 ceph systemd: Started Ceph cluster manager daemon.
May 20 05:42:25 ceph systemd: Starting Ceph cluster manager daemon... #说明已经成功启动
配置osd服务(在新的版本中部署osd的方法有多种,一下一一列举):
以下部署过程中,出现问题可能会用到的命令:
清除分区,重新格式化
dd if=/dev/zero of=/dev/sdb bs=1M count=10
parted /dev/sdb -s mklabel gpt
虚拟机不关机让系统识别磁盘:
echo "- - -" > /sys/class/scsi_host/host{0..2}/scan
刷新配置:
systemctl daemon-reload
-----------------------------------------------------------------------------------------
首先第一个:(这个沿用之前的部署方式,将整个盘作为一个分区,性能和工作方式有待验证)
# vim create_osd.sh
#!/bin/bash
disk=$1
UUID=$(uuidgen)
OSD_SECRET=$(ceph-authtool --gen-print-key)
ID=$(echo "{\"cephx_secret\": \"$OSD_SECRET\"}" | \
ceph osd new $UUID -i - \
-n client.bootstrap-osd -k /var/lib/ceph/bootstrap-osd/ceph.keyring)
mkdir /var/lib/ceph/osd/ceph-$ID
mkfs.xfs /dev/${disk}
mount /dev/${disk} /var/lib/ceph/osd/ceph-$ID
ceph-authtool --create-keyring /var/lib/ceph/osd/ceph-$ID/keyring \
--name osd.$ID --add-key $OSD_SECRET
ceph-osd -i $ID --mkfs --osd-uuid $UUID
chown -R ceph:ceph /var/lib/ceph/osd/ceph-$ID
systemctl enable ceph-osd@$ID
systemctl start ceph-osd@$ID
--------------------------------------
第二种是基于ceph-disk部署的bluestore,会给磁盘分四个区,可以在配置文件里面定义每个分区的大小:(默认是filestore)
ceph-disk prepare --bluestore /dev/sdc --block.db /dev/sdc --block.wal /dev/sdc
bluestore主要配置项有:
[global]
bluestore block create = true
bluestore block db size = 67108864
bluestore block db create = true
bluestore block wal size = 134217728
bluestore block wal create = true
[osd]
enable experimental unrecoverable data corrupting features = bluestore rocksdb
osd objectstore = bluestore
bluestore fsck on mount = true
[mon.$HOSTNAME]
host = $HOSTNAME
mon addr = 1.1.1.12:6789
[mgr]
mgr data = /var/lib/ceph/mgr/$cluster-$name
[mgr.$HOSTNAME]
key = "AQBcHf9aOJyoGBAAGgujgi67SBrImyrTaHy3vw=="
caps mds = "allow *"
caps mon = "allow profile mgr"
caps osd = "allow *"
[osd.0]
host = $HOSTNAME
osd data = /var/lib/ceph/osd/ceph-0
以下内容,可以不定义,或者在部署完成后进行添加
# bluestore block db path = /dev/disk/by-partlabel/osd-device-0-block
# bluestore block wal path = /dev/disk/by-partlabel/osd-device-0-db
# bluestore block path = /dev/disk/by-partlabel/osd-device-0-wal
------------------------------------------
第三种方法是当下官方比较推荐的ceph-volume工具,结合lvm使用,默认部署bluestore:
ceph-volume lvm create --data /dev/sdb
or:
ceph-volume lvm prepare --data /dev/sdb
ceph-volume lvm list
ceph-volume lvm activate {ID} {FSID}
filestore:
ceph-volume lvm create --filestore --data /dev/sdb --journal /dev/sdb
or:
ceph-volume lvm prepare --filestore --data {data-path} --journal {journal-path}
ceph-volume lvm list
ceph-volume lvm activate --filestore {ID} {FSID}
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
部署完之后,使用:
ceph常用操作命令:
systemctl stop ceph\*.service ceph\*.target (关闭集群)
systemctl start ceph.target (开启集群)
systemctl start ceph-osd@{0..2} (关闭一组osd)
systemctl stop ceph-osd.target (开启当前服务器上面的osd)
ceph osd pool create volumes 2048 2048 replicated (创建存储池)
使用过程中的常见报错:
# ceph osd df 报错原因, 没有授权
Error EACCES: access denied' does your client key have mgr caps? See http://docs.ceph.com/docs/master/mgr/administrator/#client-authentication
# ceph auth caps client.admin osd 'allow *' mds 'allow ' mon 'allow *' mgr 'allow *'
updated caps for client.admin
ceph的副本数不符合要求:
# ceph -s
cluster:
id: 46c8f0f7-c585-4e32-99b5-aaa5d50cb703
health: HEALTH_WARN
Reduced data availability: 2048 pgs inactive
Degraded data redundancy: 2048 pgs undersized
services:
mon: 1 daemons, quorum ceph128-06
mgr: ceph128-06(active)
osd: 6 osds: 6 up, 6 in
data:
pools: 3 pools, 2560 pgs
objects: 0 objects, 0 bytes
usage: 6384 MB used, 8931 GB / 8938 GB avail
pgs: 100.000% pgs not active
2560 undersized+peered
# ceph osd dump|grep pool
pool 1 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 29 flags hashpspool stripe_width 0
pool 2 'backups' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 34 flags hashpspool stripe_width 0
pool 3 'images' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 37 flags hashpspool stripe_width 0
# for i in `ceph osd pool ls`;do ceph osd pool set $i size 1&&ceph osd pool set $i min_size 1;done (请设置合理的副本数)
没有对创建完的pool进行定义:
# ceph -s
cluster:
id: 46c8f0f7-c585-4e32-99b5-aaa5d50cb703
health: HEALTH_WARN
application not enabled on 3 pool(s)
services:
mon: 1 daemons, quorum ceph128-06
mgr: ceph128-06(active)
osd: 6 osds: 6 up, 6 in
data:
pools: 3 pools, 2560 pgs
objects: 15 objects, 192 kB
usage: 6417 MB used, 8931 GB / 8938 GB avail
pgs: 2560 active+clean
io:
client: 383 B/s rd, 8685 B/s wr, 0 op/s rd, 0 op/s wr
# for i in `ceph osd pool ls`;do ceph osd pool application enable $i rbd;done
# for i in `ceph osd pool ls`;do rbd pool init $i;done
创建块时特性的报错:(如在配置文件中定义则不会出现这样的报错)
# rbd create -s 10T volumes/test
# rbd info volumes/test
rbd image 'test':
size 10240 GB in 2621440 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.375674b0dc51
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
create_timestamp: Sat Jul 28 03:02:36 2018
# rbd map volumes/test
rbd: sysfs write failed
RBD image feature set mismatch. Try disabling features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
# vim /etc/ceph/ceph.conf
rbd_default_features = 1
# rbd rm volumes/test
# rbd create -s 10T volumes/test
# rbd info volumes/test
rbd image 'test':
size 10240 GB in 2621440 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.375b74b0dc51
format: 2
features: layering
flags:
create_timestamp: Sat Jul 28 03:08:36 2018