ceph集群

一、ceph部署

一、环境准备

备注：local需要修改

localhost	IP
ceph01	10.10.10.121
ceph02	10.10.10.122
ceph03	10.10.10.123
ceph04	10.10.10.124

Python3.6.5(3.7及以上的存在库问题，暂时不清楚解决方法)
每台服务器需要添加一块单独的硬盘（/dev/sdb）

二、ceph-deploy方式部署

安装ceph版本：nautilus

1、环境准备（三个集群节点一起执行）

(1) 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
(2) 关闭selinux
sed -i ‘s/enforcing/disabled/’ /etc/selinux/config
setenforce 0
(3) 关闭NetworkManager
systemctl disable NetworkManager && systemctl stop NetworkManager
(4) 添加主机名与IP对应关系

10.10.10.121 ceph01
10.10.10.122 ceph02
10.10.10.123 ceph03
10.10.10.124 ceph04

(5) 设置主机名
hostnamectl set-hostname ceph01
hostnamectl set-hostname ceph02
hostnamectl set-hostname ceph03
hostnamectl set-hostname ceph04
(6) 设置时间
systemctl restart chronyd.service && systemctl enable chronyd.service
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
(7) 设置文件描述符

echo "ulimit -SHn 102400" >> /etc/rc.local
cat >> /etc/security/limits.conf << EOF
* soft nofile 65535
* hard nofile 65535
EOF

(8) 内核参数优化

cat >> /etc/sysctl.conf << EOF
kernel.pid_max = 4194303
vm.swappiness = 0
EOF

sysctl -p
(9) 设置免密登录
ssh-copy-id ceph0*

2、配置YUM（三个集群节点一起执行）

(1) 编辑yum文件

[root@ceph01 my-cluster]# cat /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.163.com/ceph/rpm-nautilus/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc

(2) 下载ceph安装包
yum --downloadonly --downloaddir=/var/www/html/ceph/rpm-nautilus/el7/x86_64/ install ceph ceph-radosgw
createrepo --update /var/www/html/ceph/rpm-nautilus

3、安装

(1) 安装ceph-deploy

yum install -y ceph-deploy

(2) 创建my-cluster目录

mkdir /my-cluster && cd /my-cluster

(3) 创建ceph集群

ceph-deploy new ceph01 ceph02 ceph03 

(4) 安装ceph(每个节点都执行)

 yum -y install epel-release
 yum install -y ceph

(5) 生成monitor检测集群所使用的的秘钥

ceph-deploy mon create-initial

(6) 安装Ceph CLI，方便执行一些管理命令

ceph-deploy admin ceph01 ceph02 ceph03

(7) 配置mgr，用于管理集群

ceph-deploy mgr create ceph01 ceph02 ceph03

(8) 部署rgw

yum install -y ceph-radosgw
ceph-deploy rgw create ceph01

(9) 部署MDS（CephFS）

ceph-deploy mds create ceph01 ceph02 ceph03

(10) 添加osd

ceph-deploy osd create --data /dev/sdb ceph01
ceph-deploy osd create --data /dev/sdb ceph02
ceph-deploy osd create --data /dev/sdb ceph03
ceph-deploy osd create --data /dev/sdc ceph01
ceph-deploy osd create --data /dev/sdd ceph01

4、ceph.conf

后续补充
基本配置

[global]  #全局
fsid = 8c0d4bd8-db28-4277-91d8-d264599e0c6a #集群标识ID
mon_initial_members = ceph01, ceph02, ceph03  
mon_host = 10.10.10.121,10.10.10.122,10.10.10.123

# 集群认证
auth_cluster_required = cephx  
auth_service_required = cephx
auth_client_required = cephx

osd_pool_default_size = 2 #最小副本数
mon_max_pg_per_osd = 2000 
[osd]
[mon]
[client]