ceph 基础知识梳理

ceph-deploy部署ceph 

部署前准备

1、系统环境基础调配

关闭防火墙

systemctl stop firewalld systemctl disable firewalld 

关闭网络管理服务

systemctl stop NetworkManager.service  systemctl disable NetworkManager.service

关闭SELinux

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config(重启，setenforce 0 )

配置主机名解析

 /etc/hosts

配置ceph-deploy节点到ceph各节点免密登录

ssh 免密登录 
2、准备repo源
CentOS-Base.repo  ceph-10.2.5.repo(163) epel-7.repo(epel-release源)

3、安装ntp 服务

 yum install ntp -y

编辑配置文件  /etc/ntp.conf 添加以下配置
 restrict 192.168.1.0 255.255.255.0 nomodify notrap
4、创建安装目录
mkdir deploy-ceph
cd deploy-ceph
5、安装ceph-deploy 
yum install ceph-deploy -y

6、创建集群
ceph-deploy new ceph-node1

7、修改配置文件
public_network = 192.168.1.0/24
cluster_network = 18.18.18.0/24
osd_journal_size = 10240
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_crush_chooseleaf_type = 1
osd_crush_update_on_start = true
max_open_files = 131072

8、部署ceph mon
ceph-deploy install --mon --no-adjust-repos ceph-node1

9、初始化ceph mon
ceph-deploy mon create-initial

10、初始化磁盘
ceph-deploy disk list ceph-node1 (查看可用磁盘)

ceph-deploy disk zap ceph-node1:/dev/sdb

11、部署osd
ceph-deploy osd prepare ceph-node1:/dev/sdb 或者ceph-deploy osd active ceph-node1:/dev/sdb1:/dev/sdb2

12、删除存储池
ceph osd pool ls (查看存储池) 
ceph osd pool delete rbd rbd --yes-i-really-really-mean-it

13、清除集群环境

ceph-deploy purge ceph-node1  (卸载软件包)

ceph-deploy purgedate ceph-node1  (清除数据)

ceph-deploy forgetkeys    (删除key)

rm -rf ~/deploy-ceph/*   (删除安装目录残留文件) 

ceph mon扩展

1、部署ceph mon软件包
ceph-deploy install --mon --no-adjust-repos ceph-node2 ceph-node3


2、配置文件修改
添加其他节点主机名、IP地址
mon_initial_members 
mon_host


3、同步配置文件到集群所有节点


ceph-deploy --overwrite-conf admim ceph-node1 ceph-node2 ceph-node3


4、扩展mon节点
ceph-deploy mon add ceph-node2 ceph-node3


5、扩展osd 
ceph-deploy install --osd --no-adjust-repos ceph-node2 ceph-node3


ceph-deploy disk zap ceph-node2:/dev/sdb ceph-node3:/dev/sdb


ceph-deploy osd prepare ceph-node2:/dev/sdb ceph-node3:/dev/sdb

ceph 块存储管理

创建卷
rbd create test --size 10240
查看卷

rbd list

rbd info test

调整卷
rbd resize --size 5120 test --allow-shrink  (缩小卷 )

rbd resize --size 10240 test （扩展卷）

删除卷
rbd rm test

卷快照
rbd snap create test --snap test-snap

查看卷快照
rbd snap ls test

恢复快照
rbd snap rollback test --snap test-snap

删除快照
rbd snap rm test --snap test-snap 

创建format2格式的卷

rbd create test2 --size 10240 --image-format 2

rbd info test2

rbd snap create test2 --snap test2-snap

保护快照
rbd snap protect test2 --snap test2-snap

解除保护
rbd snap unprotect unprotect test2 --snap test2-snap

快照克隆

rbd clone test2 --snap test2-snap test-clone  (只有保护快照状态的可以克隆)

查看
rbd list

rbd info test-clone

导入iso 镜像

rbd import centos7.iso -p rbd

krbd方式映射块设备

只有feature layering开启才开支持krbd 
rbd create rbd/test3 --size 10G --image-foremat 2 --image-feature layering

映射到本地

rbd map rbd/test3

查看映射设备

rbd showmapped

磁盘格式化系统

mkfs.xfs /dev/rbd0

挂载 

mkdir /test3

mount /dev/rbd0 /test3

解除挂载

umount /test3
rbd unmap /dev/rbd/rbd/test3

部署对象存储

ceph-deploy install --no-adjust-repos --rgw ceph-node1


修改配置
【clinet.rdosgw】
host = ceph-node1


同步配置文件
ceph-deploy --overwrite-conf config push ceph-node1


对象网关成功管理节点
ceph-deploy admin ceph-node1
配置文件修改
[client.rgw.Demo-1]
host = Demo-1
rgw_frontends = "civetweb port=8081"


定义zone
{
"name": "default",
  "system_key": {
     "access_key": "",
        "secret_key": ""
    },
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "index_pool",
                "data_pool": "data_pool",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0
            }
        }
    ],
    "metadata_heap": "",
    "realm_id": ""
}


radosgw-admin zone set --rgw-zone=default < zone.info
radosgw-admin zone get --rgw-zone=default
创建pool
ceph osd pool create data_pool 64
ceph osd pool create index_pool 64


部署网关
ceph-deploy rgw create ceph-node1
检查服务
systemctl status ceph-radosgw@rgw.ceph-node1
验证
curl ip:7480


创建用户
radosgw-admin user create --uid=test --display-name="test" --access-key=123456 --secret=123456