前言-最近使用OSS比较多,想着自己搭一套,搜了半天选了ceph,其中的坑也是比较多,所以记录下来,让小伙伴们踩坑少一点。并且我这里是以单机版为例.
注意:如果你的服务器是一个裸机那么必然格式化磁盘并且创建文件系统,这里我建议你使用xfs文件系统,因为如果用ext4后面会出现问题,如果用你已经用了ext4也没关系,请看文章末尾。
一、搭建ceph本地源
1、创建好目录并下载软件包------注意如果网络不行,直接通过url下载在scp过去
mkdir-p /var/www/html/ceph/10.2.2
cd /var/www/html/ceph/10.2.2
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-base-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-common-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-devel-compat-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-fuse-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-libs-compat-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-mds-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-mon-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-osd-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-radosgw-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-selinux-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ceph-test-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/cephfs-java-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libcephfs1-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libcephfs1-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libcephfs_jni1-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libcephfs_jni1-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librados2-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librados2-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libradosstriper1-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/libradosstriper1-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librbd1-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librbd1-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librgw2-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/librgw2-devel-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/python-ceph-compat-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/python-cephfs-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/python-rados-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/python-rbd-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/rbd-fuse-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/rbd-mirror-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/rbd-nbd-10.2.2-0.el7.x86_64.rpm
wget http://mirrors.aliyun.com/ceph/rpm-hammer/el7/noarch/ceph-deploy-1.5.36-0.noarch.rpm
2、搭建本地源----先安装yum install createrepo -y
createrepo /var/www/html/ceph/10.2.2
3、查看一下
cat /etc/yum.repos.d/ceph.repo
出现下列文字表示成功了:
[cloudera-manager]
name = ceph
baseurl = http://你的ip/ceph/10.2.2
gpgcheck = 0
4、安装httpd并开启服务
yum install httpd -y
systemctl start httpd
5、安装ceph和对象网关(ceph-radosgw)
yum clean all
yum makecache
yum install ceph ceph-radosgw ntp -y
6、检查
查看ceph版本:
ceph -v
查看ceph健康状况:
ceph -s
7、安装ceph-deploy
注意:我们这里使用ceph-deploy部署ceph,这里有一个坑,ceph-deploy的版本不能是2.*.*版本,因为后面会出现osd挂载问题,这里我用的是1.5.*的版本。
安装ceph-deploy:
第一种方法:
yum -y install ceph-deploy
第二种方法:
使用python的pip安装
yum install pip(如果无法使用yum安装pip,请看文章末尾,使用手动安装)
pip install ceph-deploy==1.5.39
注意:如果版本安装错误我建议使用pip卸载
如:pip uninstall ceph-deploy
8、部署ceph
创建一个文件夹:后续的操作都在里面执行
mkdir /root/myceph
cd /root/myceph
ceph-deploy new 你的主机名
上述操作之后在当前文件夹下会生成几个文件,如ceph.conf.
如果你和我一样是单节点集群请在ceph.conf文件末尾添加:
osd pool default size = 1
osd pool default min size = 1
9、部署monitor
ceph-deploy mon create-initial
10、查看集群状态
ceph -s 肯定是不健康的因为我们要挂在osd
11、挂载osd
如果磁盘不会格式化的同学参考这篇文章:
https://help.aliyun.com/document_detail/25426.html
我的服务器有四块磁盘,文件系统是ext4,并且只有一个分区如下;
/dev/vdb1 5.4T 5.1G 5.1T 1% /data01
/dev/vdc1 5.4T 5.1G 5.1T 1% /data02
/dev/vdd1 5.4T 5.1G 5.1T 1% /data03
/dev/vde1 5.4T 89M 5.1T 1% /data04
我挂了三块osd分别挂载在 /data01、/data02、/data03
授权:
chown ceph:ceph /data02
chown ceph:ceph /data01
chown ceph:ceph /data03
执行挂载命令:
ceph-deploy osd prepare bigdata-oss:/data02
ceph-deploy osd activate bigdata-oss:/data02
。。。。。。
查看:
ceph osd tree
ceph -s
如果出现:
HEALTH_WARN too few PGs per OSD (21 < min 30)解决方法如下
ceph osd pool set rbd pgp_num 128
ceph osd pool set rbd pg_num 128
注意:挂载的磁盘分区文件系统是ext4的话会出现问题,解决方法见文章末尾。
12、配置对象网关
之前已经安装过网关了,这时候我们需要创建一个网关实例:
ceph-deploy --overwrite rgw create 你的主机名
如果成功的话,RGW实例将在端口7480上侦听,可以根据以下命令验证:
ps axu |grep rados----通过端口查看
wget http://172.16.10.4:7480----通过index.html验证
<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Owner>
<ID>anonymous</ID>
<DisplayName></DisplayName>
</Owner>
<Buckets></Buckets>
</ListAllMyBucketsResult>
13、 使用S3 API访问ceph对象存储
创建用户:
radosgw-admin user create --secret="HELLOworld" --uid="zhanghan" --display-name="zhanghan"
成功后会出现:
{
"user_id": "radosgw",
"display_name": "radosgw",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "radosgw",
"access_key": "DKOORDOMS6YHR2OW5M23",
"secret_key": "OOBNCO0d03oiBaLCtYePPQ7gIeUR2Y7UuB24pBW4"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
注意:要记住"access_key"和"secret_key"
14、使用python-boto客户端,实现UP和DOWLOAD
安装:yum install python-boto
创建bucket:
import boto.s3.connection
conn = boto.connect_s3(
aws_access_key_id='',
aws_secret_access_key='',
host='', port=7480,
is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.create_bucket('my-new-bucket')
for bucket in conn.get_all_buckets():
print "{name} {created}".format(
name=bucket.name,
created=bucket.creation_date,
)
for key in bucket.list():
print "{name}\t{size}\t{modified}".format(
name=key.name,
size=key.size,
modified =key.last_modified,
)
上传:
import boto.s3.connection
conn = boto.connect_s3(
aws_access_key_id='',
aws_secret_access_key='',
host='', port=7480,
is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
)
bucket=conn.get_bucket('my-new-bucket')
from boto.s3.key import Key
k=Key(bucket)
k.key='根据需求写你的key'
k.set_contents_from_filename('根据需求写你的路径')
下载:
import boto.s3.connection
conn = boto.connect_s3(
aws_access_key_id='',
aws_secret_access_key='',
host='', port=7480,
is_secure=False, calling_format=boto.s3.connection.OrdinaryCallingFormat(),
)
bucket = conn.create_bucket('my-new-bucket')
key=bucket.get_key('根据需求写你的key')
key.get_contents_to_filename('根据需求写你的路径')
finally:
1、解决ext4系统出错问题
在ceph.conf中添加如下两行,并重启:
添加:
osd max object name len = 256
osd max object namespace len = 64
重启:
systemctl status ceph-mon@主机名
2、手动安装
下载:https://pypi.org/project/pip/#files
安装:
tar -xzvf pip-20.2.tar.gz
cd pip-20.2
python setup.py install**