Ceph分布式集群部署01

Ceph分布式集群部署

1.Ceph环境准备

Ceph分布式集群部署 1.Ceph环境准备 主机名 ip地址及磁盘 部署组件 node01 192.168.1.181（三块硬盘） Mon、mgr、rgs、osd node02 192.168.1.182（三块硬盘） Mon、mgr、rgs、osd node03 192.168.1.183（三块硬盘） Mon、mgr、rgs、osd 2.Hosts及防火墙设置 node01、node02、node03节点进行如下配置： #添加hosts解析； [root@node01 ~]# cat >/etc/hosts<<EOF 127.0.0.1 localhost localhost.localdomain 192.168.1.181 node01 192.168.1.182 node02 192.168.1.183 node03 EOF #临时关闭selinux和防火墙； [root@node01 ~]# sed -i '/SELINUX/s/enforcing/disabled/g' /etc/sysconfig/selinux [root@node01 ~]# setenforce 0 [root@node01 ~]# systemctl stop firewalld.service [root@node01 ~]# systemctl disable firewalld.service #同步节点时间 [root@node01 ~]# yum install ntpdate -y [root@node01 ~]# ntpdate pool.ntp.org 3.配置Ceph国内基础、EPEL和ceph yum源 #配置epel yum源 [root@node01 ~]# yum install -y epel-release #使用阿里开源镜像提供的epel源 [root@node01 ~]# cd /etc/yum.repos.d [root@node01 ~]# wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo #配置ceph yum源 [root@node01 ~]# echo "[noarch] name=ceph noarch baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch enabled=1 gpgcheck=0 [x86_64] name=ceph x86_64 baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64 enabled=1 gpgcheck=0" > /etc/yum.repos.d/ceph.repo 4.在部署机上安装ceph-deploy（2.0版本+） [root@node01 ~]# yum -y install ceph-deploy python-setuptools [root@node01 ~]# yum install ceph-deploy python-setuptools Loaded plugins: fastestmirror, langpacks Repository epel is listed more than once in the configuration Repository epel-debuginfo is listed more than once in the configuration Repository epel-source is listed more than once in the configuration Loading mirror speeds from cached hostfile ​ * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirror.bit.edu.cn   base                                                           | 3.6 kB 00:00:00       epel                                                           | 4.7 kB 00:00:00       extras                                                         | 2.9 kB 00:00:00       noarch                                                         | 1.5 kB 00:00:00       updates                                                         | 2.9 kB 00:00:00       x86_64                                                         | 1.5 kB 00:00:00       (1/3): epel/x86_64/group_gz                                     | 95 kB 00:00:00       (2/3): epel/x86_64/updateinfo                                   | 1.0 MB 00:00:00       (3/3): epel/x86_64/primary_db                                   | 6.9 MB 00:00:01       Package python-setuptools-0.9.8-7.el7.noarch already installed and latest version   Resolving Dependencies   --> Running transaction check   ---> Package ceph-deploy.noarch 0:2.0.1-0 will be installed   --> Finished Dependency Resolution ​ Dependencies Resolved ​ ======================================================================================= ​ Package               Arch             Version             Repository         Size ======================================================================================= ​ Installing: ceph-deploy           noarch           2.0.1-0             noarch           286 k ​ Transaction Summary ======================================================================================= ​ Install 1 Package ​ Total download size: 286 k Installed size: 1.2 M Is this ok [y/d/N]: y Downloading packages: ceph-deploy-2.0.1-0.noarch.rpm                                 | 286 kB 00:00:00     Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : ceph-deploy-2.0.1-0.noarch                                         1/1 Verifying : ceph-deploy-2.0.1-0.noarch                                         1/1 ​ Installed: ceph-deploy.noarch 0:2.0.1-0                                                         ​ Complete! [root@node01 ~]# [root@node01 ~]# rpm -qa|grep ceph-deploy ceph-deploy-2.0.1-0.noarch [root@node01 ~]# rpm -qa|grep python-setuptools python-setuptools-0.9.8-7.el7.noarch [root@node01 ~]# 5.更新其余节点的yum源 [root@node01 ~]# for host in node{02..03};do scp -r /etc/yum.repos.d/* $host:/etc/yum.repos.d/ ;done [root@node01 ~]# for host in node{02..03};do scp -r /etc/yum.repos.d/* $host:/etc/yum.repos.d;done The authenticity of host 'node02 (192.168.1.182)' can't be established. ECDSA key fingerprint is SHA256:V1WcZifyTKUtNITjoIv89siGUFa/aRKqhVbuSZo44IA. ECDSA key fingerprint is MD5:cb:08:52:c5:43:7c:fe:77:88:7e:cb:e3:4b:1b:26:ab. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node02,192.168.1.182' (ECDSA) to the list of known hosts. root@node02's password: Permission denied, please try again. root@node02's password: CentOS-Base.repo                                     100% 1664     2.1MB/s   00:00     CentOS-CR.repo                                       100% 1309     1.5MB/s   00:00     CentOS-Debuginfo.repo                               100% 649   741.5KB/s   00:00     CentOS-fasttrack.repo                               100% 314   562.5KB/s   00:00     CentOS-Media.repo                                   100% 630     1.0MB/s   00:00     CentOS-Sources.repo                                 100% 1331     2.3MB/s   00:00     CentOS-Vault.repo                                   100% 5701     8.8MB/s   00:00     ceph.repo                                           100% 231   492.4KB/s   00:00     epel-7.repo                                         100% 664     1.1MB/s   00:00     epel.repo                                           100% 951     2.0MB/s   00:00     epel-testing.repo                                   100% 1050     1.9MB/s   00:00     The authenticity of host 'node03 (192.168.1.183)' can't be established. ECDSA key fingerprint is SHA256:+J7KpBTku/CAcO+EjEyryEF0BuUaaDlKwxpimaJY0Qk. ECDSA key fingerprint is MD5:b0:88:a3:db:50:1a:bd:6b:6b:0a:cf:7a:a2:81:38:c7. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node03,192.168.1.183' (ECDSA) to the list of known hosts. root@node03's password: CentOS-Base.repo                                     100% 1664     2.0MB/s   00:00     CentOS-CR.repo                                       100% 1309     1.4MB/s   00:00     CentOS-Debuginfo.repo                               100% 649   555.0KB/s   00:00     CentOS-fasttrack.repo                               100% 314   338.7KB/s   00:00     CentOS-Media.repo                                   100% 630   892.6KB/s   00:00     CentOS-Sources.repo                                 100% 1331     2.1MB/s   00:00     CentOS-Vault.repo                                   100% 5701     7.3MB/s   00:00     ceph.repo                                           100% 231   325.5KB/s   00:00     epel-7.repo                                         100% 664     1.0MB/s   00:00     epel.repo                                           100% 951     1.6MB/s   00:00     epel-testing.repo                                   100% 1050     1.9MB/s   00:00     [root@node01 ~]# 6.在node01上安装相关包 [root@node01 ~]# for host in node{01..03};do ssh -l root $host yum install ceph ceph-radosgw -y ;done [root@node01 ~]# ceph -v [root@node01 ~]# rpm -qa |grep ceph [root@node01 ~]# for host in node{01..03};do ssh -l root $host yum install ceph ceph-radosgw -y;done The authenticity of host 'node01 (192.168.1.181)' can't be established. ECDSA key fingerprint is SHA256:QfJ3r+TUqEFfG/D3a8YYN5twm0IT1vCI5OtIm28GX+M. ECDSA key fingerprint is MD5:1f:36:26:8d:d8:63:81:9f:d6:30:a2:ed:ba:fb:db:ba. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node01,192.168.1.181' (ECDSA) to the list of known hosts. root@node01's password: Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Repository epel is listed more than once in the configuration Repository epel-debuginfo is listed more than once in the configuration Repository epel-source is listed more than once in the configuration * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirror.bit.edu.cn Resolving Dependencies --> Running transaction check ---> Package ceph.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: ceph-mon = 2:14.2.12-0.el7 for package: 2:ceph-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-osd = 2:14.2.12-0.el7 for package: 2:ceph-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-mds = 2:14.2.12-0.el7 for package: 2:ceph-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-mgr = 2:14.2.12-0.el7 for package: 2:ceph-14.2.12-0.el7.x86_64 ---> Package ceph-radosgw.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: librgw2 = 2:14.2.12-0.el7 for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-selinux = 2:14.2.12-0.el7 for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-base = 2:14.2.12-0.el7 for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: librados2 = 2:14.2.12-0.el7 for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: libceph-common.so.0()(64bit) for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-14.2.12-0.el7.x86_64 --> Running transaction check ---> Package ceph-base.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: libcephfs2 = 2:14.2.12-0.el7 for package: 2:ceph-base-14.2.12-0.el7.x86_64 --> Processing Dependency: librbd1 = 2:14.2.12-0.el7 for package: 2:ceph-base-14.2.12-0.el7.x86_64 --> Processing Dependency: ceph-common = 2:14.2.12-0.el7 for package: 2:ceph-base-14.2.12-0.el7.x86_64 --> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-14.2.12-0.el7.x86_64 --> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-14.2.12-0.el7.x86_64 ---> Package ceph-mds.x86_64 2:14.2.12-0.el7 will be installed ---> Package ceph-mgr.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: python-pecan for package: 2:ceph-mgr-14.2.12-0.el7.x86_64 --> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-14.2.12-0.el7.x86_64 --> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-14.2.12-0.el7.x86_64 --> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-14.2.12-0.el7.x86_64 ---> Package ceph-mon.x86_64 2:14.2.12-0.el7 will be installed ---> Package ceph-osd.x86_64 2:14.2.12-0.el7 will be installed ---> Package ceph-selinux.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: selinux-policy-base >= 3.13.1-252.el7_7.6 for package: 2:ceph-selinux-14.2.12-0.el7.x86_64 ---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed ---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed ---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated ---> Package librados2.x86_64 2:14.2.12-0.el7 will be an update ---> Package librgw2.x86_64 2:14.2.12-0.el7 will be installed --> Running transaction check ---> Package ceph-common.x86_64 2:14.2.12-0.el7 will be installed --> Processing Dependency: python-rgw = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: libradosstriper1 = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: python-ceph-argparse = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: python-rbd = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: python-cephfs = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: python-rados = 2:14.2.12-0.el7 for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: python-prettytable for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-14.2.12-0.el7.x86_64 --> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-14.2.12-0.el7.x86_64 ---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed ---> Package libcephfs2.x86_64 2:14.2.12-0.el7 will be installed ---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated ---> Package librbd1.x86_64 2:14.2.12-0.el7 will be an update ---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed --> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 --> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 ---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed ---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed --> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch --> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch --> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch --> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch --> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch ---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed ---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed --> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 ---> Package selinux-policy-targeted.noarch 0:3.13.1-229.el7 will be updated ---> Package selinux-policy-targeted.noarch 0:3.13.1-266.el7_8.1 will be an update --> Processing Dependency: selinux-policy = 3.13.1-266.el7_8.1 for package: selinux-policy-targeted-3.13.1-266.el7_8.1.noarch --> Processing Dependency: selinux-policy = 3.13.1-266.el7_8.1 for package: selinux-policy-targeted-3.13.1-266.el7_8.1.noarch --> Running transaction check ---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed ---> Package libradosstriper1.x86_64 2:14.2.12-0.el7 will be installed ---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed ---> Package python-ceph-argparse.x86_64 2:14.2.12-0.el7 will be installed ---> Package python-cephfs.x86_64 2:14.2.12-0.el7 will be installed ---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed --> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch --> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch ---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed ---> Package python-rados.x86_64 2:14.2.12-0.el7 will be installed ---> Package python-rbd.x86_64 2:14.2.12-0.el7 will be installed ---> Package python-rgw.x86_64 2:14.2.12-0.el7 will be installed ---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed ---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed ---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed ---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed ---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed ---> Package selinux-policy.noarch 0:3.13.1-229.el7 will be updated ---> Package selinux-policy.noarch 0:3.13.1-266.el7_8.1 will be an update ---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed --> Running transaction check ---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed --> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch ---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed --> Running transaction check ---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed --> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch --> Running transaction check ---> Package python-tempita.noarch 0:0.5.1-6.el7 will be instal