ceph-ansible部署L版ceph 及 通过iscsi 共享rbd 对接xencenter

仅供参考：  这里要注意python的版本  软件源的匹配

https://blog.csdn.net/cojn52/article/details/85793902

服务器端

1 准备镜像

镜像centos7.5   1804

2 关闭防火墙 selinux

systemctl status firewalld

ystemctl  stop firewalld

getenforce            #查看selinux 状态

setenforce  0          #临时关闭seLinux

也可以永久关闭selinux 但是需要重启

3 设置主机名

4 安装 Ansible  git  vim

5 从git 上下载ceph-ansible

6 进入ceph-ansible 目录

git tag

git checkout v3.2.0

git branch

7 在ceph-ansible 目录

vim hosts

8 给免密操作

ssh-keygen

8在ceph-ansible 目录 进入group_vars 编辑 all.yml

9 添加网络源

在/etc/yum.repo.d/在原有基础上添加网络源

9.1、 这个是ceph的yum源  前提是必须要有epel源解决依赖问题

[root@ceph-1 yum.repos.d]#vim  ceph_stable.repo

[ceph_stable]
baseurl = http://mirrors.163.com/ceph/rpm-luminous/el7/$basearch
gpgcheck = 1
gpgkey = http://mirrors.163.com/ceph/keys/release.asc
name = Ceph Stable repo

9.2、[root@ceph-1 yum.repos.d]# vim elrepo.repo

### Name: ELRepo.org Community Enterprise Linux Repository for el7
### URL: http://elrepo.org/

[elrepo]
name=ELRepo.org Community Enterprise Linux Repository - el7
baseurl=http://elrepo.org/linux/elrepo/el7/$basearch/
    http://mirrors.coreix.net/elrepo/elrepo/el7/$basearch/
    http://mirror.rackspace.com/elrepo/elrepo/el7/$basearch/
    http://repos.lax-noc.com/elrepo/elrepo/el7/$basearch/
    http://mirror.ventraip.net.au/elrepo/elrepo/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo.el7
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

[elrepo-testing]
name=ELRepo.org Community Enterprise Linux Testing Repository - el7
baseurl=http://elrepo.org/linux/testing/el7/$basearch/
    http://mirrors.coreix.net/elrepo/testing/el7/$basearch/
    http://mirror.rackspace.com/elrepo/testing/el7/$basearch/
    http://repos.lax-noc.com/elrepo/testing/el7/$basearch/
    http://mirror.ventraip.net.au/elrepo/testing/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-testing.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://elrepo.org/linux/kernel/el7/$basearch/
    http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/
    http://mirror.rackspace.com/elrepo/kernel/el7/$basearch/
    http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/
    http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

[elrepo-extras]
name=ELRepo.org Community Enterprise Linux Extras Repository - el7
baseurl=http://elrepo.org/linux/extras/el7/$basearch/
    http://mirrors.coreix.net/elrepo/extras/el7/$basearch/
    http://mirror.rackspace.com/elrepo/extras/el7/$basearch/
    http://repos.lax-noc.com/elrepo/extras/el7/$basearch/
    http://mirror.ventraip.net.au/elrepo/extras/el7/$basearch/
mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-extras.el7
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

9.3、[root@ceph-1 yum.repos.d]# vim epel.repo

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
 

9.4、  [root@ceph-1 yum.repos.d]# vim epel-testing.repo

[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-testing-source]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

 

 

10安装依赖包 不安装会报错丢失notario

       yum -y install epel-release

       yum -y install python-pip

       pip install notario

11 部署ceph

在ceph-ansible目录下执行

ansible-playbook site.yml.sample -i hosts

跑完之后

 

12 部署osd

12.1 安装ceph-osd安装包

yum install ceph-osd –y

12.2 查看磁盘

12.3 格式化磁盘

 ceph-disk prepare --bluestore /dev/sdb

ceph-disk prepare --bluestore /dev/sdc

ceph-disk prepare --bluestore /dev/sdd

 

  12.4 给ceph添加osd盘

      ceph-disk prepare --bluestore /dev/sdb

      ceph-disk prepare --bluestore /dev/sdc

      ceph-disk prepare --bluestore /dev/sd

     

13  ceph部署完成

14  创建rbd 块

  14.1  ceph osd pool create rbd 512   创建rbd

  14.2  rbd create volume -p rbd --size 1T   指定大小

15   TCMU实现ISCSI接口

注：大部分对接失败的原因就是没有通过tcmu 实现iscsi的借口

  15.1 在tmp目录下载tcmp代码

    cd /tmp/

    git   clone      https://github.com/open-iscsi/tcmu-runner.git  

 

15.2 安装tcmu

(1)  cd tcmu-runner/

(2)  ./extra/install_dep.sh

 

(3)    cmake -Dwith-glfs=false -Dwith-qcow=false -DSUPPORT_SYSTEMD=ON -DCMAKE_INSTALL_PREFIX=/usr .

 

 

(4)  make

(5)  make install

(6) 启动服务tcmu

   systemctl status tcmu-runner

   systemctl start tcmu-runner

   systemctl status tcmu-runner

   systemctl enable tcmu-runner

 

16 安装ISCSI targetcli  

   yum install targetcli –y

17 配置iscsi 服务器端

 

[root@ceph-1 ~]# targetcli

targetcli shell version 2.1.fb46

Copyright 2011-2013 by Datera, Inc and others.

For help on commands, type 'help'.

/> cd backstores/user:rbd/

/backstores/user:rbd> create cfgstring=rbd/volume name=disk  size=1T

/backstores/user:rbd> cd /

/> cd iscsi

/iscsi> create iqn.2019-01.redhat.ceph

/iscsi> cd iqn.2019-01.redhat.ceph/tpg1/luns

/iscsi/iqn.20...eph/tpg1/luns> create /backstores/user:rbd/disk

/iscsi/iqn.20...eph/tpg1/luns> cd /

/> cd iscsi/iqn.2019-01.redhat.ceph/tpg1/

/iscsi/iqn.20...hat.ceph/tpg1> set attribute authentication=0 demo_mode_write_protect=0

/iscsi/iqn.20...hat.ceph/tpg1> cd /

/> cd iscsi/iqn.2019-01.redhat.ceph/tpg1/portals/     # 保持不变

/iscsi/iqn.20.../tpg1/portals> exit

 

18、安装iscsi 软件包

yum install iscsi-initiator-utils

19、查看是否共享成功

    iscsiadm -m discovery -t sendtargets -p 10.22.20.25

 

 

  客户端

1. 通过ssh 或者 界面console 进入平台

1. 检查是否安装iscsi 安装包 如果没有进行安装

yum install iscsi-initiator-utils  -y

 

3、查看是否能查看到共享成功

    iscsiadm -m discovery -t sendtargets -p 10.22.20.25

 

1. 修改vim /etc/iscsi/initiatorname.iscsi 配置文件 并重启服务

iscsiadm -m discovery -t sendtargets -p 10.22.20.25的输出

 

InitiatorName=10.22.20.25:3260,1 iqn.2019-01.redhat.ceph

InitiatorAlias=xenserver

systemctl restart iscsid

1. 通过xencenter界面挂载iscsi

至此客户端挂载成功 下面测试是否可用