ceph-ansible安装指南-centos8安装ceph- pacific

前言：最近玩ceph ,发现关于ceph的安装配置文档还是太少，且不同版的坑也是不一样。经常参照的文档无法进行下一步安装操作，并有各种各样的坑。介于开源精神，分享下centos8.5环境下的ceph-P版的安装步骤。

一、环境准备：

操作系统：

hostnamectl

     Operating System: CentOS Stream 8

       CPE OS Name: cpe:/o:centos:centos:8

            Kernel: Linux 4.18.0-408.el8.x86_64

 ansible --version

ansible [core 2.13.5]

  python version = 3.9.13 (main, Jun 24 2022, 15:32:51) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]

  jinja version = 3.1.2

ceph --version

ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

 python3 --version

Python 3.6.8

网络环境

主机名	public_ip	cluster_ip	角色	备注
ceph-admin	10.255.208.1	-	ceph-ansible
ceph-mon01	10.255.208.2	-	mon
ceph-mon02	10.255.208.4	-	mon
ceph-mon03	10.255.208.9	-	mon
ceph-osd01	10.255.208.11	192.168.2.1	osd	双网卡，3*100GB
ceph-osd02	10.255.208.12	192.168.2.2	osd	双网卡，3*100GB
ceph-osd03	10.255.208.14	192.168.2.3	osd	双网卡，3*100GB
ceph-osd04	10.255.208.16	192.168.2.4	osd	双网卡，3*100GB
ceph-client01	10.255.208.17	-	client
ceph-client02	10.255.208.18	-	client

二、初始化系统

注：如未另行注明，下面配置到所有主机上

1、设置主机名

   hostnamectl set-hostname ceph-admin

     

  2、配置IP地址

 nmcli connection modify System\ ens192 ipv4.method manual ipv4.addresses 10.255.208.11/24 ipv4.gateway 10.255.208.254 ipv4.dns 10.0.1.1 connection.autoconnect yes

  OSD节点配置cluster_IP地址

    修改网卡的连接名称（con-name）

 nmcli connection modify Wired\ connection\ 1 con-name  ens224

  nmcli connection show

  nmcli connection modify ens224  ipv4.method manual ipv4.addresses 192.168.2.1/24 connection.autoconnect yes

3、关闭防火墙，selinux

  systemctl stop firewalld && systemctl disable firewalld

  setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config && getenforce

 

4、修改 /etc/hosts文件，进行本地解析。

注： 解析的IP是public_IP

cp /etc/hosts /etc/hosts.back

cat > /etc/hosts <<EOF

    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

    

    10.255.208.1      ceph-admin

    10.255.208.2      ceph-mon01

    10.255.208.4      ceph-mon02

    10.255.208.9      ceph-mon03

    10.255.208.11     ceph-osd01

    10.255.208.12      ceph-osd02

    10.255.208.14      ceph-osd03

    10.255.208.16      ceph-osd04

    10.255.208.17      ceph-client01

    10.255.208.18      ceph-client02

EOF

5、修改DNS文件

echo "nameserver  10.0.1.1" >> /etc/resolv.conf

systemctl restart network

ping -c 3 www.baidu.com

6、配置主机时间同步

配置ntp服务，保证集群服务器时间统一

统一时间非常重要，必须要做

大前提：chrony服务端客户端配置完后，重启chronyd服务即可快速完成时间同步，在这之后就不要再手动去修改时间了，一切让时间服务器自己去同步

chrony服务端：ceph-admin节点

# 1、安装

yum -y install chrony

​

# 2、修改配置文件

mv /etc/chrony.conf /etc/chrony.conf.bak

​

cat > /etc/chrony.conf << EOF

server ntp1.aliyun.com iburst minpoll 4 maxpoll 10

server ntp2.aliyun.com iburst minpoll 4 maxpoll 10

server ntp3.aliyun.com iburst minpoll 4 maxpoll 10

server ntp4.aliyun.com iburst minpoll 4 maxpoll 10

server ntp5.aliyun.com iburst minpoll 4 maxpoll 10

server ntp6.aliyun.com iburst minpoll 4 maxpoll 10

server ntp7.aliyun.com iburst minpoll 4 maxpoll 10

driftfile /var/lib/chrony/drift

makestep 10 3

rtcsync

allow 0.0.0.0/0

local stratum 10

keyfile /etc/chrony.keys

logdir /var/log/chrony

stratumweight 0.05

noclientlog

logchange 0.5        ​

EOF

​

# 4、启动chronyd服务

  # 最好重启，这样无论原来是否启动都可以重新加载配置

systemctl restart chronyd.service && systemctl enable chronyd.service &&        systemctl status chronyd.service

chrony客户端：其他节点，完全一样的配置与操作

chrony客户端：除ceph-admin节点的其他节点

 

  # 下述步骤一次性粘贴到每个客户端执行即可

# 1、安装chrony

yum -y install chrony

# 2、需改客户端配置文件

mv /etc/chrony.conf /etc/chrony.conf.bak

cat > /etc/chrony.conf << EOF

​

server ceph-admin iburst

driftfile /var/lib/chrony/drift

makestep 10 3

rtcsync

local stratum 10

keyfile /etc/chrony.key

logdir /var/log/chrony

stratumweight 0.05

noclientlog

logchange 0.5

​

EOF

# 3、启动chronyd

systemctl restart chronyd.service && systemctl enable chronyd.service &&systemctl status chronyd.service

7、添加节点信任关系,免密连接

    ssh-keygen -t rsa

    for i in {1..3}; do  ssh-copy-id ceph-client0$i; done

三、 安装ceph-ansible

1、ceph-admin节点安装 pip和ansible、git

注：ceph版本与ansible版本对应关系

ceph-ansible — ceph-ansible documentation

Releases

stable-6.0 Supports Ceph version pacific. This branch requires Ansible version 2.10.

stable-7.0 Supports Ceph version quincy. This branch requires Ansible version 2.12.

main Supports the main (devel) branch of Ceph. This branch requires Ansible version 2.12.

ansible官网安装文档

Installing Ansible on specific operating systems — Ansible Documentation

yum install -y git epel-release

yum install -y ansible

ansible --version

2、部署ceph集群

这里我选择安装的是ceph quincy版本

git clone https://github.com/ceph/ceph-ansible.git

cd ceph-ansible/

git checkout stable-6.0

pip install -r requirements.txt

###if you see an error like No module named pip, you’ll need to install pip under your chosen Python interpreter before proceeding. This may mean installing an additional OS package (for example, python3-pip), or installing the latest pip directly from the Python Packaging Authority by running the following:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

python3 get-pip.py --user

python3 -m pip install -r requirements.txt

###Successfully installed ansible-2.10.7 ansible-base-2.10.17 importlib-resources-5.4.0 netaddr-0.8.0 packaging-21.3 pyparsing-3.0.9 zipp-3.6.0

4、备份修改/etc/ansible/hosts文件，

cd /etc/ansible/

cp hosts  hosts.back

cat >> /etc/ansible/hosts <<EOF

[mons]

ceph-mon0[1:3]

[osds]

ceph-osd0[1:4]

[mgrs]

ceph-mon0[1:3]

[mdss]

ceph-mon0[1:3]

[clients]

ceph-client0[1:2]

[rgws]

ceph-mon0[1:3]

[grafana-server]

ceph-mon01

EOF

5、检查ansible连接服务器

ansible -m ping all

6、备份group_vars下的yml文件

cd ceph-ansible/group_vars

for file in *;do cp $file ${file%.*};done

7、修改group_vars/all.yml配置

cat > all.yml <<EOF

---

dummy:

mon_group_name: mons

osd_group_name: osds

rgw_group_name: rgws

mds_group_name: mdss

client_group_name: clients

mgr_group_name: mgrs

grafana_server_group_name: grafana-server

configure_firewall: False

ceph_origin: repository

ceph_repository: community

ceph_mirror: ceph安装包下载_开源镜像站-阿里云

ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc

ceph_stable_release: pacific

ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"

public_network: "10.255.208.0/24"

cluster_network: "192.168.2.0/24"

monitor_interface: ens192

monitor_address_block: 10.255.208.0/24

ip_version: ipv4

osd_auto_discovery: true

osd_objectstore: bluestroe

radosgw_interface: ens192

dashboard_admin_password: asd123456

grafana_admin_password: admin

pg_autoscale_mode: True

dashboard_enabled: false

 EOF

8、修改group_vars/osdo.yml配置

devices:

  - /dev/sdb

  - /dev/sdc

  - /dev/sdd

9、修改site.yml配置

[root@ceph-admin ceph-ansible]# pwd

/root/ceph-ansible

 cp site.yml.sample site.yml

[root@ceph-admin ceph-ansible]# cat site.yml

- hosts:

  - mons

  - osds

  - mdss

  - rgws

    #  - nfss

    #- rbdmirrors

  - clients

  - mgrs

    #- iscsigws

  - monitoring

    #- rgwloadbalancers

10、开始进行安装

ansible-playbook -i /etc/ansible/hosts site.yml

11、检查ceph状态

root@ceph-mon01 ~]# ceph -s

  cluster:

    id:     74cd55a1-69c2-4114-b73b-a90cf504eda9

    health: HEALTH_OK

  services:

    mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 (age 4h)

    mgr: ceph-mon01(active, since 2d), standbys: ceph-mon03, ceph-mon02

    mds: 1/1 daemons up, 2 standby

    osd: 12 osds: 12 up (since 99m), 12 in (since 2d)

    rgw: 3 daemons active (3 hosts, 1 zones)

  data:

    volumes: 1/1 healthy

    pools:   7 pools, 169 pgs

    objects: 247 objects, 9.6 KiB

    usage:   2.0 GiB used, 1.2 TiB / 1.2 TiB avail

    pgs:     169 active+clean

  io:

    client:   95 KiB/s rd, 0 B/s wr, 95 op/s rd, 63 op/s wr

  progress:

    Global Recovery Event (10s)

      [===========================.]