linux 基于三台物理机安装ceph nautilus

一 摘要

本文基于三台物理机部署ceph ,物理机都安装了centos7.6.1810,三台物理机品牌不一样，硬盘配置不一样。

二 环境信息及部署规划

（一）操作系统版本

[root@localhost ~]# cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)
[root@localhost ~]#

(二)ceph 版本

(三)物理机信息

2.3.1 机器配置

品牌	物理机配置
Huashan Technologies R4900 G2	Intel® Xeon® CPU E5-2650 v4 @ 2.20GHz*2；内存：128G；硬盘：5块硬盘大小未标明，总大小4.4T
Sugon I620-G20	Intel® Xeon® CPU E5-2650 v4 @ 2.20GHz*2；内存：224G；硬盘：7块硬盘大小未标明，总大小4.4T+558.4G
浪潮 SA5212M5	Intel Silver 4210 2.2G * 2 ；内存：总256G 32G*8；硬盘：240G SSD * 2 ；8T/SATA/7.2K * 6；网卡：网卡 X710 10G * 1

2.3.2机器raid 方式

系统盘用raid1,数据盘若raid 卡支持单盘raid0，就做单盘raid0,不支持 就是单盘volumn，不要把几张数据盘放在一起做raid0.

(四)部署规划

| 主机名 | IP |磁盘 |角色|
| ---- | ---- | ---- | ---- | ---- |—|
| cephtest001.ceph.kxdigit.com | 10.3.176.10 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd |ceph-deploy,monitor,mgr,mds,osd|
| cephtest002.ceph.kxdigit.com | 10.3.176.16 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf |monitor,mgr,mds,osd|
| cephtest003.ceph.kxdigit.com | 10.3.176.44 | 系统盘：/dev/sda 数据盘：/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg |monitor,mgr,mds,osd|

四 部署

（一）基础安装

4.1.1 配置dns（若内网无dns系统，请跳过该步骤）

CENTOS7 修改dns

4.1.1.1首先新增/etc/resolv.conf

[dev@10-3-170-32 centos]$ cat resolv.conf
# Generated by NetworkManager
nameserver 10.3.157.201
[dev@10-3-170-32 centos]$

4.1.1.2其次是修改/etc/NetworkManager/NetworkManager.conf

[dev@10-3-170-32 NetworkManager]$ cat NetworkManager.conf
# Configuration file for NetworkManager.
#
# See "man 5 NetworkManager.conf" for details.
#
# The directory /etc/NetworkManager/conf.d/ can contain additional configuration
# snippets. Those snippets override the settings from this main file.
#
# The files within conf.d/ directory are read in asciibetical order.
#
# If two files define the same key, the one that is read afterwards will overwrite
# the previous one.

[main]
plugins=ifcfg-rh
dns=none
[logging]
#level=DEBUG
#domains=ALL
[dev@10-3-170-32 NetworkManager]$

4.1.1.3 最后是重启NETWORKD

systemctl restart  NetworkManager

4.1.1.4

我这边都封装到ansible 脚本里

- hosts: 00base
  tasks:
   - name: register var
     command: date +%Y-%m-%d_%H-%M-%S
     register: datetime

   - name: display vars
     debug: msg="{
  {datetime.stdout}}"

   - name: check /etc/resolv.conf
     shell: ls /etc/resolv.conf
     register: resolvmsg
     ignore_errors: true

   - name: file not exist
     when: resolvmsg is failed  #调试文件不存在
     debug: msg="resolv.conf is not found..."


   - name: backup old dns file
     when: resolvmsg is success
     command: cp /etc/resolv.conf  /etc/resolv.conf.bak."{
  {datetime.stdout}}"

   - name: backup old
     command: cp /etc/NetworkManager/NetworkManager.conf  /etc/NetworkManager/NetworkManager.conf.bak."{
  {datetime.stdout}}"

   - name: copy intranet resolv.conf to server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/centos/resolv.conf",  dest: "/etc/resolv.conf" }


   - name: copy intranet networkmanager.conf to server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/centos7/NetworkManager.conf",  dest: "/etc/NetworkManager/NetworkManager.conf" }
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"

   - name: restart NetworkManager
     command: systemctl restart NetworkManager
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"


   - name: copy intranet networkmanager.conf to server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/centos8/NetworkManager.conf",  dest: "/etc/NetworkManager/NetworkManager.conf" }
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "8"


   - name: restart NetworkManager
     command: systemctl restart NetworkManager
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "8"


   - name: test ansible own variable
     debug: msg="OS ansible_distribution {
  { ansible_distribution }} major version {
  { ansible_distribution_major_version }}"

4.1.2 配置yum

配置内网yum,方便后面安装依赖。

4.1.2.1 内网yum 搭建

搭建内网yum 此处略。后续专门出专门文章。

4.1.2.2配置yum

修改该目录下
/etc/yum.repos.d 文件即可

4.1.2.3 ansible 脚本

我这边也对yum 源更新编写了脚本

- hosts: 00base
  tasks:
   - name: register var
     command: date +%Y-%m-%d_%H-%M-%S
     register: datetime

   - name: display vars
     debug: msg="{
  {datetime.stdout}}"

   - name: backup old yum file
     command: tar -zcvf  /etc/yum.repos.d."{
  {datetime.stdout}}".tgz /etc/yum.repos.d

   - name: list the files of  /etc/yum.repos.d
     shell: ls
     args:
      chdir: /etc/yum.repos.d
     register: files_list

   - name: remove files
     file:
       path: /etc/yum.repos.d/{
  { item }}
       state: absent
     with_items:
       - "{
  { files_list.stdout_lines }}"
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7"

   - name: remove files
     file:
       path: /etc/yum.repos.d/{
  { item }}
       state: absent
     with_items:
       - "{
  { files_list.stdout_lines }}"
     when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "8"

   - name: copy intranet yum to 731611 version server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/yumconfig/CentOS-73.repo",  dest: "/etc/yum.repos.d/CentOS-73.repo" }
     when: ansible_distribution == "CentOS" and ansible_distribution_version == "7.3"

   - name: copy intranet yum to 771908 version server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/yumconfig/CentOS-77.repo",  dest: "/etc/yum.repos.d/CentOS-77.repo" }
     when: ansible_distribution == "CentOS" and ansible_distribution_version == "7.7"

   - name: copy intranet yum to 771908 version server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/yumconfig/CentOS-76.repo",  dest: "/etc/yum.repos.d/CentOS-76.repo" }
     when: ansible_distribution == "CentOS" and ansible_distribution_version == "7.6"

   - name: copy intranet yum to 7 version server
     copy:
       src: "{
  { item.src }}"
       dest: "{
  { item.dest }}"
     with_items:
       - { src: "/home/dev/ansible/files/yumconfig/openstack.repo",  dest: "/etc/yum.repos.d/openstack.repo" }
       - { src: "/home/dev/ansible/files/yumconfig/