k8s学习笔记——ceph集群安装

如果//ubuntu系统

添加 release key 

wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -

添加Ceph软件包源

echo deb http://download.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list

更新仓库，并安装 ceph-deploy 

sudo apt-get update && sudo apt-get install ceph-deploy

在安装过程中会报如下错误：

[ceph_deploy][ERROR ] RuntimeError: AttributeError: module 'platform' has no attribute 'linux_distribution'

这是由于python3.7后不再支持platform.linux_distribution

//修改方法

修改/usr/lib/python3/dist-packages/ceph_deploy/hosts/remotes.py文件为如下所示

def platform_information(_linux_distribution=None):
    """ detect platform information from remote host """
    """
    linux_distribution = _linux_distribution or platform.linux_distribution
    distro, release, codename = linux_distribution()
    """
    distro = release = codename = None
    try:
          linux_distribution = _linux_distribution or platform.linux_distribution
          distro, release, codename = linux_distribution()
    except AttributeError:
          pass

下面使用ceph-deploy进行安装

如下操作必须在刚创建的ssh免密用户下进行

如果过程中出现问题可以使用如下命令清除安装数据重新进行。

ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys

用下列命令可以连 Ceph 安装包一起清除：

ceph-deploy purge {ceph-node} [{ceph-node}]

在每个节点上运行如下命令，自动修复存在异常的deb

sudo apt --fix-broken  install

创建集群monitor节点

ceph-deploy new --cluster_network 192.168.11.128/25 --public_network 10.12.70.128/25 k8sceph70201

创建成功后在当前目录下应该有ceph.conf 、ceph.mon.keyring、ceph-deploy-ceph.log三个文件

安装ceph其他节点

//ceph-deploy install admin-node node1 node2 node3
ceph-deploy install k8sceph70201 k8sceph70201 k8sceph70202 k8sceph70203

配置初始 monitor(s)、并收集所有密钥：

ceph-deploy mon create-initial

收集完后，当前目录下会多出如下文件

ceph.bootstrap-mds.keyring、ceph.bootstrap-rgw.keyring、ceph.bootstrap-osd.keyring、ceph.client.admin.keyring

把配置文件和 admin 密钥拷贝到管理节点和 Ceph 节点

ceph-deploy admin k8sceph70201 k8sceph70202 k8sceph70203

确保你对 ceph.client.admin.keyring 有正确的操作权限,，对每个节点执行

sudo chmod +r /etc/ceph/ceph.client.admin.keyring

如果如下报错就为操作权限问题 

auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory

部署管理器守护程序

ceph-deploy mgr create k8sceph70201

添加硬盘

执行下列命令列举一节点上的磁盘：

//ceph-deploy disk list {node-name [node-name]...}
ceph-deploy disk list k8sceph70201

//报错

[k8sceph70202][INFO  ] Running command: sudo fdisk -l
[ceph_deploy][ERROR ] Traceback (most recent call last):
[ceph_deploy][ERROR ]   File "/usr/lib/python3/dist-packages/ceph_deploy/util/decorators.py", line 69, in newfunc
[ceph_deploy][ERROR ]     return f(*a, **kw)
[ceph_deploy][ERROR ]   File "/usr/lib/python3/dist-packages/ceph_deploy/cli.py", line 166, in _main
[ceph_deploy][ERROR ]     return args.func(args)
[ceph_deploy][ERROR ]   File "/usr/lib/python3/dist-packages/ceph_deploy/osd.py", line 434, in disk
[ceph_deploy][ERROR ]     disk_list(args, cfg)
[ceph_deploy][ERROR ]   File "/usr/lib/python3/dist-packages/ceph_deploy/osd.py", line 375, in disk_list
[ceph_deploy][ERROR ]     if line.startswith('Disk /'):
[ceph_deploy][ERROR ] TypeError: startswith first arg must be bytes or a tuple of bytes, not str
[ceph_deploy][ERROR ] 

//解决方法
sudo vim /usr/lib/python3/dist-packages/ceph_deploy/osd.py中
 if line.startswith('Disk /'):
替换为
 if line.startswith(b'Disk /'):

擦净硬盘

ceph-deploy disk zap k8sceph70201 /dev/sdb
//会删除硬盘上所有分区和数据

创建OSD

ceph-deploy --overwrite-conf osd create --data /dev/sdb k8sceph70201

使用ceph -s或者 ceph health查看集群状态，正常显示

cluster:
    id:     d10b3028-b78d-4b23-bacb-ca26c0a35c99
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum k8sceph70201,k8sceph70202,k8sceph70203 (age 2d)
    mgr: k8sceph70201(active, since 3w)
    mds:  1 up:standby
    osd: 9 osds: 9 up (since 3w), 9 in (since 3w)
    rgw: 1 daemon active (k8sceph70201)

  task status:

  data:
    pools:   7 pools, 169 pgs
    objects: 306 objects, 182 MiB
    usage:   10 GiB used, 25 TiB / 25 TiB avail
    pgs:     169 active+clean

ceph health detail //详情查看

//报错

sudo ceph -s

 cluster:
    id:     d10b3028-b78d-4b23-bacb-ca26c0a35c99
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim

//解决方法
ceph config set mon auth_allow_insecure_global_id_reclaim false