升级ceph

最新推荐文章于 2022-04-13 14:57:33 发布

weixin_30914981

最新推荐文章于 2022-04-13 14:57:33 发布

阅读量369

点赞数

文章标签：运维操作系统

原文链接：http://www.cnblogs.com/wangmo/p/11278266.html

版权

参考文档

https://blog.51cto.com/jerrymin/2140258

https://www.virtualtothecore.com/en/upgrade-ceph-cluster-luminous/
http://www.chinastor.com/distristor/11033L502017.html

官方文档：https://docs.ceph.com/docs/master/install/upgrading-ceph/

缘起

首先看之前安装版本链接及测试
https://blog.51cto.com/jerrymin/2139045
https://blog.51cto.com/jerrymin/2139046
mon ceph0、ceph2、cphe3
osd ceph0、ceph1、ceph2、ceph3
rgw ceph1
deploy ceph0
之前在Centos7.5上测试了Jewel版本的集群，随着对Ceph了解深入，计划线上运行比较新的LTS版本Ceph集群，最终选择了Luminous版本。
本来计划重新部署Luminous版本，看到这是测试环境数据丢失风险小就想尝试升级Jewel版本到Luminous版本，由于之前是Yum安装的根据之前经验原理是更新二进制文件，最后重启服务即可。看介绍文档升级步骤比较简单，但是测试中发现国内用户肯定会遇到一个坑，升级过程中会自动修改yum源到国外的站点，由于网络延迟大300s反应不及时会自动断开连接停止升级服务，故后续的操作我改成了国内源，并找出rpm包手工升级，由于本身有依赖关系故最终就收到yum install ceph ceph-radosgw即可全部升级所有的ceph包，最后重启相关服务即完成升级，最终数据没有丢失，各个功能也正常。

升级过程

参照官方的升级指南一步一步的小心操作，要注意，升级时候要确保系统是健康运行的状态。
1、登录，确认sortbitwise是enabled状态：

[root@idcv-ceph0 yum.repos.d]# ceph osd set sortbitwise
set sortbitwise

2、设置noout标志，告诉Ceph不要重新去做集群的负载均衡，虽然这是可选项，但是建议设置一下，避免每次停止节点的时候，Ceph就尝试通过复制数据到其他可用节点来重新平衡集群。

[root@idcv-ceph0 yum.repos.d]# ceph osd set noout
set noout

3、升级时，可以选择手工升级每个节点，也可以使用使用Ceph-deploy实现自动升级。如果选择手动升级，在CentOS系统里，你需要先编辑Ceph yum repo获取新的Luminous版本来替换老版本Jewel，这就需要一个简单的文本替换操作：

[root@idcv-ceph0 yum.repos.d]# sed -i 's/jewel/luminous/' /etc/yum.repos.d/ceph.repo

4、使用Ceph-deploy可以实现一个命令完成集群的自动升级

[root@idcv-ceph0 yum.repos.d]# yum install ceph-deploy python-pushy
Running transaction
Updating : ceph-deploy-2.0.0-0.noarch 1/2
Cleanup : ceph-deploy-1.5.39-0.noarch 2/2
Verifying : ceph-deploy-2.0.0-0.noarch 1/2
Verifying : ceph-deploy-1.5.39-0.noarch 2/2
Updated:
ceph-deploy.noarch 0:2.0.0-0
Complete!
[root@idcv-ceph0 yum.repos.d]# rpm -qa |grep ceph-deploy
ceph-deploy-2.0.0-0.noarch

5、一旦Ceph-deploy升级完成，首先要做的是在同一台机器上升级Ceph。执行这步之前先解决下面两个报错，先安装缺少的包，在执行ceph-deploy
发现从这个时候开始按照官网的步骤在国内行不通了，主要原因是yum源改成了国外的网速达不到，后续就是手工升级结合官网步骤了。

[root@idcv-ceph0 yum.repos.d]# ceph -s
    cluster 812d3acb-eaa8-4355-9a74-64f2cd5209b3
     health HEALTH_WARN
            noout flag(s) set
     monmap e2: 3 mons at {idcv-ceph0=172.20.1.138:6789/0,idcv-ceph2=172.20.1.140:6789/0,idcv-ceph3=172.20.1.141:6789/0}
 election epoch 8, quorum 0,1,2 idcv-ceph0,idcv-ceph2,idcv-ceph3  osdmap e49: 4 osds: 4 up, 4 in  flags noout,sortbitwise,require_jewel_osds  pgmap v53288: 272 pgs, 12 pools, 97496 MB data, 1785 kobjects  296 GB used, 84824 MB / 379 GB avail  272 active+clean [root@idcv-ceph0 yum.repos.d]# ceph -v ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) [root@idcv-ceph0 yum.repos.d]# cd /root/cluster/ [root@idcv-ceph0 cluster]# ls ceph.bootstrap-mds.keyring ceph.bootstrap-osd.keyring ceph.client.admin.keyring ceph-deploy-ceph.log ceph.bootstrap-mgr.keyring ceph.bootstrap-rgw.keyring ceph.conf [root@idcv-ceph0 cluster]# ceph-deploy install --release luminous idcv-ceph0 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.0): /usr/bin/ceph-deploy install --release luminous idcv-ceph0 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] testing : None [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f38ae7a1d40> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] dev_commit : None [ceph_deploy.cli][INFO ] install_mds : False [ceph_deploy.cli][INFO ] stable : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] adjust_repos : True [ceph_deploy.cli][INFO ] func : <function install at 0x7f38ae9d8ed8> [ceph_deploy.cli][INFO ] install_mgr : False [ceph_deploy.cli][INFO ] install_all : False [ceph_deploy.cli][INFO ] repo : False [ceph_deploy.cli][INFO ] host : ['idcv-ceph0'] [ceph_deploy.cli][INFO ] install_rgw : False [ceph_deploy.cli][INFO ] install_tests : False [ceph_deploy.cli][INFO ] repo_url : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] install_osd : False [ceph_deploy.cli][INFO ] version_kind : stable [ceph_deploy.cli][INFO ] install_common : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] dev : master [ceph_deploy.cli][INFO ] nogpgcheck : False [ceph_deploy.cli][INFO ] local_mirror : None [ceph_deploy.cli][INFO ] release : luminous [ceph_deploy.cli][INFO ] install_mon : False [ceph_deploy.cli][INFO ] gpg_url : None [ceph_deploy.install][DEBUG ] Installing stable version luminous on cluster ceph hosts idcv-ceph0 [ceph_deploy.install][DEBUG ] Detecting platform for host idcv-ceph0 ... [idcv-ceph0][DEBUG ] connected to host: idcv-ceph0 [idcv-ceph0][DEBUG ] detect platform information from remote host [idcv-ceph0][DEBUG ] detect machine type [ceph_deploy.install][INFO ] Distro info: CentOS Linux 7.5.1804 Core [idcv-ceph0][INFO ] installing Ceph on idcv-ceph0 [idcv-ceph0][INFO ] Running command: yum clean all [idcv-ceph0][DEBUG ] Loaded plugins: fastestmirror, priorities [idcv-ceph0][DEBUG ] Cleaning repos: Ceph Ceph-noarch base ceph-source epel extras updates [idcv-ceph0][DEBUG ] Cleaning up everything [idcv-ceph0][DEBUG ] Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos [idcv-ceph0][DEBUG ] Cleaning up list of fastest mirrors [idcv-ceph0][INFO ] Running command: yum -y install epel-release [idcv-ceph0][DEBUG ] Loaded plugins: fastestmirror, priorities [idcv-ceph0][DEBUG ] Determining fastest mirrors [idcv-ceph0][DEBUG ] * base: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] * epel: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] * extras: mirrors.neusoft.edu.cn [idcv-ceph0][DEBUG ] * updates: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] 8 packages excluded due to repository priority protections [idcv-ceph0][DEBUG ] Package epel-release-7-11.noarch already installed and latest version [idcv-ceph0][DEBUG ] Nothing to do [idcv-ceph0][INFO ] Running command: yum -y install yum-plugin-priorities [idcv-ceph0][DEBUG ] Loaded plugins: fastestmirror, priorities [idcv-ceph0][DEBUG ] Loading mirror speeds from cached hostfile [idcv-ceph0][DEBUG ] * base: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] * epel: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] * extras: mirrors.neusoft.edu.cn [idcv-ceph0][DEBUG ] * updates: mirrors.huaweicloud.com [idcv-ceph0][DEBUG ] 8 packages excluded due to repository priority protections [idcv-ceph0][DEBUG ] Package yum-plugin-priorities-1.1.31-45.el7.noarch already installed and latest version [idcv-ceph0][DEBUG ] Nothing to do [idcv-ceph0][DEBUG ] Configure Yum priorities to include obsoletes [idcv-ceph0][WARNIN] check_obsoletes has been enabled for Yum priorities plugin [idcv-ceph0][INFO ] Running command: rpm --import https://download.ceph.com/keys/release.asc [idcv-ceph0][INFO ] Running command: yum remove -y ceph-release [idcv-ceph0][DEBUG ] Loaded plugins: fastestmirror, priorities [idcv-ceph0][DEBUG ] Resolving Dependencies [idcv-ceph0][DEBUG ] --> Running transaction check [idcv-ceph0][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be erased [idcv-ceph0][DEBUG ] --> Finished Dependency Resolution [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Dependencies Resolved [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] ================================================================================ [idcv-ceph0][DEBUG ] Package Arch Version Repository Size [idcv-ceph0][DEBUG ] ================================================================================ [idcv-ceph0][DEBUG ] Removing: [idcv-ceph0][DEBUG ] ceph-release noarch 1-1.el7 installed 535 [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Transaction Summary [idcv-ceph0][DEBUG ] ================================================================================ [idcv-ceph0][DEBUG ] Remove 1 Package [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Installed size: 535 [idcv-ceph0][DEBUG ] Downloading packages: [idcv-ceph0][DEBUG ] Running transaction check [idcv-ceph0][DEBUG ] Running transaction test [idcv-ceph0][DEBUG ] Transaction test succeeded [idcv-ceph0][DEBUG ] Running transaction [idcv-ceph0][DEBUG ] Erasing : ceph-release-1-1.el7.noarch 1/1 [idcv-ceph0][DEBUG ] warning: /etc/yum.repos.d/ceph.repo saved as /etc/yum.repos.d/ceph.repo.rpmsave [idcv-ceph0][DEBUG ] Verifying : ceph-release-1-1.el7.noarch 1/1 [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Removed: [idcv-ceph0][DEBUG ] ceph-release.noarch 0:1-1.el7 [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Complete! [idcv-ceph0][INFO ] Running command: yum install -y https://download.ceph.com/rpm-luminous/el7/noarch/ceph-release-1-0.el7.noarch.rpm [idcv-ceph0][DEBUG ] Loaded plugins: fastestmirror, priorities [idcv-ceph0][DEBUG ] Examining /var/tmp/yum-root-dPpRu6/ceph-release-1-0.el7.noarch.rpm: ceph-release-1-1.el7.noarch [idcv-ceph0][DEBUG ] Marking /var/tmp/yum-root-dPpRu6/ceph-release-1-0.el7.noarch.rpm to be installed [idcv-ceph0][DEBUG ] Resolving Dependencies [idcv-ceph0][DEBUG ] --> Running transaction check [idcv-ceph0][DEBUG ] ---> Package ceph-release.noarch 0:1-1.el7 will be installed [idcv-ceph0][DEBUG ] --> Finished Dependency Resolution [idcv-ceph0][DEBUG ] [idcv-ceph0][DEBUG ] Dependencies Resolved [idcv-ceph0][DEBUG ] [idcv-ceph0][