【ceph】ceph集群在执行ceph-volume命令卡住了，惊住了！_ceph 启动所有命令都卡住

2401_84263282

于 2024-04-12 20:04:38 发布

阅读量354

点赞数 4

文章标签： ceph

本文链接：https://blog.csdn.net/2401_84263282/article/details/137693190

版权

文章讲述了在部署Ceph集群时，遇到`ceph-volume`命令卡住的问题，通过分析日志发现lvm相关报错。作者详细描述了问题排查过程，包括查看日志、确认环境一致性和lvm配置错误。

摘要由CSDN通过智能技术生成

《docker学习》暂未更新

《ceph学习》ceph日常问题解决分享

《日志收集》ELK+各种中间件

《运维日常》运维日常

《linux》运维面试100问

ceph集群在执行ceph-volume命令卡住了

背景：

在创建好集群，mon起来后，创建了几个lvm之后，准备开始创建osd，结果命令卡住了

sudo /usr/sbin/ceph-volume --cluster ceph lvm prepare --bluestore --data ceph-osd-168-vg/osd-168-data --osd-id 168 --osd-fsid `uuidgen`

^CTraceback (most recent call last):
File “/usr/sbin/ceph-volume”, line 9, in
load_entry_point(‘ceph-volume==1.0.0’, ‘console_scripts’, ‘ceph-volume’)()
File “/usr/lib/python2.7/site-packages/ceph_volume/main.py”, line 38, in init
self.main(self.argv)
File “/usr/lib/python2.7/site-packages/ceph_volume/decorators.py”, line 59, in newfunc
return f(*a, **kw)
File “/usr/lib/python2.7/site-packages/ceph_volume/main.py”, line 149, in main
terminal.dispatch(self.mapper, subcommand_args)
File “/usr/lib/python2.7/site-packages/ceph_volume/terminal.py”, line 194, in dispatch
instance.main()
File “/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py”, line 40, in main
terminal.dispatch(self.mapper, self.argv)
File “/usr/lib/python2.7/site-packages/ceph_volume/terminal.py”, line 194, in dispatch
instance.main()
File “/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py”, line 355, in main
self.args = parser.parse_args(self.argv)
File “/usr/lib64/python2.7/argparse.py”, line 1688, in parse_args
args, argv = self.parse_known_args(args, namespace)
File “/usr/lib64/python2.7/argparse.py”, line 1720, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File “/usr/lib64/python2.7/argparse.py”, line 1926, in _parse_known_args
start_index = consume_optional(start_index)
File “/usr/lib64/python2.7/argparse.py”, line 1866, in consume_optional
take_action(action, args, option_string)
File “/usr/lib64/python2.7/argparse.py”, line 1778, in take_action
argument_values = self._get_values(action, argument_strings)
File “/usr/lib64/python2.7/argparse.py”, line 2218, in _get_values
value = self._get_value(action, arg_string)
File “/usr/lib64/python2.7/argparse.py”, line 2247, in _get_value
result = type_func(arg_string)
File “/usr/lib/python2.7/site-packages/ceph_volume/util/arg_validators.py”, line 16, in call
device = Device(string)
File “/usr/lib/python2.7/site-packages/ceph_volume/util/device.py”, line 91, in init
self._parse()
File “/usr/lib/python2.7/site-packages/ceph_volume/util/device.py”, line 112, in _parse
sys_info.devices = disk.get_devices()
File “/usr/lib/python2.7/site-packages/ceph_volume/util/disk.py”, line 779, in get_devices
if lvm.is_lv(diskname):
File “/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py”, line 277, in is_lv
lvs = lvs if lvs is not None else Volumes()
File “/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py”, line 780, in init
self._populate()
File “/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py”, line 784, in _populate
for lv_item in get_api_lvs():
File “/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py”, line 322, in get_api_lvs
verbose_on_failure=False
File “/usr/lib/python2.7/site-packages/ceph_volume/process.py”, line 201, in call
**kw
File “/usr/lib64/python2.7/subprocess.py”, line 711, in init
errread, errwrite)
File “/usr/lib64/python2.7/subprocess.py”, line 1308, in _execute_child
data = _eintr_retry_call(os.read, errpipe_read, 1048576)
File “/usr/lib64/python2.7/subprocess.py”, line 478, in _eintr_retry_call
return func(*args)
KeyboardInterrupt

折腾了一下午，第二天早上灵机一动，简直茅塞顿开，在分析和排错过程中，我在其他的节点也尝试创建osd，是正常的，部署过程完全一致，这时就排除环境问题了，接着就开始去看日志

tail -f /var/log/ceph/ceph-volume.log

发现有很多lvm报错：

[2023-11-17 09:56:28,867][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-39-dbs";“osd-39-dbs”;“ceph-osd-39-48-vg”;“TH2NbO-FkpP-OwWg-Takg-dA3S-HHiG-rjnMSS”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-39-wal”;“osd-39-wal”;“ceph-osd-39-48-vg”;“4LeOJZ-aO4A-lcGc-9E2T-dgGg-4slY-Z67f02”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-40-dbs”;“osd-40-dbs”;“ceph-osd-39-48-vg”;“deiH09-zMRi-IxPd-2hrx-fcNy-x7xn-C6HMns”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-40-wal”;“osd-40-wal”;“ceph-osd-39-48-vg”;“PTSVas-ELiR-2Obg-98XM-UyyA-hNsS-Iw9qdk”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-41-dbs”;“osd-41-dbs”;“ceph-osd-39-48-vg”;“hBeM5X-UMDG-5DZU-m7ph-t4tz-ynop-d3pR4v”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-39-48-vg/osd-41-wal”;“osd-41-wal”;“ceph-osd-39-48-vg”;“Vi2tq1-q2pj-nl0C-2NT3-TPGk-74Nb-YmXDxq”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-41-vg/osd-41-data”;“osd-41-data”;“ceph-osd-41-vg”;“PXOBjY-RRsG-od59-bQC7-eSQB-8z1p-1iteTT”;“<9.10t
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-6-dbs”;“osd-6-dbs”;“ceph-osd-6-15-vg”;“ZFlD3R-mrpZ-IWDE-TGPu-E9Iv-K2Oz-jQw0aO”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-6-wal”;“osd-6-wal”;“ceph-osd-6-15-vg”;“Dr92zd-fx2w-9dI4-6nXD-Xeiw-WkwZ-lnjUPl”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-7-dbs”;“osd-7-dbs”;“ceph-osd-6-15-vg”;“uf4XgZ-GpZX-A7Ra-ycGe-LFjr-Ljk3-CCQjj0”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-7-wal”;“osd-7-wal”;“ceph-osd-6-15-vg”;“chzf6p-7BGU-RuXK-T3ST-XeRv-ZyzP-0pfa0k”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-8-dbs”;“osd-8-dbs”;“ceph-osd-6-15-vg”;“LARUIQ-Nrms-1EIE-SWsj-X6is-R3N2-CUfr3U”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-6-15-vg/osd-8-wal”;“osd-8-wal”;“ceph-osd-6-15-vg”;“IdtQeV-hiJ2-AO4s-VkA7-CA5p-PF7L-rt5uWs”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-9-18-vg/osd-10-dbs”;“osd-10-dbs”;“ceph-osd-9-18-vg”;“8kURuP-f7i3-60Uz-DfBG-46XW-2w1G-fIcfqF”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-9-18-vg/osd-10-wal”;“osd-10-wal”;“ceph-osd-9-18-vg”;“tD81bi-CR8V-Xwxk-hB0B-2dhQ-dgX3-KauAaZ”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-9-18-vg/osd-11-dbs”;“osd-11-dbs”;“ceph-osd-9-18-vg”;“h625h3-ZcFv-8Jz5-1N1E-0Yhw-y4Zi-cxIEia”;“145.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-9-18-vg/osd-11-wal”;“osd-11-wal”;“ceph-osd-9-18-vg”;“crZrX3-b7GC-PnZ9-62Oa-xBDB-FPT4-W3iWUi”;“4.00g
[2023-11-17 09:56:28,868][ceph_volume.process][INFO ] stdout “;”/dev/ceph-osd-9-18-vg/osd-9-dbs”;“osd-9-dbs”;“ceph-osd-9-18-vg”;“THZUaH-s8Mi-On3C-rDre-6Ayi-rqzO-u47te4”;"145.00g