CentOS6.3上部署Ceph

一、背景知识

搭建ceph的机器分为两种：client和非client（mds、monitor、osd）。

配置时client只需要在内核编译时选上ceph就行，而其它三种则还需要编译ceph用户态源码（下载地址：http://ceph.com/download/），另外osd还要记得安装btrfs文件系统（内核编译作为模块就行）。

内核版本参考： http://ceph.com/docs/master/install/os-recommendations/#glibc

二、机器分配

IP	Roles	Hostname	备注
222.31.76.209	client	localhost.localdomain
222.31.76.178	mds&monitor	ceph_mds
222.31.76.74	osd	ceph_osd0
222.31.76.67	osd	ceph_osd1
222.31.76.235	osd	ceph_osd2

操作系统：CentOS 6.3
内核版本：linux-3.8.8.tar.xz （stable2013-04-17）
ceph版本：ceph-0.60.tar.gz （01-Apr-201317:42）

三、编译与配置

（1） client

1. 编译最新版内核3.8.8

#make mrproper

#make menuconfig //需要ncurses-devel包：#yum install ncurses-devel。配置时记得选上ceph和btrfs。

#make all //若是多核处理器（例：4核），则可以使用#make -j8命令，以多线程方式加速构建内核

#make modules_install

#make install

编译完成后，修改/etc/grub.conf文件，reboot启动新内核。到此，client端的安装配置就已完成。

（2）mds/monitor/osd

1. 编译最新版内核3.8.8（同client）

2. 编译ceph源码

#tar -xvf ceph-0.60.tar.gz

#cd ceph-0.60

#./autogen.sh

#./configure --without-tcmalloc

若提示以下错误，说明缺少相应依赖包，安装即可：

checking whether -lpthread saves the day... yes
checking for uuid_parse in -luuid... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libuuid not found
See `config.log' for more details.

安装：#yum install libuuid-devel

checking for __res_nquery in -lresolv... yes
checking for add_key in -lkeyutils... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libkeyutils not found
See `config.log' for more details.

安装：#yum install keyutils-libs-devel

checking pkg -config is at least version 0. 9. 0... yes
checking for CRYPTOPP... no
checking for library containing _ZTIN8CryptoPP14CBC_EncryptionE... no
checking for NSS... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no suitable crypto library found
See `config.log' for more details.

安装（下载的rpm包）：

#rpm -ivh cryptopp-5.6.2-2.el6.x86_64.rpm

#rpm -ivh cryptopp-devel-5.6.2-2.el6.x86_64.rpm

checking pkg -config is at least version 0. 9. 0... yes
checking for CRYPTOPP... no
checking for library containing _ZTIN8CryptoPP14CBC_EncryptionE... -lcryptopp
checking for NSS... no
configure : using cryptopp for cryptography
checking for FCGX_Init in -lfcgi... no
checking for fuse_main in -lfuse... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no FUSE found (use --without-fuse to disable)
See `config.log' for more details.

安装：#yum install fuse-devel

checking for fuse_main in -lfuse... yes
checking for fuse_getgroups... no
checking jni.h usability... no
checking jni.h presence... no
checking for jni.h... no
checking for LIBEDIT... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: No usable version of libedit found.
See `config.log' for more details.

安装：#yum install libedit-devel

checking for LIBEDIT... yes
checking atomic_ops.h usability... no
checking atomic_ops.h presence... no
checking for atomic_ops.h... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: no libatomic-ops found (use --without-libatomic-ops to disable)
See `config.log' for more details.

安装：#yum install libatomic_ops-devel （也可按提示，使用#./configure --without-tcmalloc --without-libatomic-ops命令屏蔽掉libatomic-ops）

checking for LIBEDIT... yes
checking for snappy_compress in -lsnappy... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libsnappy not found
See `config.log' for more details.

安装（下载的rpm包）：

#rpm -ivh snappy-1.0.5-1.el6.x86_64.rpm

#rpm -ivh snappy-devel-1.0.5-1.el6.x86_64.rpm

checking for snappy_compress in -lsnappy... yes
checking for leveldb_open in -lleveldb... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libleveldb not found
See `config.log' for more details.

安装（下载的rpm包）：

#rpm -ivh leveldb-1.7.0-2.el6.x86_64.rpm

#rpm -ivh leveldb-devel-1.7.0-2.el6.x86_64.rpm

checking for leveldb_open in -lleveldb... yes
checking for io_submit in -laio... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: libaio not found
See `config.log' for more details.

安装：#yum install libaio-devel

checking for sys /wait.h that is POSIX. 1 compatible... yes
checking boost /spirit /include /classic_core.hpp usability... no
checking boost /spirit /include /classic_core.hpp presence... no
checking for boost /spirit /include /classic_core.hpp... no
checking boost /spirit.hpp usability... no
checking boost /spirit.hpp presence... no
checking for boost /spirit.hpp... no
configure : error : in ` /cwn /ceph /ceph - 0. 60 ':
configure: error: "Can't find boost spirit headers "
See `config.log' for more details.

安装：#yum install boost-devel

checking if more special flags are required for pthreads... no
checking whether to check for GCC pthread /shared inconsistencies... yes
checking whether -pthread is sufficient with -shared... yes
configure : creating . /config.status
config.status : creating Makefile
config.status : creating scripts /gtest -config
config.status : creating build -aux /config.h
config.status : executing depfiles commands
config.status : executing libtool commands

见上说明#./configure --without-tcmalloc命令执行成功，会生成Makefile文件，接下来正式编译：

#make -j8

若过程中报以下错误，说明expat-devel没安装：

CXX osdmaptool.o
CXXLD osdmaptool
CXX ceph_dencoder -ceph_dencoder.o
test /encoding /ceph_dencoder.cc : In function 'int main(int, const char**)' :
test /encoding /ceph_dencoder.cc : 196 : note : variable tracking size limit exceeded with -fvar -tracking -assignments, retrying without
CXX ceph_dencoder -rgw_dencoder.o
In file included from rgw /rgw_dencoder.cc : 6 :
rgw/rgw_acl_s3.h:9:19: error: expat.h: No such file or directory
In file included from rgw /rgw_acl_s3.h : 12,
from rgw /rgw_dencoder.cc : 6 :
rgw/rgw_xml.h:62: error: 'XML_Parser' does not name a type