ceph librados librbd

ceph简介

Ceph 用统一的系统提供了对象、块、和文件存储功能，它可靠性高、管理简便、并且是自由软件。 Ceph 可提供极大的伸缩性——供成千用户访问 PB 乃至 EB 级的数据。 Ceph 节点以普通硬件和智能守护进程作为支撑点， Ceph 存储集群组织起了大量节点，它们之间靠相互通讯来复制数据、并动态地重分布数据。

下图为应用场景示意图，其核心是RADOS，也即高可用、自动化的分布式对象存储，该模块负责对OSD的自动化运行，保证整个存储系统的可用性。同时，该模块通过一个名为librados的公共库对外提供存储服务，块存储和对象存储都依赖该动态库。

同时，Ceph本身封装了对象存储的接口和块存储的接口。对象存储的接口包括对象网关和对象存储访问库等。块存储接口包括块存储访问动态库、Python绑定库和内核态的虚拟块设备。

两个概念：

RADOS   可靠 自动化 分布式 对象 存储

CRUSH    Controoled Replication Under Scalable Hashing   可控副本伸缩哈希 、一种伪随机数据分布算法

 CRUSH有两个关键优点：

•         任何组件都可以独立计算出每个object所在的位置(去中心化)。
•         只需要很少的元数据(cluster map)，只要当删除添加设备时，这些元数据才需要改变。

ceph使用基础

ceph集群搭建起来后，可以在ceph集群上进行块存储、对象存储以及文件系统存储。从架构上来看，在ceph集群的上面是rados协议，该协议为使用ceph集群的用户提供必要的支持（ceph用户通过调用rados协议来使用ceph集群）。对于块存储来说，可以通过内核模块的方式使用ceph集群也可以通过用户态调用librbd库来使用ceph集群。通过内核模块方式可以充分的利用内核的page cache机制，而通过用户态调用librbd也可以使用librbd在用户态提供的cache方式提高性能。

LIBRADOS提供客户端访问Ceph集群的原生态统一接口。其它接口或者命令行工具都基于该动态库实现。在librados中实现了Crush算法和网络通信等公共功能，数据请求操作在librados计算完成后可以直接与对应的OSD交互进行数据传输。LIBRBD 是CEPH提供的在librados上封装的块存储接口的抽象。

它们提供C/C++、PYTHON等多种接口。对于C++。

librados主要的类是

librados::Rados    负责初始化集群、读取配置、连接集群

librados::IoCtx    负责创建IO上线文环境

librados::bufferlist  负责读写缓存

librbd最主要的两个类是

LIBRADOS::rbd  主要负责创建、删除、克隆映像等操作，

LIBRADOS::image  负责映像的读写等操作。

对于任何客户端应用，都需要首先连接到一个运行良好的CEPH集群。

然后使用时代码中需要包含主要的接口，有：
1. 集群句柄创建、读取配置
2. 集群连接
3. IO上下文环境初始化（poll创建读写等）
4. 对象读写 、rbd、image 创建读写等
5. IO上下文环境关闭

6. 集群句柄关闭

librados的使用

官网链接http://docs.ceph.com/docs/master/rados/api/librados-intro/

STEP 1: GETTING LIBRADOS¶

sudo yum install librados2-devel

sudo apt-get install librados-dev

安装后在/usr/include/rados文件夹下    /usr/inluce/rbd文件夹下

ls /usr/include/rados

buffer.h  crc32c.h  librados.h  librados.hpp  memory.h  page.h  rados_types.h  rados_types.hpp

ls /usr/include/rbd

features.h  librbd.h  librbd.hpp

STEP 2: CONFIGURING A CLUSTER HANDLE

Tip

 

Talking to different Ceph Storage Clusters – or to the same cluster with different users – requires different cluster handles.

C++ EXAMPLE

The Ceph project provides a C++ example in the ceph/examples/librados directory. For C++, a simple cluster handle using the admin user requires you to initialize a librados::Rados cluster handle object:

#include <iostream>
#include <string>
#include <rados/librados.hpp>

int main(int argc, const char **argv)
{

        int ret = 0;

        /* Declare the cluster handle and required variables. */
        librados::Rados cluster;
        char cluster_name[] = "ceph";
        char user_name[] = "client.admin";
        uint64_t flags = 0;

        /* Initialize the cluster handle with the "ceph" cluster name and "client.admin" user */
        {
                ret = cluster.init2(user_name, cluster_name, flags);
                if (ret < 0) {
                        std::cerr << "Couldn't initialize the cluster handle! error " << ret << std::endl;
                        return EXIT_FAILURE;
                } else {
                        std::cout << "Created a cluster handle." << std::endl;
                }
        }

        /* Read a Ceph configuration file to configure the cluster handle. */
        {
                ret = cluster.conf_read_file("/etc/ceph/ceph.conf");
                if (ret < 0) {
                        std::cerr << "Couldn't read the Ceph configuration file! error " << ret << std::endl;
                        return EXIT_FAILURE;
                } else {
                        std::cout << "Read the Ceph configuration file." << std::endl;
                }
        }

        /* Read command line arguments */
        {
                ret = cluster.conf_parse_argv(argc, argv);
                if (ret < 0) {
                        std::cerr << "Couldn't parse command line options! error " << ret << std::endl;
                        return EXIT_FAILURE;
                } else {
                        std::cout << "Parsed command line options." << std::endl;
                }
        }

        /* Connect to the cluster */
        {
                ret = cluster.connect();
                if (ret < 0) {
                        std::cerr << "Couldn't connect to cluster! error " << ret << std::endl;
                        return EXIT_FAILURE;
                } else {
                        std::cout << "Connected to the cluster." << std::endl;
                }
        }

        return 0;
}

Compile the source; then, link librados using -lrados. For example:

g++ -g -c ceph-client.cc -o ceph-client.o
g++ -g ceph-client.o -lrados -o ceph-client

STEP 3: CREATING AN I/O CONTEXT

The following examples use the default data pool. However, you may also use the API to list pools, ensure they exist, or create and delete pools. For the write operations, the examples illustrate how to use synchronous mode. For the read operations, the examples illustrate how to use asynchronous mode.

Important

 

Use caution when deleting pools with this API. If you delete a pool, the pool and ALL DATA in the pool will be lost.

C++ EXAMPLE

#include <iostream>
#include <string>
#include <rados/librados.hpp>

int main(int argc, const char **argv)
{

        /* Continued from previous C++ example, where cluster handle and
         * connection are established. First declare an I/O Context.
         */

        librados::IoCtx io_ctx;
        const char *pool_name = "data";

        {
                ret = cluster.ioctx_create(pool_name, io_ctx);
                if (ret < 0) {
                        std::cerr << "Couldn't set up ioctx! error " << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Created an ioctx for the pool." << std::endl;
                }
        }


        /* Write an object synchronously. */
        {
                librados::bufferlist bl;
                bl.append("Hello World!");
                ret = io_ctx.write_full("hw", bl);
                if (ret < 0) {
                        std::cerr << "Couldn't write object! error " << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Wrote new object 'hw' " << std::endl;
                }
        }


        /*
         * Add an xattr to the object.
         */
        {
                librados::bufferlist lang_bl;
                lang_bl.append("en_US");
                ret = io_ctx.setxattr("hw", "lang", lang_bl);
                if (ret < 0) {
                        std::cerr << "failed to set xattr version entry! error "
                        << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Set the xattr 'lang' on our object!" << std::endl;
                }
        }


        /*
         * Read the object back asynchronously.
         */
        {
                librados::bufferlist read_buf;
                int read_len = 4194304;

                //Create I/O Completion.
                librados::AioCompletion *read_completion = librados::Rados::aio_create_completion();

                //Send read request.
                ret = io_ctx.aio_read("hw", read_completion, &read_buf, read_len, 0);
                if (ret < 0) {
                        std::cerr << "Couldn't start read object! error " << ret << std::endl;
                        exit(EXIT_FAILURE);
                }

                // Wait for the request to complete, and check that it succeeded.
                read_completion->wait_for_complete();
                ret = read_completion->get_return_value();
                if (ret < 0) {
                        std::cerr << "Couldn't read object! error " << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Read object hw asynchronously with contents.\n"
                        << read_buf.c_str() << std::endl;
                }
        }


        /*
         * Read the xattr.
         */
        {
                librados::bufferlist lang_res;
                ret = io_ctx.getxattr("hw", "lang", lang_res);
                if (ret < 0) {
                        std::cerr << "failed to get xattr version entry! error "
                        << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Got the xattr 'lang' from object hw!"
                        << lang_res.c_str() << std::endl;
                }
        }


        /*
         * Remove the xattr.
         */
        {
                ret = io_ctx.rmxattr("hw", "lang");
                if (ret < 0) {
                        std::cerr << "Failed to remove xattr! error "
                        << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Removed the xattr 'lang' from our object!" << std::endl;
                }
        }

        /*
         * Remove the object.
         */
        {
                ret = io_ctx.remove("hw");
                if (ret < 0) {
                        std::cerr << "Couldn't remove object! error " << ret << std::endl;
                        exit(EXIT_FAILURE);
                } else {
                        std::cout << "Removed object 'hw'." << std::endl;
                }
        }
}

STEP 4: CLOSING SESSIONS

C++ EXAMPLE

io_ctx.close();
cluster.shutdown();

LIBRBD的使用

下面是转载

获取集群句柄

//声明Rados对象，并初始化
librados::Rados rados;
ret = rados.init("admin"); // just use the client.admin keyring
if (ret < 0) { // let's handle any error that might have come back
    std::cerr << "couldn't initialize rados! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "we just set up a rados cluster object" << std::endl;
}

//获取配置文件信息: /etc/ceph/ceph.conf
// 1. 根据命令行参数
/*
ret = rados.conf_parse_argv(argc, argv);
if (ret < 0) {
    // This really can't happen, but we need to check to be a good citizen.
    std::cerr << "failed to parse config options! error " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "we just parsed our config options" << std::endl;
    // We also want to apply the config file if the user specified
    // one, and conf_parse_argv won't do that for us.
    for (int i = 0; i < argc; ++i) {
        if ((strcmp(argv[i], "-c") == 0) || (strcmp(argv[i], "--conf") == 0)) {
            ret = rados.conf_read_file(argv[i+1]);
            if (ret < 0) {
                // This could fail if the config file is malformed, but it'd be hard.
                std::cerr << "failed to parse config file " << argv[i+1]
                        << "! error" << ret << std::endl;
                ret = EXIT_FAILURE;
                return EXIT_FAILURE;
            }
            break;
        }
    }
}
*/
// 2. 程序里面指定
ret = rados.conf_read_file("/etc/ceph/ceph.conf");
if (ret < 0) {
    // This could fail if the config file is malformed, but it'd be hard.
    std::cerr << "failed to parse config file! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
}

连接集群

ret = rados.connect();
if (ret < 0) {
    std::cerr << "couldn't connect to cluster! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "we just connected to the rados cluster" << std::endl;
}

创建I/O上下文环境

如果没有存储池，需要先新建一个存储池。

新建存储池

const char *pool_name = "gnar";
ret = rados.pool_create(pool_name);
if (ret < 0) {
    std::cerr << "couldn't create pool! error " << ret << std::endl;
    ret = EXIT_FAILURE;
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "we just created a new pool named " << pool_name << std::endl;
}

新建I/O上下文环境

librados::IoCtx io_ctx; //I/O上下文
const char *pool_name = "gnar";
ret = rados.ioctx_create(pool_name, io_ctx);
if (ret < 0) {
    std::cerr << "couldn't setup ioctx! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE; 
} else {
    std::cout << "we just created an ioctx for our pool" << std::endl;
}

声明RBD对象，创建rbd映像

librbd::RBD rbd;
const char *image_name = "rumboo";
uint64_t init_size = (uint64_t) 200 * 1024 * 1024; //映像初始化大小200MB
uint64_t features = 1; //影响feature个数
int order = 22; //默认值为22, 即4MB (1 << 22)
ret = rbd.create2(io_ctx, image_name, init_size, features, &order);
if (ret < 0) {
    std::cerr << "couldn't create rbd image! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "We just created an rbd image" << std::endl;
}

打开rbd映像

librbd::RBD rbd;
const char *image_name = "rumboo";
librbd::Image image;
ret = rbd.open(io_ctx, image, image_name);
if (ret < 0) {
    std::cerr << "couldn't open rbd image! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "We just opened an rbd image" << std::endl;
}

查看映像大小

uint64_t size = 0;
ret = image.size(&size);
if (ret < 0) {
    std::cerr << "couldn't get image size! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "The size of the image is " << size << std::endl;
}

调整映像大小

size = (uint64_t) 500 * 1024 * 1024; //调整映像大小500MB
ret = image.resize(size);
if (ret < 0) {
    std::cerr << "couldn't change the size of the image! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "We just change the size of the image" << std::endl;
}

查看映像ID

std::string id;
ret = image.get_id(&id);
if (ret < 0) {
    std::cerr << "couldn't get image ID! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "The ID of the image is " << id << std::endl;
}

查看映像features

features = 0;
ret = image.features(&features);
if (ret < 0) {
    std::cerr << "couldn't get image features! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "The features of the image are " << features << std::endl;
}

查看映像状态信息

librbd::image_info_t info;
ret = image.stat(info, sizeof(info));
if (ret < 0) {
    std::cerr << "couldn't get image stat_info! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "info.size is " << info.size << std::endl;
    std::cout << "info.obj_size is " << info.obj_size << std::endl;
    std::cout << "info.num_objs is " << info.num_objs << std::endl;
    std::cout << "info.order is " << info.order << std::endl;
    std::cout << "info.block_name_prefix is " << info.block_name_prefix << std::endl;
}

查看存储池ID

std::cout << "data pool id is " << image.get_data_pool_id() << std::endl;

查看block_name_prefix

std::cout << "block name prefix is " << image.get_block_name_prefix() << std::endl;

查看flags

uint64_t flags = 0;
ret = image.get_flags(&flags);
if (ret < 0) {
    std::cerr << "couldn't get image flags! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "image flags is " << flags << std::endl;
}

查看条带化参数

std::cout << "image stripe unit is " << image.get_stripe_unit() << std::endl;
std::cout << "image stripe count is " << image.get_stripe_count() << std::endl;

RBD映像数据读写

数据读写 – synchronous

uint64_t ofs_w = (uint64_t) 0; //读写偏移量
uint64_t ofs_r = (uint64_t) 0;
size_t len_w = 100; //读写长度
size_t len_r = 100;
ceph::bufferlist bl_w; //读写bufferlist
ceph::bufferlist bl_r;
const char *fn_i = "input"; //读写文件名
const char *fn_o = "output";
std::string error;

ret = bl_r.read_file(fn_i, &error);
std::cout << "read file ret = " << ret << std::endl;
if (ret < 0) {
    std::cerr << "couldn't read file! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "We just read a file" << std::endl;
}
ssize_t ret_w = image.write2(ofs_w, len_w, bl_r, 0);

ssize_t ret_r = image.read2(ofs_r, len_r, bl_w, 0);
ret = bl_w.write_file(fn_o, 0644);
std::cout << "write file ret = " << ret << std::endl;
if (ret < 0) {
    std::cerr << "couldn't write file! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "We just wrote a file" << std::endl;
}

数据读写 – asynchronous

std::string data = "foo"; 
uint64_t ofs_aiow = (uint64_t) 100; //读写偏移量
uint64_t ofs_aior = (uint64_t) 100;
size_t len_aiow = 600; //读写长度
size_t len_aior = 600;
ceph::bufferlist bl_aiow; //读写bufferlist
ceph::bufferlist bl_aior;
librbd::RBD::AioCompletion *write_completion = new librbd::RBD::AioCompletion(
    NULL, (librbd::callback_t) simple_write_cb); //读写AioCompletion
librbd::RBD::AioCompletion *read_completion = new librbd::RBD::AioCompletion(
    NULL, (librbd::callback_t) simple_read_cb);

for (int i = 0; i < 200; ++i) {
    bl_aior.append(data);
}
std::cout << bl_aior.to_str() << std::endl;

ret = image.aio_write2(ofs_aiow, len_aiow, bl_aior, write_completion, 0);
if (ret < 0) {
    std::cerr << "couldn't start write! error " << ret << std::endl;
    ret = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
}
write_completion->wait_for_complete(); //等待写完成
ret_w = write_completion->get_return_value();
if (ret_w < 0) {
    std::cerr << "couldn't write! error " << ret << std::endl;
    ret_w = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "we just write data successfully, return value is " << ret_w << std::endl;
}

ret = image.aio_read2(ofs_aior, len_aior, bl_aiow, read_completion, 0);
if (ret < 0) {
    std::cerr << "couldn't start read! error " << ret << std::endl;
    ret = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
}
read_completion->wait_for_complete(); //等待读完成
ret_r = read_completion->get_return_value();
if (ret_r < 0) {
    std::cerr << "couldn't read! error " << ret << std::endl;
    ret_r = EXIT_FAILURE;
    image.close(); //关闭rbd映像
    io_ctx.close(); //关闭I/O上下文
    rados.shutdown(); //断开集群连接
    return EXIT_FAILURE;
} else {
    std::cout << "we just read data successfully, return value is " << ret_r << std::endl;
}
std::cout << bl_aiow.to_str() << std::endl;

write_completion->release();
read_completion->release();

void simple_write_cb(librbd::completion_t cb, void *arg) {
    std::cout << "write completion cb called!" << std::endl;
}

//简单的回调函数，用于librbd::RBD::AioCompletion
void simple_read_cb(librbd::completion_t cb, void *arg) {
    std::cout << "read completion cb called!" << std::endl;
}

• 
  
• 
  

收尾工作

在最后，一定不能忘记关闭rbd映像、I/O上下文，断开集群连接。

ret = image.close(); //关闭rbd映像
if (ret < 0) {
    std::cerr << "couldn't close rbd image! err " << ret << std::endl;
    ret = EXIT_FAILURE;
    return EXIT_FAILURE;
} else {
    std::cout << "we just closed an rbd image" << std::endl;
}
io_ctx.close(); //关闭I/O上下文
rados.shutdown(); //断开集群连接
return EXIT_SUCCESS;