https://blog.csdn.net/zrs19800702/article/details/53101213
http://blog.csdn.net/lzw06061139/article/details/51445311
https://my.oschina.net/linuxhunter/blog/654080
rgw 概述
Ceph 通过radosgw提供RESTFul HTTP API接口支持对象存储能力,radosgw构建在librados之上,兼容Amazon S3以及Opensack Swift。
radosgw本质上是一个客户端程序,提供FastCGI 服务。作为一个客户端程序,需要满足如下要求:
- 一个实例名称,默认为:gateway
- 一个合法用户
- 多个存储池
- 一个数据目录
- 在ceph.conf中添加一个配置项
- 前端配置文件
radosgw支持以Apache、Civetweb、Nginx作为前端。Civetweb是默认前端,通过修改ceph.conf配置文件能够很容易的替换为Apache,通过配置能也以nginx作为前端。
下面分别给出centos7上rgw的安装配置过程
安装
通过ceph-deploy可以方便的在rgw node上安装rgw包:
<code class="hljs vala has-numbering"><span class="hljs-preprocessor">#> ceph-deploy --rgw install {rgw-node-name}</span></code><ul class="pre-numbering" style=""><li>1</li></ul><ul class="pre-numbering" style=""><li>1</li></ul>
创建用户
每个rgw实例都需要一个授权用户及key,下面的例子中创建了一个名为gateway的用户,并将密钥文件存储在/etc/ceph目录下
<code class="hljs oxygene has-numbering">#> ceph auth get-<span class="hljs-keyword">or</span>-<span class="hljs-keyword">create</span> client.radosgw.gateway osd <span class="hljs-string">'allow rwx'</span> mon <span class="hljs-string">'allow rwx'</span> -o /etc/ceph/ceph.client.radosgw.keyring</code><ul class="pre-numbering" style=""><li>1</li></ul><ul class="pre-numbering" style=""><li>1</li></ul>
创建存储池
rgw需要存储池来存储数据,如果授权用户具有相关权限,rgw将会自动创建存储池,如果使用默认的区域(region)和可用区(zone),将包含如下的池:
<code class="hljs avrasm has-numbering"><span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.root</span> <span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.control</span> <span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.gc</span> <span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.buckets</span> <span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.buckets</span><span class="hljs-preprocessor">.index</span> <span class="hljs-preprocessor">.rgw</span><span class="hljs-preprocessor">.buckets</span><span class="hljs-preprocessor">.extra</span> <span class="hljs-preprocessor">.log</span> <span class="hljs-preprocessor">.intent</span>-log <span class="hljs-preprocessor">.usage</span> <span class="hljs-preprocessor">.users</span> <span class="hljs-preprocessor">.users</span><span class="hljs-preprocessor">.email</span> <span class="hljs-preprocessor">.users</span><span class="hljs-preprocessor">.swift</span> <span class="hljs-preprocessor">.users</span><span class="hljs-preprocessor">.uid</span></code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul>
当然,您也可以手动创建各个存储池:
<code class="hljs mathematica has-numbering">#> ceph osd pool create <span class="hljs-list">{poolname}</span> <span class="hljs-list">{pg-num}</span> <span class="hljs-list">{pgp-num}</span> <span class="hljs-list">{replicated | erasure}</span> [<span class="hljs-list">{erasure-code-profile}</span>] <span class="hljs-list">{ruleset-name}</span> <span class="hljs-list">{ruleset-number}</span></code><ul class="pre-numbering" style=""><li>1</li></ul><ul class="pre-numbering" style=""><li>1</li></ul>
添加rgw配置
在ceph.conf中添加一个名为gateway的实例。
Civetweb
如果以civetweb作为前端,配置如下:
<code class="hljs avrasm has-numbering">[client<span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.gateway</span>] host = {hostname} keyring = /etc/ceph/ceph<span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.keyring</span> log file = /var/log/radosgw/client<span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.gateway</span>-node1<span class="hljs-preprocessor">.log</span> rgw_frontends = civetweb port=<span class="hljs-number">80</span></code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li></ul>
civetweb默认监听在7480端口,上述的配置中显示指定监听端口为80(
port=80
)
Apache
如果以apache作为前端,配置如下:
<code class="hljs avrasm has-numbering">[client<span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.gateway</span>] host = {hostname} keyring = /etc/ceph/ceph<span class="hljs-preprocessor">.client</span><span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.keyring</span> rgw socket path = <span class="hljs-string">""</span> log file = /var/log/radosgw/client<span class="hljs-preprocessor">.radosgw</span><span class="hljs-preprocessor">.gateway</span><span class="hljs-preprocessor">.log</span> rgw frontends = fastcgi socket_port=<span class="hljs-number">9000</span> socket_host=<span class="hljs-number">0.0</span><span class="hljs-number">.0</span><span class="hljs-number">.0</span> rgw print continue = false</code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li></ul>
配置apache服务,创建/etc/httpd/conf.d/rgw.conf
,并写入如下内容:
<code class="hljs apache has-numbering"><span class="hljs-tag"><VirtualHost *:80></span> <span class="hljs-keyword"><span class="hljs-common">ServerName</span></span> localhost <span class="hljs-keyword"><span class="hljs-common">DocumentRoot</span></span> /var/www/html <span class="hljs-keyword">ErrorLog</span> /var/log/httpd/rgw_error.log <span class="hljs-keyword">CustomLog</span> /var/log/httpd/rgw_access.log combined <span class="hljs-comment"># LogLevel debug</span> <span class="hljs-keyword"><span class="hljs-common">RewriteEngine</span></span> <span class="hljs-literal">On</span> <span class="hljs-keyword"><span class="hljs-common">RewriteRule</span></span> .* -<span class="hljs-sqbracket"> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]</span> <span class="hljs-keyword"><span class="hljs-common">SetEnv</span></span> proxy-nokeepalive 1 <span class="hljs-keyword">ProxyPass</span> / fcgi://localhost:9000/ <span class="hljs-tag"></VirtualHost></span></code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li></ul>
配置好之后,重启apache服务:systemctl restart httpd.service
Nginx
ceph.conf配置如下:
<code class="hljs ini has-numbering"><span class="hljs-title">[client.radosgw.gateway]</span> <span class="hljs-setting">rgw_frontends = <span class="hljs-value">fastcgi</span></span> <span class="hljs-setting">host = <span class="hljs-value">{hostname}</span></span> <span class="hljs-setting">keyring = <span class="hljs-value">/etc/ceph/ceph.client.radosgw.keyring</span></span> <span class="hljs-setting">rgw_socket_path = <span class="hljs-value">/var/run/ceph/ceph.radosgw.gateway.sock</span></span> <span class="hljs-setting">log_file = <span class="hljs-value">/var/log/ceph/radosgw.log</span></span> <span class="hljs-setting">rgw_print_continue = <span class="hljs-value"><span class="hljs-keyword">false</span></span></span> <span class="hljs-setting">rgw_content_length_compat = <span class="hljs-value"><span class="hljs-keyword">true</span></span></span></code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li></ul>
配置nginx服务,在/etc/nginx/nginx.conf
文件的http段下添加如下内容:
<code class="hljs php has-numbering">http { server { listen <span class="hljs-number">80</span> <span class="hljs-keyword">default</span>; server_name {hostname}; location / { fastcgi_pass_header Authorization; fastcgi_pass_request_headers on; fastcgi_param QUERY_STRING <span class="hljs-variable">$query_string</span>; fastcgi_param REQUEST_METHOD <span class="hljs-variable">$request_method</span>; fastcgi_param CONTENT_LENGTH <span class="hljs-variable">$content_length</span>; fastcgi_param CONTENT_LENGTH <span class="hljs-variable">$content_length</span>; <span class="hljs-keyword">if</span> (<span class="hljs-variable">$request_method</span> = PUT) { rewrite ^ /PUT<span class="hljs-variable">$request_uri</span>; } <span class="hljs-keyword">include</span> fastcgi_params; fastcgi_pass unix:/<span class="hljs-keyword">var</span>/run/ceph/ceph.radosgw.gateway.sock; } location /PUT/ { internal; fastcgi_pass_header Authorization; fastcgi_pass_request_headers on; <span class="hljs-keyword">include</span> fastcgi_params; fastcgi_param QUERY_STRING <span class="hljs-variable">$query_string</span>; fastcgi_param REQUEST_METHOD <span class="hljs-variable">$request_method</span>; fastcgi_param CONTENT_LENGTH <span class="hljs-variable">$content_length</span>; fastcgi_param CONTENT_TYPE <span class="hljs-variable">$content_type</span>; fastcgi_pass unix:/<span class="hljs-keyword">var</span>/run/ceph/ceph.radosgw.gateway.sock; } }</code><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul>
注意: fastcgi_pass 指向的路径需要与ceph.conf中配置的路径一致
让nginx配置生效:nginx -s reload
启动rgw实例
通过上面的安装->创建用户->创建存储池->配置
过程,rgw也就准备就绪了,可以通过下面的命令启动实例:
<code class="hljs vala has-numbering"><span class="hljs-comment">//radosgw -c {conf_file} -n {rgw-name}</span> <span class="hljs-preprocessor">#> radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway</span></code><ul class="pre-numbering" style=""><li>1</li><li>2</li></ul><ul class="pre-numbering" style=""><li>1</li><li>2</li></ul>
测试
ceph rgw的测试方式有:s3cmd, cosbench,也可以通过Python库boto自己写测试程序。个人感觉cosbench很不错,大家可以试试。
rgw多实例
有时为了提高rgw的并发能力,需要部署多个rgw实例。其实也很简单,在多个节点上部署多个rgw实例:只需要安装rgw包,并将ceph.conf文件,密钥文件,前端配置文件拷贝到相应的节点,然后启动实例就好。
RadosGW分析。
一、RadosGW的main()函数解析。
RadosGW的main()函数位于rgw_main.cc中,该函数是RadosGW的入口函数。这里主要解析和RadosGW相关的内容,尽量省略WebServer的内容。
main()函数解析如下:
1、根据g_conf->rgw_frontends得到默认支持的前端WebServers,目前默认支持的前端WebServer包括fastcgi和civetweb。对于每一个支持的前端都有一个RGWFrontendConfig类与之对应,最后将WebServer的名字和对应的RGWFrontendConfig类设置到fe_map集合中;
2、调用global_init()函数做初始化操作;
3、调用rgw_tool_init()函数从g_conf->rgw_mime_type_file指定的文件中读取该文件中的数据并初始化;
4、调用curl_global_init()函数初始化libcurl,由此可以看出rgw使用curl进行网络数据传输;
5、调用FCGX_Init()初始化WebServer;
6、调用RGWStoreManager::get_storage()函数初始化librados对象,即:以后对Ceph集群的操作都得通过该librados对象;
7、调用rgw_rest_init()初始化rest部分;
8、调用rgw_user_init()函数初始化radosgw中关于用户部分的类RGWUserMetadataHandler,且将该类注册到store->meta_mgr中;
9、调用rgw_buckt_init()函数初始化radosgw中关于bucket部分的类RGWBucketMetadataHandler和RGWBucketInstanceMetadataHandler,且将这两个类注册到store->meta_mgr中;
10、对于兼容S3的API来说,使用RGWRESTMgr_S3类作为radosgw中rest接口的默认处理类;
11、对于支持admin的操作来说,注册管理员操作对应的类RGWRESTMgr_Usage/RGWRESTMgr_User/RGWRESTMgr_Bucket/RGWRESTMgr_Metadata/RGWRESTMgr_Log/RGWRESTMgr_Opstate/RGWRESTMgr_ReplicaLog/RGWRESTMgr_Config/RGWRESTMgr_Realm;
12、注册信号处理函数,处理诸如SIGUP/SIGTERM等信号;
13、初始化前端WebServer并运行前端WebServer;
14、调用wait_shutdown()函数等待radosgw进程执行完毕;
实际的Web请求处理位于rgw_process.cc::process_request()函数内,该函数负责处理Web前端的数据读写请求。
二、process_request()函数分析流程。
int process_request(RGWRados* store, RGWREST* rest, RGWRequest* req, RGWStreamIO* client_io, OpsLogSocket* olog)
参数说明:
store:在rgw_main.cc中的main()函数中调用的RGWStoreManager::get_storage()的返回值;
rest:在rgw_main.cc中的main()函数中调用rest.register_default_mgr(new RGWRESTMgr_S3())的返回值,也就是说rest是类RGWRESTMgr_S3的实例;
处理流程:
1)rest->get_handler():得到RGWHandler_REST类实例,包括:RGWHandler_REST_Service_S3/RGWHandler_REST_Bucket_S3/RGWHandler_REST_Obj_S3;
2)handler->get_op():根据request的op得到对应的操作处理类,如:GET/PUT/DELETE/POST等;
3)handler->authorize():执行认证操作。具体调用流程如下:
handler->authorize()
|__RGWHandler_REST_S3::authorize()
|__RGW_Auth_S3::authorize()
|__RGW_Auth_S3::authorize_v4()/RGW_Auth_S3::authorize_v2()/
根据客户端的请求头中是否包含http_auth以及http_auth的内容,确定采用S3v4或S3v2进行认证;
4)handler->postauth_init():检查bucket以及object名字的有效性。
handler->postauth_init()
|__rgw_parse_url_bucket() 解析出tenant名称和bucket名称
|__validate_tenant_name() 校验tenant名称的有效性
|__validate_s3_bucket_name() 校验S3bucket名称的有效性
|__validate_object_name() 校验对象名称的有效性
5)handler->init_permissions():读取并创建bucket有效性检查信息。
handler->init_permissions()
|__rgw_build_bucket_policies()
|__初始化req_state->bucket_acl为RGWAccessControlPolicy_S3类实例
|__RGWRados::get_bucket_info()/RGWRados::get_bucket_instance_info()获取bucket info信息
|__read_policy()读取bucket的ACL信息,bucket的属性信息等
|__RGWRados::get_zonegroup()读取bucket的zone信息
6)handler->retarget() 重新定位和计算存储对象,这个对于某些website访问的方式有效。
7)handler->read_permissions() 获取bucket以及object的ACL信息
8)op->init_processing()
op->init_processing()
|__RGWOp::init_processing()
|__RGWOp::init_quota()
|__获取bucket_quota
|__获取user_quota
9)op->verify_op_mask()
op->verify_op_mask()
|__RGWOp::verify_op_mask() 检查操作的op_mask和用户的op_mask是否一致
10)op->verify_permission()
op->verify_permission()
|__调用RGWOp子类对应的verify_permission()函数,由RGWOp子类自行判断权限是否满足
11)op->verify_params()
op->verify_params()
|__RGWPutObj_ObjStore::verify_params()/RGWPostObj_ObjStore::verify_params() 检查上传的数据大小是否超过g_conf->rgw_max_put_size值
12)op->pre_exec()
op->pre_exec()
|__调用RGWOp子类对应的pre_exec()函数,由RGWOp子类自行处理
13)op->execute()
op->execute()
|__调用RGWOp子类对应的execute()函数,由RGWOp子类自行处理
14)op->complete()
op->complete()
|__RGWOp::complete()
|__调用RGWOp子类对应的send_response()函数,由RGWOp子类自行处理
因此在实际处理对象存储请求时,需要RGWOp子类处理的主要函数包括:
1)RGWOp::verify_permission()
2)RGWOp::pre_exec()
3)RGWOp::execute()
4)RGWOp::send_response()