ceph命令执行源码IO路径

1.命令入口：脚本 /usr/bin/ceph

/usr/bin/ceph 在源代码中的文件是ceph.in

...
import argparse	
import rados
...

from ceph_argparse import \
    concise_sig, descsort_key, parse_json_funcsigs, \
    validate_command, find_cmd_target, \
    json_command, run_in_thread, Flag
...

上述导入了参数解析模块argparse ，rados模块，和ceph命令参数解析模块ceph_argparse
argparse模块的使用可参考这个：argparse模块的使用

获取rados client句柄：

cluster_handle = run_in_thread(rados.Rados,
                                       name=name, clustername=clustername,
                                       conf_defaults=conf_defaults,
                                       conffile=conffile)

执行命令json_command：

for target in targets:		#targets为解析完的参数
        # prettify?  prefix output with target, if there was a wildcard used
        prefix = ''
        suffix = ''
        if not parsed_args.output_file and len(targets) > 1:
            prefix = '{0}.{1}: '.format(*target)
            suffix = '\n'

        ret, outbuf, outs = json_command(cluster_handle, target=target,
                                         prefix='get_command_descriptions')	#执行命令

2.调用librados和LibCephFS进行命令分发

然后调用send_command函数
根据target[0]判断是哪个daemon的命令，然后调用librados库中的函数
ceph_argparse.py中导入LibCephFS库，用于调用mds命令

from cephfs import LibCephFS

if target[0] == 'osd':
           
            ret, outbuf, outs = run_in_thread(
                cluster.osd_command, osdid, cmd, inbuf, timeout=timeout)

        elif target[0] == 'mgr':
            ret, outbuf, outs = run_in_thread(
                cluster.mgr_command, cmd, inbuf, timeout=timeout)

        elif target[0] == 'pg':
            ret, outbuf, outs = run_in_thread(
                cluster.pg_command, pgid, cmd, inbuf, timeout=timeout)

        elif target[0] == 'mon':
                ret, outbuf, outs = run_in_thread(
                    cluster.mon_command, cmd, inbuf, timeout=timeout, target=target[1])
        elif target[0] == 'mds':
            ret, outbuf, outs = \
                filesystem.mds_command(mds_spec, cmd, inbuf)
            filesystem.shutdown()
        else:
            raise ArgumentValid("Bad target type '{0}'".format(target[0]))

rados库函数：

在RadosClient.cc实现：

  int mon_command(const vector<string>& cmd, const bufferlist &inbl,
	          bufferlist *outbl, string *outs);
  void mon_command_async(const vector<string>& cmd, const bufferlist &inbl,
                         bufferlist *outbl, string *outs, Context *on_finish);
  int mon_command(int rank,
		  const vector<string>& cmd, const bufferlist &inbl,
	          bufferlist *outbl, string *outs);
  int mon_command(string name,
		  const vector<string>& cmd, const bufferlist &inbl,
	          bufferlist *outbl, string *outs);
  int mgr_command(const vector<string>& cmd, const bufferlist &inbl,
	          bufferlist *outbl, string *outs);
  int osd_command(int osd, vector<string>& cmd, const bufferlist& inbl,
                  bufferlist *poutbl, string *prs);
  int pg_command(pg_t pgid, vector<string>& cmd, const bufferlist& inbl,
	         bufferlist *poutbl, string *prs);

这里对应的命令通过每个daemon各自的client发送到对应的daemon去执行，
mon_command发送给mon，mgr_command发给mgr，osd发给osd
pg发送Objecter

mgr_command函数为例：

int librados::RadosClient::mgr_command(const vector<string>& cmd,
				       const bufferlist &inbl,
				       bufferlist *outbl, string *outs)
{
  std::lock_guard l(lock);

  C_SaferCond cond;
  int r = mgrclient.start_command(cmd, inbl, outbl, outs, &cond);
  if (r < 0)
    return r;

  lock.Unlock();
  if (conf->rados_mon_op_timeout) {
    r = cond.wait_for(conf->rados_mon_op_timeout);
  } else {
    r = cond.wait();
  }
  lock.Lock();

  return r;
}

start_command函数中会把命令封装为mgr命令消息发送给mgr，mgr消息调度去解析执行命令