HBase 0.99 源代码分析 - Master启动过程(2)

下面继续从HMaster.finishActiveMasterInitialization方法开始分析HBase 0.99 Master启动过程。HMaster.finishActiveMasterInitialization方法主要完成如下功能:

1. 初始化Master功能组件:file system manager, server manager, assignment manager, region server tracker 等。

2. 启动服务线程:balancer,catalog janitor, executor services 等。

3. 在ZooKeeper中设置集群状态为UP

4. 等待Region server登录

5. 如果需要的话,拆分Log, 执行数据恢复

6. 分配meta, namespace和region

7. 处理集群启动或master failover

8. 调用coprocessor

HMaster.finishActiveMasterInitialization方法源代码比较长,下面我们逐步进行分析。

  private void finishActiveMasterInitialization(MonitoredTask status)
      throws IOException, InterruptedException, KeeperException, CoordinatedStateException {

    isActiveMaster = true;

    /*
     * We are active master now... go initialize components we need to run.
     * Note, there may be dross in zk from previous runs; it'll get addressed
     * below after we determine if cluster startup or failover.
     */

    status.setStatus("Initializing Master file system");

    this.masterActiveTime = System.currentTimeMillis();
    // TODO: Do this using Dependency Injection, using PicoContainer, Guice or Spring.
    this.fileSystemManager = new MasterFileSystem(this, this);

    // publish cluster ID
    status.setStatus("Publishing Cluster ID in ZooKeeper");
    ZKClusterId.setClusterId(this.zooKeeper, fileSystemManager.getClusterId());
    this.serverManager = createServerManager(this, this);

    synchronized (this) {
      if (shortCircuitConnection == null) {
        shortCircuitConnection = createShortCircuitConnection();
        metaTableLocator = new MetaTableLocator();
      }
    }

    // Invalidate all write locks held previously
    this.tableLockManager.reapWriteLocks();

    this.tableStateManager = new TableStateManager(this);
    this.tableStateManager.start();

    status.setStatus("Initializing ZK system trackers");
    initializeZKBasedSystemTrackers();

    // initialize master side coprocessors before we start handling requests
    status.setStatus("Initializing master coprocessors");
    this.cpHost = new MasterCoprocessorHost(this, this.conf);

上面代码执行了如下功能:

1. 设置本Master的isActiveMaster标志为true

2. 设置masterActiveTime为当前系统时间

3. 初始化filesystem manager类MasterFileSystem,该类包装了Master与底层文件系统交互时的一些操作,例如Log分割,检查文件系统状态等。初始化时读取配置并设置HBase跟路径,临时路径等,并初始化SplitLogManager

4. 向ZooKeeper设置cluster ID.

5. 创建server manager, server manager管理所有Region server, 包括在线和死亡的Region server, 处理Region server的启动,关闭和死亡

6. 创建到自身Regionserver的HConnection连接,存储到成员变量shortCircuitConnection中,可通过该连接绕过RPC, 执行HConnection客户端请求。

7. 初始化metaTableLocator对象, 该对象用于定位meta 表hbase:meta

8. 释放所有tablewrite lock

9. 初始化tablestate manager对象,该对象用于管理表的状态

10. 初始化所有基于ZooKeeper的system tracker, 包括:

  1). load balancer:默认实现类为 StochasticLoadBalancer, load balancer负责管理region 在regionserver之间的迁移方案。可以通过配置项hbase.master.loadbalancer.class更改实现类。对于默认实现类StochasticLoadBalancer,可以通过如下配置项控制其行为:

    hbase.master.balancer.stochastic.regionLoadCost

    hbase.master.balancer.stochastic.moveCost

    hbase.master.balancer.stochastic.tableLoadCost

    hbase.master.balancer.stochastic.localityCost

    hbase.master.balancer.stochastic.memstoreSizeCost

    hbase.master.balancer.stochastic.storefileSizeCost    

    hbase.master.balancer.stochastic.maxMoveRegions

    hbase.master.balancer.stochastic.stepsPerRegion

    hbase.master.balancer.stochastic.maxSteps

  2). load balancertracker: 基于ZooKeeper跟踪load balancer状态。

  3). assignmentmanager:负责region的分配。

  4). region servertracker: 通过ZooKeeper跟踪在线的region server

  5). draining server tracker:通过ZooKeeper管理drainingregion server。 在大型集群中, 用户有时需要把一部分region server服务器一起停止服务迁出HBase集群,这些将要被迁出的region server称为draining region server。被迁走的region server上的region需要被分配到其他region server上,为避免把一个draining server上的region迁到另一个draining server上,hbase会一次把所有draining server加入draining server列表。

  6). snapshot manager:负责管理 HBase快照功能,包括创建快照,恢复快照。快照是HBase从0.94.6开始提供的功能,创建快照时并不复制数据,因此速度很快,对region server性能基本没有影响。

11. 初始化master协处理器(coprocessor) MasterCoprocessorHost,负责管理所有master相关操作的协处理器。

继续看finishActiveMasterInitialization接下来的代码

    // start up all service threads.
    status.setStatus("Initializing master service threads");
    startServiceThreads();

    // Wake up this server to check in
    sleeper.skipSleepCycle();

    // Wait for region servers to report in
    this.serverManager.waitForRegionServers(status);
    // Check zk for region servers that are up but didn't register
    for (ServerName sn: this.regionServerTracker.getOnlineServers()) {
      // The isServerOnline check is opportunistic, correctness is handled inside
      if (!this.serverManager.isServerOnline(sn)
          && serverManager.checkAndRecordNewServer(sn, ServerLoad.EMPTY_SERVERLOAD)) {
        LOG.info("Registered server found up in zk but who has not yet reported in: " + sn);
      }
    }

上面代码执行如下功能:

1. 调用startServiceThreads方法,该方法首先按下表中的个数启动如下服务线程:

服务线程类型

配置项

默认线程个数

MASTER_OPEN_REGION

hbase.master.executor.openregion.threads

5

MASTER_CLOSE_REGION

hbase.master.executor.closeregion.threads

5

MASTER_SERVER_OPERATIONS

hbase.master.executor.serverops.threads

5

MASTER_META_SERVER_OPERATIONS

hbase.master.executor.serverops.threads

5

M_LOG_REPLAY_OPS

hbase.master.executor.logreplayops.threads

10

MASTER_TABLE_OPERATIONS

不可配置

1

 

然后读取配置项hbase.master.cleaner.interval作为log cleaner时间间隔,默认为1分钟,启动log cleaner线程,该线程每到一个时间间隔清理oldWALs目录下旧的HLog.

然后启动HFile archive清理线程,清理archive目录下的HFile archive.

置服务线程启动标志serviceStarted为true.

2. 唤醒当前server允许check in.

3. 等待region server check in. 满足如下条件之一则等待完毕:

   - master 停止

   -达到配置项hbase.master.wait.on.regionservers.maxtostart指定的region server个数

   -达到配置项hbase.master.wait.on.regionservers.mintostart指定的region server个数并且  在配置项hbase.master.wait.on.regionservers.interval指定的间隔内没有新的region server并且达到配置项hbase.master.wait.on.regionservers.timeout超时时间

4. 获取在线region server列表并进行注册。

继续看finishActiveMasterInitialization:

    // get a list for previously failed RS which need log splitting work
    // we recover hbase:meta region servers inside master initialization and
    // handle other failed servers in SSH in order to start up master node ASAP
    Set previouslyFailedServers = this.fileSystemManager
        .getFailedServersFromLogFolders();

    // remove stale recovering regions from previous run
    this.fileSystemManager.removeStaleRecoveringRegionsFromZK(previouslyFailedServers);

    // log splitting for hbase:meta server
    ServerName oldMetaServerLocation = metaTableLocator.getMetaRegionLocation(this.getZooKeeper());
    if (oldMetaServerLocation != null && previouslyFailedServers.contains(oldMetaServerLocation)) {
      splitMetaLogBeforeAssignment(oldMetaServerLocation);
      // Note: we can't remove oldMetaServerLocation from previousFailedServers list because it
      // may also host user regions
    }
    Set previouslyFailedMetaRSs = getPreviouselyFailedMetaServersFromZK();
    // need to use union of previouslyFailedMetaRSs recorded in ZK and previouslyFailedServers
    // instead of previouslyFailedMetaRSs alone to address the following two situations:
    // 1) the chained failure situation(recovery failed multiple times in a row).
    // 2) master get killed right before it could delete the recovering hbase:meta from ZK while the
    // same server still has non-meta wals to be replayed so that
    // removeStaleRecoveringRegionsFromZK can't delete the stale hbase:meta region
    // Passing more servers into splitMetaLog is all right. If a server doesn't have hbase:meta wal,
    // there is no op for the server.
    previouslyFailedMetaRSs.addAll(previouslyFailedServers);

getFailedServersFromLogFolders方法找出之前正在recovering但已down掉,却在WALs目录下仍然遗留log的server

removeStaleRecoveringRegionsFromZK从ZooKeerper删除/hbase/recovering-regions/下的不在线的节点,即上面找出的server. 此处并不是删除log, 而是删除ZooKeeper中down掉的server信息,后面会对每个region重新分配region server

如果meta表被分配给down掉的server,先分离meta log

从ZooKeeper获取之前down掉的meta server,合并down掉的meta server和region server列表。

    this.initializationBeforeMetaAssignment = true;

    // Wait for regionserver to finish initialization.
    if (BaseLoadBalancer.tablesOnMaster(conf)) {
      waitForServerOnline();
    }

    //initialize load balancer
    this.balancer.setClusterStatus(getClusterStatus());
    this.balancer.setMasterServices(this);
    this.balancer.initialize();

    // Check if master is shutting down because of some issue
    // in initializing the regionserver or the balancer.
    if(isStopped()) return;

    // Make sure meta assigned before proceeding.
    status.setStatus("Assigning Meta Region");
    assignMeta(status, previouslyFailedMetaRSs);
    // check if master is shutting down because above assignMeta could return even hbase:meta isn't
    // assigned when master is shutting down
    if(isStopped()) return;

等待region server初始化完成。

初始化load balancer。

分配meta region到region server。

    status.setStatus("Submitting log splitting work for previously failed region servers");
    // Master has recovered hbase:meta region server and we put
    // other failed region servers in a queue to be handled later by SSH
    for (ServerName tmpServer : previouslyFailedServers) {
      this.serverManager.processDeadServer(tmpServer, true);
    }

    // Fix up assignment manager status
    status.setStatus("Starting assignment manager");
    this.assignmentManager.joinCluster();

    //set cluster status again after user regions are assigned
    this.balancer.setClusterStatus(getClusterStatus());

    // Start balancer and meta catalog janitor after meta and regions have
    // been assigned.
    status.setStatus("Starting balancer and catalog janitor");
    this.clusterStatusChore = new ClusterStatusChore(this, balancer);
    Threads.setDaemonThreadRunning(clusterStatusChore.getThread());
    this.balancerChore = new BalancerChore(this);
    Threads.setDaemonThreadRunning(balancerChore.getThread());
    this.catalogJanitorChore = new CatalogJanitor(this, this);
    Threads.setDaemonThreadRunning(catalogJanitorChore.getThread());

processDeadServer会调用ServerShutdownHandler类进行日志分离。

调用assignment manager进行用户region 分配。

    status.setStatus("Starting namespace manager");
    initNamespace();

    status.setStatus("Starting quota manager");
    initQuotaManager();

    if (this.cpHost != null) {
      try {
        this.cpHost.preMasterInitialization();
      } catch (IOException e) {
        LOG.error("Coprocessor preMasterInitialization() hook failed", e);
      }
    }

    status.markComplete("Initialization successful");
    LOG.info("Master has completed initialization");
    initialized = true;
    // clear the dead servers with same host name and port of online server because we are not
    // removing dead server with same hostname and port of rs which is trying to check in before
    // master initialization. See HBASE-5916.
    this.serverManager.clearDeadServersWithSameHostNameAndPortOfOnlineServer();

    if (this.cpHost != null) {
      // don't let cp initialization errors kill the master
      try {
        this.cpHost.postStartMaster();
      } catch (IOException ioe) {
        LOG.error("Coprocessor postStartMaster() hook failed", ioe);
      }
    }

启动namespace manager和quota manager。

最后调用master启动完成协处理器。

至此master启动完成。

发布了11 篇原创文章 · 获赞 8 · 访问量 2万+
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览