PartitioinLeaderSelector分析

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/zhanglh046/article/details/72822066

PartitionLeaderSelector主要负责分区leader副本的选举。

1 NoOpLeaderSelector

def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
    warn("I should never have been asked to perform leader election,returning the current LeaderAndIsr and replica assignment.")
    (currentLeaderAndIsr, controllerContext.partitionReplicaAssignment(topicAndPartition))
  }

2 OfflinePartitionLeaderSelector

从该分区的AR副本集中过滤出可用的ISR列表

如果 ISR至少有一个可用副本,则从ISR列表中选举出一个副本作为leader

如果 ISR中没有存活副本,而且如果AR中没有存活的副本抛出异常

如果ISR中没有存活副本,但是AR副本集中有副本,那么就把AR第一个副本作为leader

def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
  controllerContext.partitionReplicaAssignment.get(topicAndPartition) match {
    case Some(assignedReplicas) =>
      // 从该分区的AR副本集中过滤出可用的副本
      val liveAssignedReplicas = assignedReplicas.filter(r => controllerContext.liveBrokerIds.contains(r))
      // 从传递进来的ISR列表过滤出可用的副本
      val liveBrokersInIsr = currentLeaderAndIsr.isr.filter(r => controllerContext.liveBrokerIds.contains(r))
      val currentLeaderEpoch = currentLeaderAndIsr.leaderEpoch
      val currentLeaderIsrZkPathVersion = currentLeaderAndIsr.zkVersion
      val newLeaderAndIsr =
        // 如果 ISR中没有存活副本
        if (liveBrokersInIsr.isEmpty) {
          if (!LogConfig.fromProps(config.originals, AdminUtils.fetchEntityConfig(controllerContext.zkUtils,
            ConfigType.Topic, topicAndPartition.topic)).uncleanLeaderElectionEnable) {
            throw new NoReplicaOnlineException(("No broker in ISR for partition " +
              "%s is alive. Live brokers are: [%s],".format(topicAndPartition, controllerContext.liveBrokerIds)) +
              " ISR brokers are: [%s]".format(currentLeaderAndIsr.isr.mkString(",")))
          }
          debug("No broker in ISR is alive for %s. Pick the leader from the alive assigned replicas: %s"
            .format(topicAndPartition, liveAssignedReplicas.mkString(",")))
          // 而且如果AR中没有存活的副本
          if (liveAssignedReplicas.isEmpty) {
            throw new NoReplicaOnlineException(("No replica for partition " +
              "%s is alive. Live brokers are: [%s],".format(topicAndPartition, controllerContext.liveBrokerIds)) +
              " Assigned replicas are: [%s]".format(assignedReplicas))
          } else {// 如果AR中有存活的副本
            ControllerStats.uncleanLeaderElectionRate.mark()
            // AR的存活的副本中取出第一个副本作为leader,这样的话就是有数据丢失的风险
            val newLeader = liveAssignedReplicas.head
            warn("No broker in ISR is alive for %s. Elect leader %d from live brokers %s. There's potential data loss."
              .format(topicAndPartition, newLeader, liveAssignedReplicas.mkString(",")))
            new LeaderAndIsr(newLeader, currentLeaderEpoch + 1, List(newLeader), currentLeaderIsrZkPathVersion + 1)
          }
        } else {// 如果 ISR有存活副本
          // AR存活的副本列表过滤出ISR存活的副本列表
          val liveReplicasInIsr = liveAssignedReplicas.filter(r => liveBrokersInIsr.contains(r))
          // ISR中存活的副本列表第一个作为新的leader
          val newLeader = liveReplicasInIsr.head
          debug("Some broker in ISR is alive for %s. Select %d from ISR %s to be the leader."
            .format(topicAndPartition, newLeader, liveBrokersInIsr.mkString(",")))
          new LeaderAndIsr(newLeader, currentLeaderEpoch + 1, liveBrokersInIsr.toList, currentLeaderIsrZkPathVersion + 1)
        }
      info("Selected new leader and ISR %s for offline partition %s".format(newLeaderAndIsr.toString(), topicAndPartition))
      (newLeaderAndIsr, liveAssignedReplicas)
    case None =>
      throw new NoReplicaOnlineException("Partition %s doesn't have replicas assigned to it".format(topicAndPartition))
  }
}

 

3 ReassignedPartitionLeaderSelector

获取正在重新分配的副本,然后从可用ISR列表中选举出一个leader,然后将当前的ISR更新为新ISR,然后将重新分配非副本集合作为接受LeaderAndIsr请求的副本集

def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
  // ControllerContext获取该分区正在重新分配的副本
  val reassignedInSyncReplicas = controllerContext.partitionsBeingReassigned(topicAndPartition).newReplicas
  val currentLeaderEpoch = currentLeaderAndIsr.leaderEpoch
  val currentLeaderIsrZkPathVersion = currentLeaderAndIsr.zkVersion
  // 过滤出当前可用的副本且ISR列表包含该副本
  val aliveReassignedInSyncReplicas = reassignedInSyncReplicas.filter(r => controllerContext.liveBrokerIds.contains(r) &&
                                                                           currentLeaderAndIsr.isr.contains(r))
  val newLeaderOpt = aliveReassignedInSyncReplicas.headOption
  newLeaderOpt match {
    case Some(newLeader) => (new LeaderAndIsr(newLeader, currentLeaderEpoch + 1, currentLeaderAndIsr.isr,
      currentLeaderIsrZkPathVersion + 1), reassignedInSyncReplicas)
    case None =>
      reassignedInSyncReplicas.size match {
        case 0 =>
          throw new NoReplicaOnlineException("List of reassigned replicas for partition " +
            " %s is empty. Current leader and ISR: [%s]".format(topicAndPartition, currentLeaderAndIsr))
        case _ =>
          throw new NoReplicaOnlineException("None of the reassigned replicas for partition " +
            "%s are in-sync with the leader. Current leader and ISR: [%s]".format(topicAndPartition, currentLeaderAndIsr))
      }
  }
}

 

4 PreferredReplicaPartitionLeaderSelector

先取出当前分区分配的所有副本(AR),可能包括不可用,然后取出第一个副本,如果第一个副本就是leader抛出异常,否则如果该副本所在broker是存活的且它在ISR列表中,就把他作为leader,否则抛出异常

def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
  // 获取该分区的AR副本集
  val assignedReplicas = controllerContext.partitionReplicaAssignment(topicAndPartition)
  // 获取副本集第一个副本
  val preferredReplica = assignedReplicas.head
  // 检查该副本是不是leader,如果是leader抛出异常
  val currentLeader = controllerContext.partitionLeadershipInfo(topicAndPartition).leaderAndIsr.leader
  if (currentLeader == preferredReplica) {
    throw new LeaderElectionNotNeededException("Preferred replica %d is already the current leader for partition %s"
                                                 .format(preferredReplica, topicAndPartition))
  } else {
    info("Current leader %d for partition %s is not the preferred replica.".format(currentLeader, topicAndPartition) +
      " Trigerring preferred replica leader election")
    // 在检查该副本是不是可用的且它在ISR列表中
    if (controllerContext.liveBrokerIds.contains(preferredReplica) && currentLeaderAndIsr.isr.contains(preferredReplica)) {
      (new LeaderAndIsr(preferredReplica, currentLeaderAndIsr.leaderEpoch + 1, currentLeaderAndIsr.isr,
        currentLeaderAndIsr.zkVersion + 1), assignedReplicas)
    } else {
      throw new StateChangeFailedException("Preferred replica %d for partition ".format(preferredReplica) +
        "%s is either not alive or not in the isr. Current leader and ISR: [%s]".format(topicAndPartition, currentLeaderAndIsr))
    }
  }
}

 

5 ControlledShutdownLeaderSelector

将ISR中处于关闭状态的副本从集合中去除掉,返回一个新新的ISR集合,然后选取第一个副本作为leader,然后令当前AR作为接收LeaderAndIsr请求的副本

 

def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
  val currentLeaderEpoch = currentLeaderAndIsr.leaderEpoch
  val currentLeaderIsrZkPathVersion = currentLeaderAndIsr.zkVersion
  // 获取leader
  val currentLeader = currentLeaderAndIsr.leader
  // 获取该分区的AR副本集
  val assignedReplicas = controllerContext.partitionReplicaAssignment(topicAndPartition)
  // 获取可用或者正处于关闭的broker
  val liveOrShuttingDownBrokerIds = controllerContext.liveOrShuttingDownBrokerIds

  val liveAssignedReplicas = assignedReplicas.filter(r => liveOrShuttingDownBrokerIds.contains(r))
  // 过滤掉正处于关闭的副本
  val newIsr = currentLeaderAndIsr.isr.filter(brokerId => !controllerContext.shuttingDownBrokerIds.contains(brokerId))
  liveAssignedReplicas.find(newIsr.contains) match {
    case Some(newLeader) =>
      debug("Partition %s : current leader = %d, new leader = %d".format(topicAndPartition, currentLeader, newLeader))
      (LeaderAndIsr(newLeader, currentLeaderEpoch + 1, newIsr, currentLeaderIsrZkPathVersion + 1), liveAssignedReplicas)
    case None =>
      throw new StateChangeFailedException(("No other replicas in ISR %s for %s besides" +
        " shutting down brokers %s").format(currentLeaderAndIsr.isr.mkString(","), topicAndPartition, controllerContext.shuttingDownBrokerIds.mkString(",")))
  }
}

 

 

分析分析 大家看看

06-01

这是什么程序 有没有错误 能够完全运行吗>rnrnrn#include "stdio.h"rn#include "stdlib.h"rn#include "string.h"rn#define MAX 32767rntypedef struct node rnrn int address,size;rn struct node *next;rnRECT;rnRECT *assignment(RECT *head,int application);rnvoid acceptment1(RECT *head,RECT *back1);rnvoid acceptment2(RECT *head,RECT *back1) ;rnint backcheck(RECT *head,RECT *back1);rnvoid print(RECT *head);rnRECT *head,*back,*assign1,*p;rnint application1,maxblocknum;rnchar way;rn/*主函数*/rnmain()rnrn char choose[10];rn int check;rn head=malloc(sizeof(RECT)); rn p=malloc(sizeof(RECT));rn head->size=MAX;rn head->address=0;rn head->next=p;rn maxblocknum=1;rn p->size=MAX;rn p->address=0;rn p->next=NULL;rn print(head); rn printf("Enter the way(best or first(b/f)\n");rn scanf("%c",&way);rn dorn printf("Enter the assign or accept(as/ac)\n");rn scanf("%s",choose); /*选择分配或回收*/rn if(strcmp(choose,"as")==0) rn rn printf("Input application:\n");rn scanf("%d",&application1);rn assign1=assignment(head,application1);rn if(assign1->address==-1)rn printf("Too large application!,assign fails!!\n\n");rn elsern printf("Success!!ADDRESS=%5d\n",assign1->address); rn print(head);rn rn elsern if(strcmp(choose,"ac")==0) /*回收*/rn rn back=malloc(sizeof(RECT));rn printf("Input Adress and Size!!\n");rn scanf("%d%d",&back->address,&back->size);rn check=backcheck(head,back); rn if(check==1)rn rn if(tolower(way)=='f')/*首先适应算法*/rn acceptment1(head,back); /*首先适应*/rn elsern acceptment2(head,back);/*最佳适应*/rn print(head);rn rn rn while(!strcmp(choose,"as")||!strcmp(choose,"ac"));rnrn/*分配函数*/rnRECT *assignment(RECT *head,int application)rnrn RECT *after,*before,*assign;rn assign=malloc(sizeof(RECT)); rn assign->size=application;rn assign->next=NULL;rn if(application>head->size||application<=0)rn assign->address=-1; /*申请无效*/rn elsern rn before=head;rn after=head->next;rn while(after->sizenext;rn after=after->next;rn rn if(after->size==application) /*结点大小等于申请大小则完全分配*/rn rn if(after->size==head->size)rn maxblocknum--;rn before->next=after->next;rn assign->address=after->address;rn free(after);rn rn elsern rn if(after->size==head->size) maxblocknum--;rn after->size=after->size-application; /*大于申请空间则截取相应大小分配*/rn assign->address=after->address+after->size;rn if(tolower(way)=='b')rn rn before->next=after->next;rn back=after;rn acceptment2(head,back);rn rn rn if(maxblocknum==0) rn rn before=head;rn head->size=0;rn maxblocknum=1;rn while(before!=NULL)rn rn if(before->size>head->size)rn rn head->size=before->size;rn maxblocknum=1;rn rn elsern if(before->size==head->size)rn maxblocknum++;rn before=before->next;rn rn rn rn assign1=assign;rn return assign1; rnrnvoid acceptment1(RECT *head,RECT *back1)/*首先适应*/rnrn RECT *before,*after;rn int insert;rn before=head;rn after=head->next;rn insert=0;rn while(!insert) rn rn if((after==NULL)||rn ((back1->address<=after->address)&&rn (back1->address>=before->address)))rn rn before->next=back1;rn back1->next=after;rn insert=1;rn rn elsern rn before=before->next;rn after=after->next;rn rn rn if(back1->address==before->address+before->size)rn rn before->size=before->size+back1->size;rn before->next=back1->next;rn free(back1);rn back1=before;rn rn if(after!=NULL&&(after->address==back1->address+back1->size))rn rn back1->size=back1->size+after->size;rn back1->next=after->next;rn free(after);rn rn if(head->sizesize) rn rn head->size=back1->size;rn maxblocknum=1;rn rn elsern if(head->size==back1->size)rn maxblocknum++;rnrn/*最佳适应*/rnvoid acceptment2(RECT *head,RECT *back1)rnrn RECT *before,*after;rn int insert ;rn insert=0;rn before=head;rn after=head->next;rn if(head->next==NULL) rn rn head->size=back1->size;rn head->next=back1;rn maxblocknum++;rn back1->next=NULL;rn rn elsern rn while(after!=NULL) rn if(back1->address==after->size+after->address)rn rn before->next=after->next;rn back->size=after->size+back1->size;rn free(after);rn after=NULL;rn rn elsern rn after=after->next;rn before=before->next;rn rn before=head;rn after=head->next;rn while(after!=NULL)rn if(after->address==back1->size+back1->address) rn rn back1->size=back1->size+after->size;rn before->next=after->next;rn free(after);rn after=NULL;rn rn elsern rn before=before->next;rn after=after->next;rn rn before=head;rn after=head->next;rn dorn if(after==NULL||(after->size>back1->size))rn rn before->next=back1;rn back1->next=after;rn insert=1;rn rn elsern rn before=before->next;rn after=after->next;rn rn while(!insert);rn if(head->sizesize) rn rn head->size=back1->size;rn maxblocknum++;rn rn elsern if(head->size==back1->size)rn maxblocknum++;rn rnrnrnvoid print(RECT *head) rnrn RECT *before,*after;rn int index,k;rn before=head->next;rn index=1;rn if(head->next==NULL)rn printf("NO part for assignment!!\n");rn elsern rn printf("*****index*******address********end*********size*****\n");rn while(before!=NULL)rn rn printf("----------------------------------------------------\n");rn printf(" %-13d%-13d%-13d%-13d\n",index,before->address,before->address+before->size-1,before->size);rn printf("----------------------------------------------------\n");rn index++;rn before=before->next;rn rn rnrnint backcheck(RECT *head,RECT *back1)rnrn RECT *before,*after;rn int check=1;rn if(back1->address<0||back1->size<0)rn check=0;rn before=head->next;rn while((before!=NULL)&&check)rn if(((back1->addressaddress)rn &&(back1->address+back1->size>before->address))rn ||((back1->address>=before->address)rn &&(back1->addressaddress+before->size)))rn check=0;rn elsern before=before->next;rn if(check==0)rn printf("Error input!!\n");rn return check; rn 论坛

没有更多推荐了,返回首页