10.深入k8s：调度的优先级及抢占机制源码分析

最新推荐文章于 2024-05-10 17:04:18 发布
最新推荐文章于 2024-05-10 17:04:18 发布
阅读量668 收藏
点赞数
原文链接：https://www.cnblogs.com/luozhiyun/p/13659035.html
版权

上一篇我们将了获取node成功的情况，如果是一个优先pod获取node失败，那么就会进入到抢占环节中，那么抢占环节k8s会做什么呢，抢占是如何发生的，哪些资源会被抢占这些都是我们这篇要研究的内容。

调度的优先级与抢占机制#
正常情况下，当一个 Pod 调度失败后，它就会被暂时“搁置”起来，直到 Pod 被更新，或者集群状态发生变化，调度器才会对这个 Pod 进行重新调度。但是我们可以通过PriorityClass优先级来避免这种情况。通过设置优先级一些优先级比较高的pod，如果pod 调度失败，那么并不会被”搁置”，而是会”挤走”某个 node 上的一些低优先级的 pod，这样就可以保证高优先级的 pod 调度成功。

要使用PriorityClass，首先我们要定义一个PriorityClass对象，例如：

Copy
apiVersion: v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: “This priority class should be used for XYZ service pods only.”
value越高则优先级越高；globalDefault 被设置为 true 的话，那就意味着这个 PriorityClass 的值会成为系统的默认值，如果是false则表示我们只希望声明使用该 PriorityClass 的 Pod 拥有值为 1000000 的优先级，而对于没有声明 PriorityClass 的 Pod 来说，它们的优先级就是 0。

Pod 就可以声明使用它了：

Copy
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:

  • name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
    priorityClassName: high-priority
    高优先级的 Pod 调度失败的时候，调度器的抢占能力就会被触发。调度器就会试图从当前集群里寻找一个节点，使得当这个节点上的一个或者多个低优先级 Pod 被删除后，待调度的高优先级 Pod 就可以被调度到这个节点上。

高优先级Pod进行抢占的时候会将pod的 nominatedNodeName 字段，设置为被抢占的 Node 的名字。然后，在下一周期中决定是不是要运行在被抢占的节点上，当这个Pod在等待的时候，如果有其他更高优先级的 Pod 也要抢占同一个节点，那么调度器就会清空原抢占者的 spec.nominatedNodeName 字段，从而允许更高优先级的抢占者执行抢占。

源码解析#
这里我依旧拿出这张图来进行讲解，上一篇我们将了获取node成功的情况，如果是一个优先pod获取node失败，那么就会进入到抢占环节中。

调度流程

通过上一篇的分析，我们知道，在scheduleOne方法中执行sched.Algorithm.Schedule会选择一个合适的node节点，如果获取node失败，那么就会进入到一个if逻辑中执行抢占。

代码路径：pkg/scheduler/scheduler.go

Copy
func (sched *Scheduler) scheduleOne(ctx context.Context) {
…
//为pod资源对象选择一个合适的节点
scheduleResult, err := sched.Algorithm.Schedule(schedulingCycleCtx, prof, state, pod)
//获取node失败，抢占逻辑
if err != nil {
//上面调用失败之后，下面会根据pod执行抢占
nominatedNode := “”
if fitError, ok := err.(*core.FitError); ok {
if !prof.HasPostFilterPlugins() {
klog.V(3).Infof(“No PostFilter plugins are registered, so no preemption will be performed.”)
} else {
result, status := prof.RunPostFilterPlugins(ctx, state, pod, fitError.FilteredNodesStatuses)
if status.Code() == framework.Error {
klog.Errorf(“Status after running PostFilter plugins for pod %v/%v: %v”, pod.Namespace, pod.Name, status)
} else {
klog.V(5).Infof(“Status after running PostFilter plugins for pod %v/%v: %v”, pod.Namespace, pod.Name, status)
}
//抢占成功后，将nominatedNodeName设置为被抢占的 Node 的名字，然后重新进入下一个调度周期
if status.IsSuccess() && result != nil {
nominatedNode = result.NominatedNodeName
}
}
metrics.PodUnschedulable(prof.Name, metrics.SinceInSeconds(start))
} else if err == core.ErrNoNodesAvailable {
metrics.PodUnschedulable(prof.Name, metrics.SinceInSeconds(start))
} else {
klog.ErrorS(err, “Error selecting node for pod”, “pod”, klog.KObj(pod))
metrics.PodScheduleError(prof.Name, metrics.SinceInSeconds(start))
}
sched.recordSchedulingFailure(prof, podInfo, err, v1.PodReasonUnschedulable, nominatedNode)
return
}
…
}
在这个方法里面RunPostFilterPlugins会执行具体的抢占逻辑，然后返回被抢占的node节点。抢占者并不会立刻被调度到被抢占的 node 上，调度器只会将抢占者的 status.nominatedNodeName 字段设置为被抢占的 node 的名字。然后，抢占者会重新进入下一个调度周期，在新的调度周期里来决定是不是要运行在被抢占的节点上，当然，即使在下一个调度周期，调度器也不会保证抢占者一定会运行在被抢占的节点上。

这样设计的一个重要原因是调度器只会通过标准的 DELETE API 来删除被抢占的 pod，所以，这些 pod 必然是有一定的“优雅退出”时间（默认是 30s）的。而在这段时间里，其他的节点也是有可能变成可调度的，或者直接有新的节点被添加到这个集群中来。

而在抢占者等待被调度的过程中，如果有其他更高优先级的 pod 也要抢占同一个节点，那么调度器就会清空原抢占者的 status.nominatedNodeName 字段，从而允许更高优先级的抢占者执行抢占，并且，这也使得原抢占者本身也有机会去重新抢占其他节点。

接着我们继续看，RunPostFilterPlugins会遍历所有的postFilterPlugins，然后执行runPostFilterPlugin方法：

Copy
func (f *frameworkImpl) RunPostFilterPlugins(ctx context.Context, state *framework.CycleState, pod *v1.Pod, filteredNodeStatusMap framework.NodeToStatusMap) (_ *framework.PostFilterResult, status *framework.Status) {
startTime := time.Now()
defer func() {
metrics.FrameworkExtensionPointDuration.WithLabelValues(postFilter, status.Code().String(), f.profileName).Observe(metrics.SinceInSeconds(startTime))
}()

statuses := make(framework.PluginToStatus)
//postFilterPlugins里面只有一个defaultpreemption
for _, pl := range f.postFilterPlugins {
	r, s := f.runPostFilterPlugin(ctx, pl, state, pod, filteredNodeStatusMap)
	if s.IsSuccess() {
		return r, s
	} else if !s.IsUnschedulable() {
		// Any status other than Success or Unschedulable is Error.
		return nil, framework.NewStatus(framework.Error, s.Message())
	}
	statuses[pl.Name()] = s
}

return nil, statuses.Merge()

}
根据我们上一节看的scheduler的初始化可以知道设置的PostFilter如下：

代码路径：pkg/scheduler/algorithmprovider/registry.go

Copy
PostFilter: &schedulerapi.PluginSet{
Enabled: []schedulerapi.Plugin{
{Name: defaultpreemption.Name},
},
},
可见，目前只有一个defaultpreemption来执行抢占逻辑，在postFilterPlugins循环里面会调用到runPostFilterPlugin然后运行defaultpreemption的PostFilter方法，最后执行到preempt执行具体抢占逻辑。

代码路径：pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go

Copy
func (pl *DefaultPreemption) PostFilter(…) (*framework.PostFilterResult, *framework.Status) {
…
//执行抢占
nnn, err := pl.preempt(ctx, state, pod, m)
…
return &framework.PostFilterResult{NominatedNodeName: nnn}, framework.NewStatus(framework.Success)
}
抢占的执行流程图如下：

抢占

代码路径：pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go

Copy
func (pl *DefaultPreemption) preempt(ctx context.Context, state *framework.CycleState, pod *v1.Pod, m framework.NodeToStatusMap) (string, error) {
cs := pl.fh.ClientSet()
ph := pl.fh.PreemptHandle()
//返回node列表
nodeLister := pl.fh.SnapshotSharedLister().NodeInfos()

pod, err := util.GetUpdatedPod(cs, pod)
if err != nil {
	klog.Errorf("Error getting the updated preemptor pod object: %v", err)
	return "", err
}

//确认抢占者是否能够进行抢占，如果对应的node节点上的pod正在优雅退出（Graceful Termination ），那么就不应该进行抢占
if !PodEligibleToPreemptOthers(pod, nodeLister, m[pod.Status.NominatedNodeName]) {
	klog.V(5).Infof("Pod %v/%v is not eligible for more preemption.", pod.Namespace, pod.Name)
	return "", nil
}

// 查找所有抢占候选者
candidates, err := FindCandidates(ctx, cs, state, pod, m, ph, nodeLister, pl.pdbLister)
if err != nil || len(candidates) == 0 {
	return "", err
}

//若有 extender 则执行
candidates, err = CallExtenders(ph.Extenders(), pod, nodeLister, candidates)
if err != nil {
	return "", err
}

// 查找最佳抢占候选者
bestCandidate := SelectCandidate(candidates)
if bestCandidate == nil || len(bestCandidate.Name()) == 0 {
	return "", nil
}

// 在抢占一个node之前做一些准备工作
if err := PrepareCandidate(bestCandidate, pl.fh, cs, pod); err != nil {
	return "", err
}

return bestCandidate.Name(), nil

}
preempt方法首先会去获取node列表，然后获取最新的要执行抢占的pod信息，接着分下面几步执行抢占：

调用PodEligibleToPreemptOthers方法，检查抢占者是否能够进行抢占，如果当前的pod已经抢占了一个node节点或者在被抢占node节点中有pod正在执行优雅退出，那么不应该执行抢占；
调用FindCandidates找到所有node中能被抢占的node节点，并返回候选列表以及node节点中需要被删除的pod；
若有 extender 则执行CallExtenders；
调用SelectCandidate方法在所有候选列表中找出最合适的node节点执行抢占；
调用PrepareCandidate方法删除被抢占的node节点中victim（牺牲者），以及清除NominatedNodeName字段信息；
PodEligibleToPreemptOthers

Copy
func PodEligibleToPreemptOthers(pod *v1.Pod, nodeInfos framework.NodeInfoLister, nominatedNodeStatus *framework.Status) bool {
if pod.Spec.PreemptionPolicy != nil && *pod.Spec.PreemptionPolicy == v1.PreemptNever {
klog.V(5).Infof(“Pod %v/%v is not eligible for preemption because it has a preemptionPolicy of %v”, pod.Namespace, pod.Name, v1.PreemptNever)
return false
}
//查看抢占者是否已经抢占过
nomNodeName := pod.Status.NominatedNodeName
if len(nomNodeName) > 0 {
if nominatedNodeStatus.Code() == framework.UnschedulableAndUnresolvable {
return true
}
//获取被抢占的node节点
if nodeInfo, _ := nodeInfos.Get(nomNodeName); nodeInfo != nil {
//查看是否存在正在被删除并且优先级比抢占者pod低的pod
podPriority := podutil.GetPodPriority(pod)
for _, p := range nodeInfo.Pods {
if p.Pod.DeletionTimestamp != nil && podutil.GetPodPriority(p.Pod) < podPriority {
// There is a terminating pod on the nominated node.
return false
}
}
}
}
return true
}
这个方法会检查该pod是否已经抢占过其他node节点，如果是的话就遍历节点上的所有pod对象，如果发现节点上有pod资源对象的优先级小于待调度pod资源对象并处于终止状态，则返回false，不会发生抢占。

接下来看FindCandidates方法：

FindCandidates

Copy
func FindCandidates(ctx context.Context, cs kubernetes.Interface, state *framework.CycleState, pod *v1.Pod,
m framework.NodeToStatusMap, ph framework.PreemptHandle, nodeLister framework.NodeInfoLister,
pdbLister policylisters.PodDisruptionBudgetLister) ([]Candidate, error) {
allNodes, err := nodeLister.List()
if err != nil {
return nil, err
}
if len(allNodes) == 0 {
return nil, core.ErrNoNodesAvailable
}

//找 predicates 阶段失败但是通过抢占也许能够调度成功的 nodes
potentialNodes := nodesWherePreemptionMightHelp(allNodes, m)
if len(potentialNodes) == 0 {
	klog.V(3).Infof("Preemption will not help schedule pod %v/%v on any node.", pod.Namespace, pod.Name) 
	if err := util.ClearNominatedNodeName(cs, pod); err != nil {
		klog.Errorf("Cannot clear 'NominatedNodeName' field of pod %v/%v: %v", pod.Namespace, pod.Name, err) 
	}
	return nil, nil
}
if klog.V(5).Enabled() {
	var sample []string
	for i := 0; i < 10 && i < len(potentialNodes); i++ {
		sample = append(sample, potentialNodes[i].Node().Name)
	}
	klog.Infof("%v potential nodes for preemption, first %v are: %v", len(potentialNodes), len(sample), sample)
}
//获取PDB对象，PDB能够限制同时终端的pod资源对象的数量，以保证集群的高可用性
pdbs, err := getPodDisruptionBudgets(pdbLister)
if err != nil {
	return nil, err
}
//寻找符合条件的node，并封装成candidate数组返回
return dryRunPreemption(ctx, ph, state, pod, potentialNodes, pdbs), nil

}
FindCandidates方法首先会获取node列表，然后调用nodesWherePreemptionMightHelp方法来找出predicates 阶段失败但是通过抢占也许能够调度成功的nodes，因为并不是所有的node都可以通过抢占来调度成功。最后调用dryRunPreemption方法来获取符合条件的node节点。

dryRunPreemption

Copy
func dryRunPreemption(ctx context.Context, fh framework.PreemptHandle, state *framework.CycleState,
pod *v1.Pod, potentialNodes []*framework.NodeInfo, pdbs []*policy.PodDisruptionBudget) []Candidate {
var resultLock sync.Mutex
var candidates []Candidate

checkNode := func(i int) {
	nodeInfoCopy := potentialNodes[i].Clone()
	stateCopy := state.Clone()
	//找到node上被抢占的pod，也就是victims
	pods, numPDBViolations, fits := selectVictimsOnNode(ctx, fh, stateCopy, pod, nodeInfoCopy, pdbs)
	if fits {
		resultLock.Lock()
		victims := extenderv1.Victims{
			Pods:             pods,
			NumPDBViolations: int64(numPDBViolations),
		}
		c := candidate{
			victims: &victims,
			name:    nodeInfoCopy.Node().Name,
		}
		candidates = append(candidates, &c)
		resultLock.Unlock()
	}
}
parallelize.Until(ctx, len(potentialNodes), checkNode)
return candidates

}
这里会开启16个线程调用checkNode方法，checkNode方法里面会调用selectVictimsOnNode方法来检查这个node是不是能被执行抢占，如果能执行抢占返回的pods表示需要删除的节点，然后封装成candidate添加到candidates列表中返回。

selectVictimsOnNode

Copy
func selectVictimsOnNode(
ctx context.Context,
ph framework.PreemptHandle,
state *framework.CycleState,
pod *v1.Pod,
nodeInfo *framework.NodeInfo,
pdbs []*policy.PodDisruptionBudget,
) ([]*v1.Pod, int, bool) {
var potentialVictims []*v1.Pod

//移除node节点的pod
removePod := func(rp *v1.Pod) error {
	if err := nodeInfo.RemovePod(rp); err != nil {
		return err
	}
	status := ph.RunPreFilterExtensionRemovePod(ctx, state, pod, rp, nodeInfo)
	if !status.IsSuccess() {
		return status.AsError()
	}
	return nil
}
//将node节点添加pod
addPod := func(ap *v1.Pod) error {
	nodeInfo.AddPod(ap)
	status := ph.RunPreFilterExtensionAddPod(ctx, state, pod, ap, nodeInfo)
	if !status.IsSuccess() {
		return status.AsError()
	}
	return nil
} 
// 获取pod的优先级，并将node中所有优先级低于该pod的调用removePod方法pod移除
podPriority := podutil.GetPodPriority(pod)
for _, p := range nodeInfo.Pods {
	if podutil.GetPodPriority(p.Pod) < podPriority {
		potentialVictims = append(potentialVictims, p.Pod)
		if err := removePod(p.Pod); err != nil {
			return nil, 0, false
		}
	}
}

//没有优先级低的node，直接返回
if len(potentialVictims) == 0 {
	return nil, 0, false
}

if fits, _, err := core.PodPassesFiltersOnNode(ctx, ph, state, pod, nodeInfo); !fits {
	if err != nil {
		klog.Warningf("Encountered error while selecting victims on node %v: %v", nodeInfo.Node().Name, err)
	}

	return nil, 0, false
}
var victims []*v1.Pod
numViolatingVictim := 0
//将potentialVictims集合里的pod按照优先级进行排序
sort.Slice(potentialVictims, func(i, j int) bool { return util.MoreImportantPod(potentialVictims[i], potentialVictims[j]) }) 
//将pdb的pod分离出来
//基于 pod 是否有 PDB 被分为两组 violatingVictims 和 nonViolatingVictims
//PDB：https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
violatingVictims, nonViolatingVictims := filterPodsWithPDBViolation(potentialVictims, pdbs)
reprievePod := func(p *v1.Pod) (bool, error) {
	if err := addPod(p); err != nil {
		return false, err
	}
	fits, _, _ := core.PodPassesFiltersOnNode(ctx, ph, state, pod, nodeInfo)
	if !fits {
		if err := removePod(p); err != nil {
			return false, err
		}
		// 加入到 victims 中
		victims = append(victims, p)
		klog.V(5).Infof("Pod %v/%v is a potential preemption victim on node %v.", p.Namespace, p.Name, nodeInfo.Node().Name)
	}
	return fits, nil
}
//删除pod，并记录删除个数
for _, p := range violatingVictims {
	if fits, err := reprievePod(p); err != nil {
		klog.Warningf("Failed to reprieve pod %q: %v", p.Name, err)
		return nil, 0, false
	} else if !fits {
		numViolatingVictim++
	}
} 
//删除pod
for _, p := range nonViolatingVictims {
	if _, err := reprievePod(p); err != nil {
		klog.Warningf("Failed to reprieve pod %q: %v", p.Name, err)
		return nil, 0, false
	}
}
return victims, numViolatingVictim, true

}
这个方法首先定义了两个方法，一个是removePod，另一个是addPod，这两个方法都差不多，如果是removePod就会将pod从node中移除，然后修改node一些属性，如将Requested.MilliCPU、Requested.Memory中减去，表示已用资源大小，将该pod从node节点的Pods列表中移除等等。

回到selectVictimsOnNode继续往下，会遍历node里面的pod列表，如果找到优先级小于抢占pod的就加入到potentialVictims集合中，并调用removePod方法，将当前被遍历的pod从node中移除。

接着会调用PodPassesFiltersOnNode方法，这个方法会运行两次。第一次会调用addNominatedPods方法将调度队列中找到节点上优先级大于或等于当前pod资源对象的nominatedPods加入到nodeInfo对象中，然后执行FilterPlugin列表；第二次则直接执行FilterPlugins列表。之所以要这么做，是由于亲和性的关系，k8s需要判断当前调度的pod亲和性是否依赖了nominatedPods。

继续往下会对potentialVictims按照优先级进行排序，优先级高的在前面。

接着会调用filterPodsWithPDBViolation方法，将 PDB 约束的 Pod和未约束的Pod分离成两个组，然后会分别遍历violatingVictims和nonViolatingVictims调用reprievePod方法对pod进行移除。这里我们在官方文档也可以看其设计理念，PodDisruptionBudget 是在抢占中被支持的，但不提供保证，然后将被移除的pod添加到victims列表中，并记录好被删除的删除pod个数，最后返回。

到这里整个FindCandidates方法就探索完毕了，还是比较长的，我们继续回到preempt方法中往下看，SelectCandidate方法会查找最佳抢占候选者。

SelectCandidate

Copy
func SelectCandidate(candidates []Candidate) Candidate {
if len(candidates) == 0 {
return nil
}
if len(candidates) == 1 {
return candidates[0]
}

victimsMap := candidatesToVictimsMap(candidates)
// 选择1个 node 用于 schedule
candidateNode := pickOneNodeForPreemption(victimsMap)

for _, candidate := range candidates {
	if candidateNode == candidate.Name() {
		return candidate
	}
} 
klog.Errorf("None candidate can be picked from %v.", candidates) 
return candidates[0]

}
这个方法里面会调用candidatesToVictimsMap方法做一个name和victims映射map，然后调用pickOneNodeForPreemption执行主要过滤逻辑。

pickOneNodeForPreemption

Copy
func pickOneNodeForPreemption(nodesToVictims map[string]*extenderv1.Victims) string {
//若该 node 没有 victims 则返回
if len(nodesToVictims) == 0 {
return “”
}
minNumPDBViolatingPods := int64(math.MaxInt32)
var minNodes1 []string
lenNodes1 := 0
//寻找 PDB violations 数量最小的 node
for node, victims := range nodesToVictims {
numPDBViolatingPods := victims.NumPDBViolations
if numPDBViolatingPods < minNumPDBViolatingPods {
minNumPDBViolatingPods = numPDBViolatingPods
minNodes1 = nil
lenNodes1 = 0
}
if numPDBViolatingPods == minNumPDBViolatingPods {
minNodes1 = append(minNodes1, node)
lenNodes1++
}
}
//如果最小的node只有一个，直接返回
if lenNodes1 == 1 {
return minNodes1[0]
}

minHighestPriority := int32(math.MaxInt32)
var minNodes2 = make([]string, lenNodes1)
lenNodes2 := 0
// 找到node里面pods 最高优先级最小的
for i := 0; i < lenNodes1; i++ {
	node := minNodes1[i]
	victims := nodesToVictims[node] 
	highestPodPriority := podutil.GetPodPriority(victims.Pods[0])
	if highestPodPriority < minHighestPriority {
		minHighestPriority = highestPodPriority
		lenNodes2 = 0
	}
	if highestPodPriority == minHighestPriority {
		minNodes2[lenNodes2] = node
		lenNodes2++
	}
}
if lenNodes2 == 1 {
	return minNodes2[0]
}

// 找出node里面Victims列表优先级加和最小的
minSumPriorities := int64(math.MaxInt64)
lenNodes1 = 0
for i := 0; i < lenNodes2; i++ {
	var sumPriorities int64
	node := minNodes2[i]
	for _, pod := range nodesToVictims[node].Pods { 
		sumPriorities += int64(podutil.GetPodPriority(pod)) + int64(math.MaxInt32+1)
	}
	if sumPriorities < minSumPriorities {
		minSumPriorities = sumPriorities
		lenNodes1 = 0
	}
	if sumPriorities == minSumPriorities {
		minNodes1[lenNodes1] = node
		lenNodes1++
	}
}
if lenNodes1 == 1 {
	return minNodes1[0]
}

// 找到node列表中需要牺牲的pod数量最小的
minNumPods := math.MaxInt32
lenNodes2 = 0
for i := 0; i < lenNodes1; i++ {
	node := minNodes1[i]
	numPods := len(nodesToVictims[node].Pods)
	if numPods < minNumPods {
		minNumPods = numPods
		lenNodes2 = 0
	}
	if numPods == minNumPods {
		minNodes2[lenNodes2] = node
		lenNodes2++
	}
}
if lenNodes2 == 1 {
	return minNodes2[0]
}

//若多个 node 的 pod 数量相等，则选出高优先级 pod 启动时间最短的
latestStartTime := util.GetEarliestPodStartTime(nodesToVictims[minNodes2[0]])
if latestStartTime == nil { 
	klog.Errorf("earliestStartTime is nil for node %s. Should not reach here.", minNodes2[0])
	return minNodes2[0]
}
nodeToReturn := minNodes2[0]
for i := 1; i < lenNodes2; i++ {
	node := minNodes2[i] 
	earliestStartTimeOnNode := util.GetEarliestPodStartTime(nodesToVictims[node])
	if earliestStartTimeOnNode == nil {
		klog.Errorf("earliestStartTime is nil for node %s. Should not reach here.", node)
		continue
	}
	if earliestStartTimeOnNode.After(latestStartTime.Time) {
		latestStartTime = earliestStartTimeOnNode
		nodeToReturn = node
	}
}

return nodeToReturn

}
这个方法看起来很长，其实逻辑十分的清晰：

https://www.chinaacc.com/ask/detail/free/18102577
https://www.chinaacc.com/ask/detail/free/18102538
https://www.chinaacc.com/ask/detail/free/18102489
https://www.chinaacc.com/ask/detail/free/18102451
https://www.chinaacc.com/ask/detail/free/18102424
https://www.chinaacc.com/ask/detail/free/18102392
https://www.chinaacc.com/ask/detail/free/18102358
https://www.chinaacc.com/ask/detail/free/18102320
https://www.chinaacc.com/ask/detail/free/18102278
https://www.chinaacc.com/ask/detail/free/18102249
https://www.chinaacc.com/ask/detail/free/18102207
https://www.chinaacc.com/ask/detail/free/18102168
https://www.chinaacc.com/ask/detail/free/18102129
https://www.chinaacc.com/ask/detail/free/18101820
https://www.chinaacc.com/ask/detail/free/18101539
https://www.chinaacc.com/ask/detail/free/18101462
https://www.chinaacc.com/ask/detail/free/18101396
https://www.chinaacc.com/ask/detail/free/18101320
https://www.chinaacc.com/ask/detail/free/18101256
https://www.chinaacc.com/ask/detail/free/18101218
https://www.chinaacc.com/ask/detail/free/18101137
https://www.chinaacc.com/ask/detail/free/18101064
https://www.chinaacc.com/ask/detail/free/18100963
https://www.chinaacc.com/ask/detail/free/18100895
https://www.chinaacc.com/ask/detail/free/18100849
https://www.chinaacc.com/ask/detail/free/18100803
https://www.chinaacc.com/ask/detail/free/18100741
https://www.chinaacc.com/ask/detail/free/18100677
https://www.chinaacc.com/ask/detail/free/18100618
https://www.chinaacc.com/ask/detail/free/18100543
https://www.chinaacc.com/ask/detail/free/18100465
https://www.chinaacc.com/ask/detail/free/18100408
https://www.chinaacc.com/ask/detail/free/18100328
https://www.chinaacc.com/ask/detail/free/18100233
https://www.chinaacc.com/ask/detail/free/18100174
https://www.chinaacc.com/ask/detail/free/18100087
https://www.chinaacc.com/ask/detail/free/18099974
https://www.chinaacc.com/ask/detail/free/18099882
https://www.chinaacc.com/ask/detail/free/18099802
https://www.chinaacc.com/ask/detail/free/18099710
https://www.chinaacc.com/ask/detail/free/18099599
https://www.chinaacc.com/ask/detail/free/18099323
https://www.chinaacc.com/ask/detail/free/18099250
https://www.chinaacc.com/ask/detail/free/18099165
https://www.chinaacc.com/ask/detail/free/18099079
https://www.chinaacc.com/ask/detail/free/18099003
https://www.chinaacc.com/ask/detail/free/18098892
https://www.chinaacc.com/ask/detail/free/18098812
https://www.chinaacc.com/ask/detail/free/18098729
https://www.chinaacc.com/ask/detail/free/18098604
https://www.chinaacc.com/ask/detail/free/18098529
https://www.chinaacc.com/ask/detail/free/18098434
https://www.chinaacc.com/ask/detail/free/18098371
https://www.chinaacc.com/ask/detail/free/18098311
https://www.chinaacc.com/ask/detail/free/18098209
https://www.chinaacc.com/ask/detail/free/18098111
https://www.chinaacc.com/ask/detail/free/18098028
https://www.chinaacc.com/ask/detail/free/18097966
https://www.chinaacc.com/ask/detail/free/18097870
https://www.chinaacc.com/ask/detail/free/18097762
https://www.chinaacc.com/ask/detail/free/18097687
https://www.chinaacc.com/ask/detail/free/18097590
https://www.chinaacc.com/ask/detail/free/18097482
https://www.chinaacc.com/ask/detail/free/18097331
https://www.chinaacc.com/ask/detail/free/18097252
https://www.chinaacc.com/ask/detail/free/18097178
https://www.chinaacc.com/ask/detail/free/18097054
https://www.chinaacc.com/ask/detail/free/18096971
https://www.chinaacc.com/ask/detail/free/18096829
https://www.chinaacc.com/ask/detail/free/18096770
https://www.chinaacc.com/ask/detail/free/18096669
https://www.chinaacc.com/ask/detail/free/18096574
https://www.chinaacc.com/ask/detail/free/18096526
https://www.chinaacc.com/ask/detail/free/18096440
https://www.chinaacc.com/ask/detail/free/18096328
https://www.chinaacc.com/ask/detail/free/18096245
https://www.chinaacc.com/ask/detail/free/18096156
https://www.chinaacc.com/ask/detail/free/18096058
https://www.chinaacc.com/ask/detail/free/18095884
https://www.chinaacc.com/ask/detail/free/18095748
https://www.chinaacc.com/ask/detail/free/18095658
https://www.chinaacc.com/ask/detail/free/18095559
https://www.chinaacc.com/ask/detail/free/18095431
https://www.chinaacc.com/ask/detail/free/18095315
https://www.chinaacc.com/ask/detail/free/18095207
https://www.chinaacc.com/ask/detail/free/18095067
https://www.chinaacc.com/ask/detail/free/18094928
https://www.chinaacc.com/ask/detail/free/18094851
https://www.chinaacc.com/ask/detail/free/18094728
https://www.chinaacc.com/ask/detail/free/18094664
https://www.chinaacc.com/ask/detail/free/18094561
https://www.chinaacc.com/ask/detail/free/18094485
https://www.chinaacc.com/ask/detail/free/18094396
https://www.chinaacc.com/ask/detail/free/18094318
https://www.chinaacc.com/ask/detail/free/18094237
https://www.chinaacc.com/ask/detail/free/18094126
https://www.chinaacc.com/ask/detail/free/18094029
https://www.chinaacc.com/ask/detail/free/18093898
https://www.chinaacc.com/ask/detail/free/18093807
https://www.chinaacc.com/ask/detail/free/18093674
https://www.chinaacc.com/ask/detail/free/18093575
https://www.chinaacc.com/ask/detail/free/18093465
https://www.chinaacc.com/ask/detail/free/18093387
https://www.chinaacc.com/ask/detail/free/18093020
https://www.chinaacc.com/ask/detail/free/18092937
https://www.chinaacc.com/ask/detail/free/18092860
https://www.chinaacc.com/ask/detail/free/18092712
https://www.chinaacc.com/ask/detail/free/18092652
https://www.chinaacc.com/ask/detail/free/18092565
https://www.chinaacc.com/ask/detail/free/18092426
https://www.chinaacc.com/ask/detail/free/18092317
https://www.chinaacc.com/ask/detail/free/18092188
https://www.chinaacc.com/ask/detail/free/18092106
https://www.chinaacc.com/ask/detail/free/18091976
https://www.chinaacc.com/ask/detail/free/18091878
https://www.chinaacc.com/ask/detail/free/18091799
https://www.chinaacc.com/ask/detail/free/18091727
https://www.chinaacc.com/ask/detail/free/18091573
https://www.chinaacc.com/ask/detail/free/18091432
https://www.chinaacc.com/ask/detail/free/18091295
https://www.chinaacc.com/ask/detail/free/18091182
https://www.chinaacc.com/ask/detail/free/18091026
https://www.chinaacc.com/ask/detail/free/18088204
https://www.chinaacc.com/ask/detail/free/18088065
https://www.chinaacc.com/ask/detail/free/18087962
https://www.chinaacc.com/ask/detail/free/18087824
https://www.chinaacc.com/ask/detail/free/18087729
https://www.chinaacc.com/ask/detail/free/18087559
https://www.chinaacc.com/ask/detail/free/18087456
https://www.chinaacc.com/ask/detail/free/18087385
https://www.chinaacc.com/ask/detail/free/18087280
https://www.chinaacc.com/ask/detail/free/18087196
https://www.chinaacc.com/ask/detail/free/18087062
https://www.chinaacc.com/ask/detail/free/18086994
https://www.chinaacc.com/ask/detail/free/18086925
https://www.chinaacc.com/ask/detail/free/18086811
https://www.chinaacc.com/ask/detail/free/18086724
https://www.chinaacc.com/ask/detail/free/18086613
https://www.chinaacc.com/ask/detail/free/18086459
https://www.chinaacc.com/ask/detail/free/18086358
https://www.chinaacc.com/ask/detail/free/18086267
https://www.chinaacc.com/ask/detail/free/18085875
https://www.chinaacc.com/ask/detail/free/18085781
https://www.chinaacc.com/ask/detail/free/18085714
https://www.chinaacc.com/ask/detail/free/18085562
https://www.chinaacc.com/ask/detail/free/18085454
https://www.chinaacc.com/ask/detail/free/18085394
https://www.chinaacc.com/ask/detail/free/18085287
https://www.chinaacc.com/ask/detail/free/18085199
https://www.chinaacc.com/ask/detail/free/18085095
https://www.chinaacc.com/ask/detail/free/18085015
https://www.chinaacc.com/ask/detail/free/18084865
https://www.chinaacc.com/ask/detail/free/18084791
https://www.chinaacc.com/ask/detail/free/18084690
https://www.chinaacc.com/ask/detail/free/18084517
https://www.chinaacc.com/ask/detail/free/18084413
https://www.chinaacc.com/ask/detail/free/18084310
https://www.chinaacc.com/ask/detail/free/18084200
https://www.chinaacc.com/ask/detail/free/18084098
https://www.chinaacc.com/ask/detail/free/18083992
https://www.chinaacc.com/ask/detail/free/18083889
https://www.chinaacc.com/ask/detail/free/18083805
https://www.chinaacc.com/ask/detail/free/18083720
https://www.chinaacc.com/ask/detail/free/18083589
https://www.chinaacc.com/ask/detail/free/18083499
https://www.chinaacc.com/ask/detail/free/18083433
https://www.chinaacc.com/ask/detail/free/18083262
https://www.chinaacc.com/ask/detail/free/18083135
https://www.chinaacc.com/ask/detail/free/18082967
https://www.chinaacc.com/ask/detail/free/18082876
https://www.chinaacc.com/ask/detail/free/18082774
https://www.chinaacc.com/ask/detail/free/18082640
https://www.chinaacc.com/ask/detail/free/18082560
https://www.chinaacc.com/ask/detail/free/18082462
https://www.chinaacc.com/ask/detail/free/18082416
https://www.chinaacc.com/ask/detail/free/18082362
https://www.chinaacc.com/ask/detail/free/18082232
https://www.chinaacc.com/ask/detail/free/18082147
https://www.chinaacc.com/ask/detail/free/18081976
https://www.chinaacc.com/ask/detail/free/18081896
https://www.chinaacc.com/ask/detail/free/18081768
https://www.chinaacc.com/ask/detail/free/18081650
https://www.chinaacc.com/ask/detail/free/18080696
https://www.chinaacc.com/ask/detail/free/18080541
https://www.chinaacc.com/ask/detail/free/18080465
https://www.chinaacc.com/ask/detail/free/18080385
https://www.chinaacc.com/ask/detail/free/18080319
https://www.chinaacc.com/ask/detail/free/18080207
https://www.chinaacc.com/ask/detail/free/18080126
https://www.chinaacc.com/ask/detail/free/18080036
https://www.chinaacc.com/ask/detail/free/18079953
https://www.chinaacc.com/ask/detail/free/18079864
https://www.chinaacc.com/ask/detail/free/18079788
https://www.chinaacc.com/ask/detail/free/18079690
https://www.chinaacc.com/ask/detail/free/18079526
https://www.chinaacc.com/ask/detail/free/18079319
https://www.chinaacc.com/ask/detail/free/18079238
https://www.chinaacc.com/ask/detail/free/18079051
https://www.chinaacc.com/ask/detail/free/18078896
https://www.chinaacc.com/ask/detail/free/18078704
https://www.chinaacc.com/ask/detail/free/18078466
https://www.chinaacc.com/ask/detail/free/18078340
https://www.chinaacc.com/ask/detail/free/18078213
https://www.chinaacc.com/ask/detail/free/18078062
https://www.chinaacc.com/ask/detail/free/18077960
https://www.chinaacc.com/ask/detail/free/18077797
https://www.chinaacc.com/ask/detail/free/18077683
https://www.chinaacc.com/ask/detail/free/18077589
https://www.chinaacc.com/ask/detail/free/18077501
https://www.chinaacc.com/ask/detail/free/18077386
https://www.chinaacc.com/ask/detail/free/18077045
https://www.chinaacc.com/ask/detail/free/18076936
https://www.chinaacc.com/ask/detail/free/18076771
https://www.chinaacc.com/ask/detail/free/18076688
https://www.chinaacc.com/ask/detail/free/18076544
https://www.chinaacc.com/ask/detail/free/18076450
https://www.chinaacc.com/ask/detail/free/18076358
https://www.chinaacc.com/ask/detail/free/18076292
https://www.chinaacc.com/ask/detail/free/18076190
https://www.chinaacc.com/ask/detail/free/18076080
https://www.chinaacc.com/ask/detail/free/18075923
https://www.chinaacc.com/ask/detail/free/18075818
https://www.chinaacc.com/ask/detail/free/18075709
https://www.chinaacc.com/ask/detail/free/18075565
https://www.chinaacc.com/ask/detail/free/18075489
https://www.chinaacc.com/ask/detail/free/18075293
https://www.chinaacc.com/ask/detail/free/18075088
https://www.chinaacc.com/ask/detail/free/18075019
https://www.chinaacc.com/ask/detail/free/18074924
https://www.chinaacc.com/ask/detail/free/18074818
https://www.chinaacc.com/ask/detail/free/18074666
https://www.chinaacc.com/ask/detail/free/18074554
https://www.chinaacc.com/ask/detail/free/18074437
https://www.chinaacc.com/ask/detail/free/18074247
https://www.chinaacc.com/ask/detail/free/18074159
https://www.chinaacc.com/ask/detail/free/18074065
https://www.chinaacc.com/ask/detail/free/18073909
https://www.chinaacc.com/ask/detail/free/18073703
https://www.chinaacc.com/ask/detail/free/18073631
https://www.chinaacc.com/ask/detail/free/18073494
https://www.chinaacc.com/ask/detail/free/18073303
https://www.chinaacc.com/ask/detail/free/18073207
https://www.chinaacc.com/ask/detail/free/18073111
https://www.chinaacc.com/ask/detail/free/18072957
https://www.chinaacc.com/ask/detail/free/18072862
https://www.chinaacc.com/ask/detail/free/18072784
https://www.chinaacc.com/ask/detail/free/18072610
https://www.chinaacc.com/ask/detail/free/18072469
https://www.chinaacc.com/ask/detail/free/18072333
https://www.chinaacc.com/ask/detail/free/18072224
https://www.chinaacc.com/ask/detail/free/18071847
https://www.chinaacc.com/ask/detail/free/18071345
https://www.chinaacc.com/ask/detail/free/18071185
https://www.chinaacc.com/ask/detail/free/18071047
https://www.chinaacc.com/ask/detail/free/18067970
https://www.chinaacc.com/ask/detail/free/18067386
https://www.chinaacc.com/ask/detail/free/18064319
https://www.chinaacc.com/ask/detail/free/18064148
https://www.chinaacc.com/ask/detail/free/18064080
https://www.chinaacc.com/ask/detail/free/18063998
https://www.chinaacc.com/ask/detail/free/18063936
https://www.chinaacc.com/ask/detail/free/18063836
https://www.chinaacc.com/ask/detail/free/18063745
https://www.chinaacc.com/ask/detail/free/18063612
https://www.chinaacc.com/ask/detail/free/18063478
https://www.chinaacc.com/ask/detail/free/18063391
https://www.chinaacc.com/ask/detail/free/18063241
https://www.chinaacc.com/ask/detail/free/18063163
https://www.chinaacc.com/ask/detail/free/18063105
https://www.chinaacc.com/ask/detail/free/18062985
https://www.chinaacc.com/ask/detail/free/18062929
https://www.chinaacc.com/ask/detail/free/18062851
https://www.chinaacc.com/ask/detail/free/18062744
https://www.chinaacc.com/ask/detail/free/18062647
https://www.chinaacc.com/ask/detail/free/18062579
https://www.chinaacc.com/ask/detail/free/18062504
https://www.chinaacc.com/ask/detail/free/18062435
https://www.chinaacc.com/ask/detail/free/18062293
https://www.chinaacc.com/ask/detail/free/18062189
https://www.chinaacc.com/ask/detail/free/18062076
https://www.chinaacc.com/ask/detail/free/18061970
https://www.chinaacc.com/ask/detail/free/18061805
https://www.chinaacc.com/ask/detail/free/18061697
https://www.chinaacc.com/ask/detail/free/18061595
https://www.chinaacc.com/ask/detail/free/18061420
https://www.chinaacc.com/ask/detail/free/18061314
https://www.chinaacc.com/ask/detail/free/18061150
https://www.chinaacc.com/ask/detail/free/18061031
https://www.chinaacc.com/ask/detail/free/18060962
https://www.chinaacc.com/ask/detail/free/18060882
https://www.chinaacc.com/ask/detail/free/18060767
https://www.chinaacc.com/ask/detail/free/18060670
https://www.chinaacc.com/ask/detail/free/18060531
https://www.chinaacc.com/ask/detail/free/18060434
https://www.chinaacc.com/ask/detail/free/18060329
https://www.chinaacc.com/ask/detail/free/18060236
https://www.chinaacc.com/ask/detail/free/18060124
https://www.chinaacc.com/ask/detail/free/18060026
https://www.chinaacc.com/ask/detail/free/18059843
https://www.chinaacc.com/ask/detail/free/18059757
https://www.chinaacc.com/ask/detail/free/18059685
https://www.chinaacc.com/ask/detail/free/18059502
https://www.chinaacc.com/ask/detail/free/18059425
https://www.chinaacc.com/ask/detail/free/18059319
https://www.chinaacc.com/ask/detail/free/18059247
https://www.chinaacc.com/ask/detail/free/18059129
https://www.chinaacc.com/ask/detail/free/18059036
https://www.chinaacc.com/ask/detail/free/18058942
https://www.chinaacc.com/ask/detail/free/18058845
https://www.chinaacc.com/ask/detail/free/18058746
https://www.chinaacc.com/ask/detail/free/18058689
https://www.chinaacc.com/ask/detail/free/18058412
https://www.chinaacc.com/ask/detail/free/18058313
https://www.chinaacc.com/ask/detail/free/18058265
https://www.chinaacc.com/ask/detail/free/18058155
https://www.chinaacc.com/ask/detail/free/18057980
https://www.chinaacc.com/ask/detail/free/18057905
https://www.chinaacc.com/ask/detail/free/18057823
https://www.chinaacc.com/ask/detail/free/18057733
https://www.chinaacc.com/ask/detail/free/18057626
https://www.chinaacc.com/ask/detail/free/18057563
https://www.chinaacc.com/ask/detail/free/18057467
https://www.chinaacc.com/ask/detail/free/18057333
https://www.chinaacc.com/ask/detail/free/18057254
https://www.chinaacc.com/ask/detail/free/18057145
https://www.chinaacc.com/ask/detail/free/18057034
https://www.chinaacc.com/ask/detail/free/18056902
https://www.chinaacc.com/ask/detail/free/18056759
https://www.chinaacc.com/ask/detail/free/18056662
https://www.chinaacc.com/ask/detail/free/18056489
https://www.chinaacc.com/ask/detail/free/18056339
https://www.chinaacc.com/ask/detail/free/18056259
https://www.chinaacc.com/ask/detail/free/18056101
https://www.chinaacc.com/ask/detail/free/18056010
https://www.chinaacc.com/ask/detail/free/18055866
https://www.chinaacc.com/ask/detail/free/18055771
https://www.chinaacc.com/ask/detail/free/18055634
https://www.chinaacc.com/ask/detail/free/18055378
https://www.chinaacc.com/ask/detail/free/18055264
https://www.chinaacc.com/ask/detail/free/18055120
https://www.chinaacc.com/ask/detail/free/18055039
https://www.chinaacc.com/ask/detail/free/18054879
https://www.chinaacc.com/ask/detail/free/18054766
https://www.chinaacc.com/ask/detail/free/18054677
https://www.chinaacc.com/ask/detail/free/18054555
https://www.chinaacc.com/ask/detail/free/18054375
https://www.chinaacc.com/ask/detail/free/18054270
https://www.chinaacc.com/ask/detail/free/18054151
https://www.chinaacc.com/ask/detail/free/18054086
https://www.chinaacc.com/ask/detail/free/18053946
https://www.chinaacc.com/ask/detail/free/18053824
https://www.chinaacc.com/ask/detail/free/18053686
https://www.chinaacc.com/ask/detail/free/18053598
https://www.chinaacc.com/ask/detail/free/18053451
https://www.chinaacc.com/ask/detail/free/18053331
https://www.chinaacc.com/ask/detail/free/18053200
https://www.chinaacc.com/ask/detail/free/18053121
https://www.chinaacc.com/ask/detail/free/18053023
https://www.chinaacc.com/ask/detail/free/18052822
https://www.chinaacc.com/ask/detail/free/18052682
https://www.chinaacc.com/ask/detail/free/18052586
https://www.chinaacc.com/ask/detail/free/18052445
https://www.chinaacc.com/ask/detail/free/18052302
https://www.chinaacc.com/ask/detail/free/18052191
https://www.chinaacc.com/ask/detail/free/18052027
https://www.chinaacc.com/ask/detail/free/18051843
https://www.chinaacc.com/ask/detail/free/18051728
https://www.chinaacc.com/ask/detail/free/18051527
https://www.chinaacc.com/ask/detail/free/18051395
https://www.chinaacc.com/ask/detail/free/18051230
https://www.chinaacc.com/ask/detail/free/18051110
https://www.chinaacc.com/ask/detail/free/18050976
https://www.chinaacc.com/ask/detail/free/18050683
https://www.chinaacc.com/ask/detail/free/18050585
https://www.chinaacc.com/ask/detail/free/18050435
https://www.chinaacc.com/ask/detail/free/18050346
https://www.chinaacc.com/ask/detail/free/18050257
https://www.chinaacc.com/ask/detail/free/18050109
https://www.chinaacc.com/ask/detail/free/18049977
https://www.chinaacc.com/ask/detail/free/18049915
https://www.chinaacc.com/ask/detail/free/18049823
https://www.chinaacc.com/ask/detail/free/18049707
https://www.chinaacc.com/ask/detail/free/18049571
https://www.chinaacc.com/ask/detail/free/18049503
https://www.chinaacc.com/ask/detail/free/18049416
https://www.chinaacc.com/ask/detail/free/18049115
https://www.chinaacc.com/ask/detail/free/18048917
https://www.chinaacc.com/ask/detail/free/18048829
https://www.chinaacc.com/ask/detail/free/18048721
https://www.chinaacc.com/ask/detail/free/18048584
https://www.chinaacc.com/ask/detail/free/18048443
https://www.chinaacc.com/ask/detail/free/18048325
https://www.chinaacc.com/ask/detail/free/18048212
https://www.chinaacc.com/ask/detail/free/18048136
https://www.chinaacc.com/ask/detail/free/18048028
https://www.chinaacc.com/ask/detail/free/18047932
https://www.chinaacc.com/ask/detail/free/18047864
https://www.chinaacc.com/ask/detail/free/18047741
https://www.chinaacc.com/ask/detail/free/18047626
https://www.chinaacc.com/ask/detail/free/18047508
https://www.chinaacc.com/ask/detail/free/18047381
https://www.chinaacc.com/ask/detail/free/18046988
https://www.chinaacc.com/ask/detail/free/18046852
https://www.chinaacc.com/ask/detail/free/18046758
https://www.chinaacc.com/ask/detail/free/18046683
https://www.chinaacc.com/ask/detail/free/18046549
https://www.chinaacc.com/ask/detail/free/18046483
https://www.chinaacc.com/ask/detail/free/18046325
https://www.chinaacc.com/ask/detail/free/18046236
https://www.chinaacc.com/ask/detail/free/18046098
https://www.chinaacc.com/ask/detail/free/18045995
https://www.chinaacc.com/ask/detail/free/18045887
https://www.chinaacc.com/ask/detail/free/18045814
https://www.chinaacc.com/ask/detail/free/18045703
https://www.chinaacc.com/ask/detail/free/18045609
https://www.chinaacc.com/ask/detail/free/18045539
https://www.chinaacc.com/ask/detail/free/18045476
https://www.chinaacc.com/ask/detail/free/18045421
https://www.chinaacc.com/ask/detail/free/18045285
https://www.chinaacc.com/ask/detail/free/18045184
https://www.chinaacc.com/ask/detail/free/18045093
https://www.chinaacc.com/ask/detail/free/18045041
https://www.chinaacc.com/ask/detail/free/18044942
https://www.chinaacc.com/ask/detail/free/18044882
https://www.chinaacc.com/ask/detail/free/18044824
https://www.chinaacc.com/ask/detail/free/18044751
https://www.chinaacc.com/ask/detail/free/18044668
https://www.chinaacc.com/ask/detail/free/18044549
https://www.chinaacc.com/ask/detail/free/18044429
https://www.chinaacc.com/ask/detail/free/18044307
https://www.chinaacc.com/ask/detail/free/18044197
https://www.chinaacc.com/ask/detail/free/18044079
https://www.chinaacc.com/ask/detail/free/18043870
https://www.chinaacc.com/ask/detail/free/18043767
https://www.chinaacc.com/ask/detail/free/18043690
https://www.chinaacc.com/ask/detail/free/18043582
https://www.chinaacc.com/ask/detail/free/18043461
https://www.chinaacc.com/ask/detail/free/18043407
https://www.chinaacc.com/ask/detail/free/18043323
https://www.chinaacc.com/ask/detail/free/18043261
https://www.chinaacc.com/ask/detail/free/18043161
https://www.chinaacc.com/ask/detail/free/18043073
https://www.chinaacc.com/ask/detail/free/18042988
https://www.chinaacc.com/ask/detail/free/18042900
https://www.chinaacc.com/ask/detail/free/18042812
https://www.chinaacc.com/ask/detail/free/18042720
https://www.chinaacc.com/ask/detail/free/18042652
https://www.chinaacc.com/ask/detail/free/18042543
https://www.chinaacc.com/ask/detail/free/18042407
https://www.chinaacc.com/ask/detail/free/18042284
https://www.chinaacc.com/ask/detail/free/18042191
https://www.chinaacc.com/ask/detail/free/18042059
https://www.chinaacc.com/ask/detail/free/18041969
https://www.chinaacc.com/ask/detail/free/18041869
https://www.chinaacc.com/ask/detail/free/18041782
https://www.chinaacc.com/ask/detail/free/18041685
https://www.chinaacc.com/ask/detail/free/18041590
https://www.chinaacc.com/ask/detail/free/18041514
https://www.chinaacc.com/ask/detail/free/18041453
https://www.chinaacc.com/ask/detail/free/18041345
https://www.chinaacc.com/ask/detail/free/18041245
https://www.chinaacc.com/ask/detail/free/18041145
https://www.chinaacc.com/ask/detail/free/18041028
https://www.chinaacc.com/ask/detail/free/18040928
https://www.chinaacc.com/ask/detail/free/18040797
https://www.chinaacc.com/ask/detail/free/18040633
https://www.chinaacc.com/ask/detail/free/18040561
https://www.chinaacc.com/ask/detail/free/18040482
https://www.chinaacc.com/ask/detail/free/18040394
https://www.chinaacc.com/ask/detail/free/18040327
https://www.chinaacc.com/ask/detail/free/18040216
https://www.chinaacc.com/ask/detail/free/18040132
https://www.chinaacc.com/ask/detail/free/18040076
https://www.chinaacc.com/ask/detail/free/18040003
https://www.chinaacc.com/ask/detail/free/18039934
https://www.chinaacc.com/ask/detail/free/18039837
https://www.chinaacc.com/ask/detail/free/18039767
https://www.chinaacc.com/ask/detail/free/18039681
https://www.chinaacc.com/ask/detail/free/18039548
https://www.chinaacc.com/ask/detail/free/18039483
https://www.chinaacc.com/ask/detail/free/18039422
https://www.chinaacc.com/ask/detail/free/18039345
https://www.chinaacc.com/ask/detail/free/18039191
https://www.chinaacc.com/ask/detail/free/18039071
https://www.chinaacc.com/ask/detail/free/18038981
https://www.chinaacc.com/ask/detail/free/18038829
https://www.chinaacc.com/ask/detail/free/18038670
https://www.chinaacc.com/ask/detail/free/18038572
https://www.chinaacc.com/ask/detail/free/18038455
https://www.chinaacc.com/ask/detail/free/18038334
https://www.chinaacc.com/ask/detail/free/18038222
https://www.chinaacc.com/ask/detail/free/18038133
https://www.chinaacc.com/ask/detail/free/18038046
https://www.chinaacc.com/ask/detail/free/18037796
https://www.chinaacc.com/ask/detail/free/18037681
https://www.chinaacc.com/ask/detail/free/18037565
https://www.chinaacc.com/ask/detail/free/18037413
https://www.chinaacc.com/ask/detail/free/18037268
https://www.chinaacc.com/ask/detail/free/18037137
https://www.chinaacc.com/ask/detail/free/18037034
https://www.chinaacc.com/ask/detail/free/18036925
https://www.chinaacc.com/ask/detail/free/18036833
https://www.chinaacc.com/ask/detail/free/18036730
https://www.chinaacc.com/ask/detail/free/18036653
https://www.chinaacc.com/ask/detail/free/18036595
https://www.chinaacc.com/ask/detail/free/18036469
https://www.chinaacc.com/ask/detail/free/18036382
https://www.chinaacc.com/ask/detail/free/18036305
https://www.chinaacc.com/ask/detail/free/18036244
https://www.chinaacc.com/ask/detail/free/18036098
https://www.chinaacc.com/ask/detail/free/18036018
https://www.chinaacc.com/ask/detail/free/18035939
https://www.chinaacc.com/ask/detail/free/18035851
https://www.chinaacc.com/ask/detail/free/18035789
https://www.chinaacc.com/ask/detail/free/18035686
https://www.chinaacc.com/ask/detail/free/18035607
https://www.chinaacc.com/ask/detail/free/18035539
https://www.chinaacc.com/ask/detail/free/18035425
https://www.chinaacc.com/ask/detail/free/18035305
https://www.chinaacc.com/ask/detail/free/18035207
https://www.chinaacc.com/ask/detail/free/18035143
https://www.chinaacc.com/ask/detail/free/18035067
https://www.chinaacc.com/ask/detail/free/18034959
https://www.chinaacc.com/ask/detail/free/18034855
https://www.chinaacc.com/ask/detail/free/18034793
https://www.chinaacc.com/ask/detail/free/18034715
https://www.chinaacc.com/ask/detail/free/18034579
https://www.chinaacc.com/ask/detail/free/18034510
https://www.chinaacc.com/ask/detail/free/18034433
https://www.chinaacc.com/ask/detail/free/18034365
https://www.chinaacc.com/ask/detail/free/18034277
https://www.chinaacc.com/ask/detail/free/18034152
https://www.chinaacc.com/ask/detail/free/18034046
https://www.chinaacc.com/ask/detail/free/18033956
https://www.chinaacc.com/ask/detail/free/18033870
https://www.chinaacc.com/ask/detail/free/18033780
https://www.chinaacc.com/ask/detail/free/18033715
https://www.chinaacc.com/ask/detail/free/18033544
https://www.chinaacc.com/ask/detail/free/18033409
https://www.chinaacc.com/ask/detail/free/18033317
https://www.chinaacc.com/ask/detail/free/18033207
https://www.chinaacc.com/ask/detail/free/18033096
https://www.chinaacc.com/ask/detail/free/18033021
https://www.chinaacc.com/ask/detail/free/18032912
https://www.chinaacc.com/ask/detail/free/18032862
https://www.chinaacc.com/ask/detail/free/18032787
https://www.chinaacc.com/ask/detail/free/18032681
https://www.chinaacc.com/ask/detail/free/18032556
https://www.chinaacc.com/ask/detail/free/18032447
https://www.chinaacc.com/ask/detail/free/18032330
https://www.chinaacc.com/ask/detail/free/18032238
https://www.chinaacc.com/ask/detail/free/18032163
https://www.chinaacc.com/ask/detail/free/18032077
https://www.chinaacc.com/ask/detail/free/18032010
https://www.chinaacc.com/ask/detail/free/18031936
https://www.chinaacc.com/ask/detail/free/18031790
https://www.chinaacc.com/ask/detail/free/18031724
https://www.chinaacc.com/ask/detail/free/18031672
https://www.chinaacc.com/ask/detail/free/18031606
https://www.chinaacc.com/ask/detail/free/18031346
https://www.chinaacc.com/ask/detail/free/18031289
https://www.chinaacc.com/ask/detail/free/18031201
https://www.chinaacc.com/ask/detail/free/18030958
https://www.chinaacc.com/ask/detail/free/18030882
https://www.chinaacc.com/ask/detail/free/18030810
https://www.chinaacc.com/ask/detail/free/18030721
https://www.chinaacc.com/ask/detail/free/18030631
https://www.chinaacc.com/ask/detail/free/18030531
https://www.chinaacc.com/ask/detail/free/18030431
https://www.chinaacc.com/ask/detail/free/18030368
https://www.chinaacc.com/ask/detail/free/18030296
https://www.chinaacc.com/ask/detail/free/18030146
https://www.chinaacc.com/ask/detail/free/18030084
https://www.chinaacc.com/ask/detail/free/18030022
https://www.chinaacc.com/ask/detail/free/18029930
https://www.chinaacc.com/ask/detail/free/18029827
https://www.chinaacc.com/ask/detail/free/18029731
https://www.chinaacc.com/ask/detail/free/18029510
https://www.chinaacc.com/ask/detail/free/18029438
https://www.chinaacc.com/ask/detail/free/18029369
https://www.chinaacc.com/ask/detail/free/18029295
https://www.chinaacc.com/ask/detail/free/18029225
https://www.chinaacc.com/ask/detail/free/18029150
https://www.chinaacc.com/ask/detail/free/18029071
https://www.chinaacc.com/ask/detail/free/18028943
https://www.chinaacc.com/ask/detail/free/18028875
https://www.chinaacc.com/ask/detail/free/18028801
https://www.chinaacc.com/ask/detail/free/18028743
https://www.chinaacc.com/ask/detail/free/18028677
https://www.chinaacc.com/ask/detail/free/18028613
https://www.chinaacc.com/ask/detail/free/18028553

找出最少的的PDB violations的node节点，如果找出的node集合大于1则往下走；
找出找到node里面pods 最高优先级最小的node，如果还是找出的node集合大于1则往下走；
找出node里面Victims列表优先级加和最小的，如果还是找出的node集合大于1则往下走；
找到node列表中需要牺牲的pod数量最小的，如果还是找出的node集合大于1则往下走；
若多个 node 的 pod 数量相等，则选出高优先级 pod 启动时间最短的，然后返回。
然后preempt方法往下走到调用PrepareCandidate方法：

adasdlulu
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
k8s.gcr.io/pause:3.1镜像包
03-06
kubernetes的k8s.gcr.io/pause:3.1镜像包，版本为3.1。文件是pause_3_1.tar
k8s.gcr.io/etcd:3.2.24
11-01
使用方法请关注blog: https://blog.csdn.net/wenwst 导入命令： #docker load < etcd.3.2.24.tar # ocker tag 3cab8e1b9802 k8s.gcr.io/etcd:3.2.24
K8S的调度优先级与抢占机制
zyxpaomian的博客
12-28 1342
Big Picture 在调度过程中，会有各种预选和优选算法，在经过了那么多道门槛之后，一个POD才能完成调度，提供服务，在调度过程中，假设有一个很重要的系统服务调度失败了，导致了故障，为了避免这种情况，我们可以对该应用设置一个相对较高的优先级，在调度失败后，将一些优先级相对较低的不是那么重要的应用”挤走“，这就是K8S调度的优先级和抢占的作用。 优先级 K8S有一个专门的API对象，就是优先级，比如: apiVersion: scheduling.k8s.io/v1beta1 kind: Priority
k8s 中调度优先级问题
Angus
01-09 1991
站在设计的角度上去考虑，任何一个系统都会有优先级的问题。最典型的栗子就是操作系统的调度问题。 对于k8s也同样是。 问题提炼 分配之前的阶段- 如何排Pod的优先级？ 分配之后的阶段-当资源不足时，如何抢占资源？ 抢占策略是什么？ 调度优先级问题 对于提交的Pod的申请，一定会有轻重缓急。比如，机器资源 90G内存。四个Pod，一个20G，那么最后一个Pod肯定得不到分配。假如前边的Pod不是那么重要，或者说优先级没有那么高，则可以先把前边的关了，先把后边的Pod起来。 ...
K8s源码分析（一）-K8s调度框架及调度器初始化介绍
最新发布
slipegg的博客
05-10 855
对K8s调度框架进行了介绍，还介绍了Cobra，并分析了K8s调度器的初始化代码
hualinux 进阶 1.24：k8s调度器的优先级与抢占机制 （整理）
hualinux（阿华的linux）
08-05 427
目录 一、优先级和抢占机制说明 二、优级先和抢占机制设计和实现 k8s调度器的优先级与抢占机制，这个比较重要，需要了解。 优先级和抢占机制，解决的是 Pod 调度失败时该怎么办的问题。 一、优先级和抢占机制说明 优先级和抢占机制，解决的是 Pod 调度失败时该怎么办的问题。 正常情况下，当一个 Pod 调度失败后，它就会被暂时“搁置”起来，直到 Pod 被更新，或者集群状态发生变化，调度器才会对这个 Pod 进行重新调度。 但在有时候，我们希望的是这样一个场景。当一个高优先级的 Po..
K8s的调度策略
weixin_30947043的博客
02-25 549
Scuedulor是K8s的调度器 sheduler 是作为单独的程序运行的，启动之后会一直坚挺 API Server，获取PodSpec.NodeName为空的 pod，对每个 pod 都会创建一个 binding，表明该 pod 应该放到哪个节点上 Kubernetes的调度器以插件化形式实现的，方便用户定制和二次开发。用户可以自定义调度器并以插件形式与Kubernetes集成，或...
k8s驱逐篇(4)-kube-scheduler抢占调度驱逐
良凯尔的博客
09-03 570
kube-scheduler抢占调度驱逐概述：当一个高优先级的 Pod 调度失败后，该 Pod 并不会被“搁置”，而是会“挤走”某个 Node 上的一些低优先级的 Pod，这样一来就可以保证高优先级 Pod 会优先调度成功。抢占发生的原因，一定是一个高优先级的 pod 调度失败，我们称这个 pod 为“抢占者”，称被抢占的 pod 为“牺牲者”(victims)。
prometheus-adapter:使用Prometheus实现custom.metrics.k8s.io API
04-28
适用于Kubernetes指标API的Prometheus适配器 该存储库包含Kubernetes ， 和API的实现。 因此，该适配器适合与Kubernetes 1.6+中的autoscaling / v2 Horizo​​ntal Pod Autoscaler一起使用。 它还可以替换已经运行...
rabbitmq-peer-discovery-k8s：RabbitMQ的基于Kubernetes的对等发现机制
02-04
`rabbitmq-peer-discovery-k8s`插件是RabbitMQ官方提供的一个发现机制，它允许RabbitMQ节点通过Kubernetes API来发现集群中的其他节点。在K8s环境中，每个RabbitMQ Pod都可以配置为使用这个插件，通过阅读Kubernetes...
k8s.gcr.io/pause:3.1
11-01
使用方法请关注blog: https://blog.csdn.net/wenwst 导入命令： #docker load < pause.3.1.tar # docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1
Kubernetes PriorityClass 高级调度能力 优先级调度/抢占调度
小楼一夜听春雨，深巷明朝卖杏花
03-24 1210
介绍完了基础调度能力之后，下面来了解一下高级调度能力。 优先级调度 优先级调度和抢占，主要概念有： Priority Preemption 首先来看一下调度过程提到的四个特点，我们如何做到集群的合理利用？ 当集群资源足够的话，只需要通过基础调度能力就能组合出合理的使用方式。 但是假如资源不够，我们怎么做到集群的合理利用呢？ 通常的策略有两类： 先到先得策略 (FIFO) -简单、相对公平，上手快 优先级策略 (Priority) - 符合日常公司业务特点 在实际生产中，如果使用先到先得策
kube-scheduler源码分析（3）-抢占调度分析
良凯尔的博客
03-06 561
kube-scheduler组件是kubernetes中的核心组件之一，主要负责pod资源对象的调度工作，具体来说，kube-scheduler组件负责根据调度算法（包括预选算法和优选算法）将未调度的pod调度到合适的最优的node节点上。而优先级和抢占机制，解决的是 Pod 调度失败时该怎么办的问题。抢占发生的原因，一定是一个高优先级的 pod 调度失败，我们称这个 pod 为“抢占者”，称被抢占的 pod 为“牺牲者”(victims)。
Kubernetes核心概念汇总—调度、抢占和驱逐
深圳市多克创新科技有限公司
07-02 534
Kubernetes核心概念汇总—调度、抢占和驱逐
一文轻松说透 K8S Pod 各种驱逐场景
力哥讲技术
05-04 737
如果您合理地管理这些资源，不仅可以降低成本，还可以确保重要流程无论如何都能继续运行。所有新水果都将包含更高的优先级，称为。第一个将具有更高的值，表示更高的优先级。k8s中的pod被终止，通常是没有足够资源的结果。这样，候选节点从最适合到最不适合排序。如果两个节点的分数相同，这意味着在抢占的情况下，树莓和草莓更有可能被驱逐，为更高优先级的。在抢占时更有可能继续运行，因为它们被驱逐的可能性较小。如果您在容器中添加非常低的请求，它们的。可以被驱逐，但比下一个类别更不可能。对现有节点应用污点，所有不能容忍它的。
k8s-pod设置执行优先级
xiaofeixia666888的博客
04-09 650
PriorityClass是Kubernetes中用于指定Pod优先级的机制。通过为Pod分配不同的优先级，K8s调度器可以更好地管理资源，确保高优先级的Pod被更优先地调度。globalDefault： 是否将该优先级类设为全局默认，即在没有为Pod指定PriorityClass的情况下使用该类的优先级。将定义好的PriorityClass应用到实际的Pod。value： 优先级的数值，数值越大，优先级越高。name： 优先级类的名称，用于标识该类。description： 对优先级类的描述。
《k8s-1.13版本源码分析》-抢占调度
weixin_30763397的博客
03-29 183
源码分析系列文章已经开源到github，地址如下： github：https://github.com/farmer-hutao/k8s-source-code-analysis gitbook：https://farmer-hutao.github.io/k8s-source-code-analysis 本文大纲 ...
深入剖析k8s
Zen_y的专栏
03-16 816
深入剖析k8s 课程信息k8s技能图谱Kubernetes项⽬与基础设施“⺠主化”的探索 背景 Docker 一举走红的重要原因 解决了应用打包这个根本性的问题 同开发者与生俱来的的亲密关系 PaaS 概念已经深入人心的完美契机 容器基础 对 Docker 项目来说，它最核心的原理实际上就是为待创建的用户进程： 隔离：启用 Linux Namespace 配置； 约束：设置指定的 Cgroups 参数； 切换进程的根目录（Change Root） Namespace ..
K8S污点容忍度调度
liujiangxu
10-11 633
污点类型 解释(相当于做标签) kubectl taint nodes node key=value:NoSchedule 添加effect类型污点，新的pod不能调度过来，老的不影响 kubectl taint nodes node key=value:NoExecute 添加effecf类型污点，新的pod不能调度过来，老的被驱除 kubectl taint nodes node key=value:PreferNoSchedule 会尝试将pod分配到该节点 污点、容忍度 ...
K8S Pod配置详解（spec）
lzy__123的博客
02-13 4050
Pod 是 Kubernetes 中最基本的资源对象之一，代表一组容器的单位。以下是几个重要的 Pod 字段的含义和用法：metadata：包含了 Pod 的元数据，例如名称、标签、命名空间等。spec：定义了 Pod 的行为和构建，例如容器镜像、资源限制、环境变量等。status：存储了 Pod 的运行状态，例如 Pod IP 地址、容器状态、节点名称等。containers：定义了 Pod 中包含的容器，包括容器名称、镜像、端口映射等。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值