statefulset简介
statefulset是Kubernetes提供的管理有状态应用的对象,而deployment用于管理无状态应用。
有状态的pod与无状态的pod不一样的是,有状态的pod有时候需要通过其主机名来定位,而无状态的不需要,因为无状态的pod每个都是一样的,随机选一个就行,但对于有状态的来说,每一个pod都不一样,通常希望操作的是特定的某一个。
statefulset适用于需要以下特点的应用:
(1)稳定的网络标志(主机名):Pod重新调度后其PodName和HostName不变;
(2)稳定的持久化存储:基于PVC,Pod重新调度后仍能访问到相同的持久化数据;
(3)稳定的创建与扩容次序:有序创建或扩容,从序号小到大的顺序对pod进行创建(即从0到N-1),且在下一个Pod创建运行之前,所有之前的Pod必须都是Running和Ready状态;
(4)稳定的删除与缩容次序:有序删除或缩容,从序号大到小的顺序对pod进行删除(即从N-1到0),且在下一个Pod终止与删除之前,所有之前的Pod必须都已经被删除;
(5)稳定的滚动更新次序:从序号大到小的顺序对pod进行更新(即从N-1到0),先删除后创建,且需等待当前序号的pod再次创建完成且状态为Ready时才能进行下一个pod的更新处理。
statefulset controller简介
statefulset controller是kube-controller-manager组件中众多控制器中的一个,是 statefulset 资源对象的控制器,其通过对statefulset、pod资源的监听,当资源发生变化时会触发 statefulset controller 对相应的statefulset资源对象进行调谐操作,从而完成statefulset对于pod的创建、删除、更新、扩缩容、statefulset的滚动更新、statefulset状态status更新、旧版本statefulset清理等操作。
statefulset controller架构图
statefulset controller的大致组成和处理流程如下图,statefulset controller对statefulset、pod对象注册了event handler,当有事件时,会watch到然后将对应的statefulset对象放入到queue中,然后syncStatefulSet
方法为statefulset controller调谐statefulset对象的核心处理逻辑所在,从queue中取出statefulset对象,做调谐处理。
![](https://img-blog.csdnimg.cn/img_convert/22697630f0fc153e3e9308181847e367.png)
statefulset pod的命名规则、pod创建与删除
如果创建一个名称为web、replicas为3的statefulset对象,则其pod名称分别为web-0、web-1、web-2。
statefulset pod的创建按0 - n的顺序创建,且在创建下一个pod之前,需要等待前一个pod创建完成并处于ready状态。
同样拿上面的例子来说明,在web statefulset创建后,3 个 Pod 将按顺序创建 web-0,web-1,web-2。在 web-0 处于 ready 状态之前,web-1 将不会被创建,同样当 web-1 处于ready状态之前 web-2也不会被创建。如果在 web-1 ready后,web-2 创建之前, web-0 不处于ready状态了,这个时候 web-2 将不会被创建,直到 web-0 再次回到ready状态。
statefulset滚动更新或缩容过程中pod的删除按n - 0的顺序删除,且在删除下一个pod之前,需要等待前一个pod删除完成。
另外,当statefulset.Spec.VolumeClaimTemplates中定义了pod所需的pvc时,statefulset controller在创建pod时,会同时创建对应的pvc出来,但删除pod时,不会做对应pvc的删除操作,这些pvc需要人工额外做删除操作。
statefulset更新策略
(1)OnDelete:使用 OnDelete 更新策略时,在更新 statefulset pod模板后,只有当你手动删除老的 statefulset pods 之后,新的 statefulset Pod 才会被自动创建。
(2)RollingUpdate:使用 RollingUpdate 更新策略时,在更新 statefulset pod模板后, 老的 statefulset pods 将被删除,并且将根据滚动更新配置自动创建新的 statefulset pods。滚动更新期间,每个序号的statefulset pod最多只能有一个,且滚动更新下一个pod之前,需等待前一个pod更新完成并处于ready状态。与statefulset pod按0 - n的顺序创建不同,滚动更新时Pod按逆序的方式(即n - 0)删除并创建。
statefulset的滚动升级中还有一个Partition配置,在设置partition后,滚动更新过程中,statefulset的Pod中序号大于或等于partition的Pod会进行滚动升级,而其余的Pod保持不变,不会进行滚动更新。
statefulset controller分析将分为两大块进行,分别是:
(1)statefulset controller初始化与启动分析;
(2)statefulset controller处理逻辑分析。
源码分析
基于 tag 1.30.0
1.statefulset controller初始化与启动分析
startStatefulSetController主要逻辑:
(1)调用statefulset.NewStatefulSetController新建并初始化StatefulSetController;
(2)拉起一个goroutine,跑StatefulSetController的Run方法。
func startStatefulSetController(ctx context.Context, controllerContext ControllerContext, controllerName string) (controller.Interface, bool, error) {
go statefulset.NewStatefulSetController(
ctx,
controllerContext.InformerFactory.Core().V1().Pods(),
controllerContext.InformerFactory.Apps().V1().StatefulSets(),
controllerContext.InformerFactory.Core().V1().PersistentVolumeClaims(),
controllerContext.InformerFactory.Apps().V1().ControllerRevisions(),
controllerContext.ClientBuilder.ClientOrDie("statefulset-controller"),
).Run(ctx, int(controllerContext.ComponentConfig.StatefulSetController.ConcurrentStatefulSetSyncs))
return nil, true, nil
}
1.1 statefulset.NewStatefulSetController
从statefulset.NewStatefulSetController
函数代码中可以看到,statefulset controller注册了statefulset、pod对象的EventHandler,也即对这几个对象的event进行监听,把event放入事件队列并做处理,对statefulset controller做了初始化。
// NewStatefulSetController creates a new statefulset controller.
func NewStatefulSetController(
ctx context.Context,
podInformer coreinformers.PodInformer,
setInformer appsinformers.StatefulSetInformer,
pvcInformer coreinformers.PersistentVolumeClaimInformer,
revInformer appsinformers.ControllerRevisionInformer,
kubeClient clientset.Interface,
) *StatefulSetController {
logger := klog.FromContext(ctx)
eventBroadcaster := record.NewBroadcaster(record.WithContext(ctx))
recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "statefulset-controller"})
ssc := &StatefulSetController{
kubeClient: kubeClient,
control: NewDefaultStatefulSetControl(
NewStatefulPodControl(
kubeClient,
podInformer.Lister(),
pvcInformer.Lister(),
recorder),
NewRealStatefulSetStatusUpdater(kubeClient, setInformer.Lister()),
history.NewHistory(kubeClient, revInformer.Lister()),
),
pvcListerSynced: pvcInformer.Informer().HasSynced,
revListerSynced: revInformer.Informer().HasSynced,
queue: workqueue.NewTypedRateLimitingQueueWithConfig(
workqueue.DefaultTypedControllerRateLimiter[string](),
workqueue.TypedRateLimitingQueueConfig[string]{Name: "statefulset"},
),
podControl: controller.RealPodControl{KubeClient: kubeClient, Recorder: recorder},
eventBroadcaster: eventBroadcaster,
}
podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
// lookup the statefulset and enqueue
AddFunc: func(obj interface{}) {
ssc.addPod(logger, obj)
},
// lookup current and old statefulset if labels changed
UpdateFunc: func(oldObj, newObj interface{}) {
ssc.updatePod(logger, oldObj, newObj)
},
// lookup statefulset accounting for deletion tombstones
DeleteFunc: func(obj interface{}) {
ssc.deletePod(logger, obj)
},
})
ssc.podLister = podInformer.Lister()
ssc.podListerSynced = podInformer.Informer().HasSynced
setInformer.Informer().AddEventHandler(
cache.ResourceEventHandlerFuncs{
AddFunc: ssc.enqueueStatefulSet,
UpdateFunc: func(old, cur interface{}) {
oldPS := old.(*apps.StatefulSet)
curPS := cur.(*apps.StatefulSet)
if oldPS.Status.Replicas != curPS.Status.Replicas {
logger.V(4).Info("Observed updated replica count for StatefulSet", "statefulSet", klog.KObj(curPS), "oldReplicas", oldPS.Status.Replicas, "newReplicas", curPS.Status.Replicas)
}
ssc.enqueueStatefulSet(cur)
},
DeleteFunc: ssc.enqueueStatefulSet,
},
)
ssc.setLister = setInformer.Lister()
ssc.setListerSynced = setInformer.Informer().HasSynced
// TODO: Watch volumes
return ssc
}
1.2 Run
主要看到for循环处,根据workers的值(可通过kube-controller-manager组件启动参数concurrent-statefulset-syncs
来设置,默认值为5),启动相应数量的goroutine,跑ssc.worker
方法,调用daemonset controller核心处理方法ssc.sync
来调谐statefulset对象。
// Run runs the statefulset controller.
func (ssc *StatefulSetController) Run(ctx context.Context, workers int) {
defer utilruntime.HandleCrash()
// Start events processing pipeline.
ssc.eventBroadcaster.StartStructuredLogging(3)
ssc.eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: ssc.kubeClient.CoreV1().Events("")})
defer ssc.eventBroadcaster.Shutdown()
defer ssc.queue.ShutDown()
logger := klog.FromContext(ctx)
logger.Info("Starting stateful set controller")
defer logger.Info("Shutting down statefulset controller")
if !cache.WaitForNamedCacheSync("stateful set", ctx.Done(), ssc.podListerSynced, ssc.setListerSynced, ssc.pvcListerSynced, ssc.revListerSynced) {
return
}
for i := 0; i < workers; i++ {
go wait.UntilWithContext(ctx, ssc.worker, time.Second)
}
<-ctx.Done()
}
1.2.1 ssc.worker
从queue队列中取出事件key,并调用ssc.sync
(关于ssc.sync方法会在后面做详细分析)对statefulset对象做调谐处理。queue队列里的事件来源前面讲过,是statefulset controller注册的statefulset、pod对象的EventHandler,它们的变化event会被监听到然后放入queue中。
// worker runs a worker goroutine that invokes processNextWorkItem until the controller's queue is closed
func (ssc *StatefulSetController) worker(ctx context.Context) {
for ssc.processNextWorkItem(ctx) {
}
}
// processNextWorkItem dequeues items, processes them, and marks them done. It enforces that the syncHandler is never
// invoked concurrently with the same key.
func (ssc *StatefulSetController) processNextWorkItem(ctx context.Context) bool {
key, quit := ssc.queue.Get()
if quit {
return false
}
defer ssc.queue.Done(key)
if err := ssc.sync(ctx, key); err != nil {
utilruntime.HandleError(fmt.Errorf("error syncing StatefulSet %v, requeuing: %w", key, err))
ssc.queue.AddRateLimited(key)
} else {
ssc.queue.Forget(key)
}
return true
}
2.statefulset controller核心处理逻辑分析
sync
直接看到statefulset controller核心处理入口sync方法。
主要逻辑:
(1)获取执行方法时的当前时间,并定义defer
函数,用于计算该方法总执行时间,也即统计对一个 statefulset 进行同步调谐操作的耗时;
(2)根据 statefulset 对象的命名空间与名称,获取 statefulset 对象;
(3)调用ssc.adoptOrphanRevisions,检查是否有孤儿 controllerrevisions 对象(即.spec.ownerReferences中无controller属性定义或其属性值为false),若有且其与 statefulset 对象的selector匹配 的则添加 ownerReferences 进行关联;
(4)调用ssc.getPodsForStatefulSet,根据 statefulset 对象的selector去查找pod列表,且若有孤儿pod的label与 statefulset 的selector能匹配的则进行关联,若已关联的pod的label不再与statefulset的selector匹配,则更新解除它们的关联关系;
(5)调用ssc.syncStatefulSet,对 statefulset 对象做调谐处理。
// sync syncs the given statefulset.
func (ssc *StatefulSetController) sync(ctx context.Context, key string) error {
startTime := time.Now()
logger := klog.FromContext(ctx)
defer func() {
logger.V(4).Info("Finished syncing statefulset", "key", key, "time", time.Since(startTime))
}()
namespace, name, err := cache.SplitMetaNamespaceKey(key)
if err != nil {
return err
}
set, err := ssc.setLister.StatefulSets(namespace).Get(name)
if errors.IsNotFound(err) {
logger.Info("StatefulSet has been deleted", "key", key)
return nil
}
if err != nil {
utilruntime.HandleError(fmt.Errorf("unable to retrieve StatefulSet %v from store: %v", key, err))
return err
}
selector, err := metav1.LabelSelectorAsSelector(set.Spec.Selector)
if err != nil {
utilruntime.HandleError(fmt.Errorf("error converting StatefulSet %v selector: %v", key, err))
// This is a non-transient error, so don't retry.
return nil
}
if err := ssc.adoptOrphanRevisions(ctx, set); err != nil {
return err
}
pods, err := ssc.getPodsForStatefulSet(ctx, set, selector)
if err != nil {
return err
}
return ssc.syncStatefulSet(ctx, set, pods)
}
2.1 ssc.getPodsForStatefulSet
ssc.getPodsForStatefulSet方法主要作用是获取属于 statefulset 对象的pod列表并返回,并检查孤儿pod与已匹配的pod,看是否需要更新statefulset与pod的匹配。
主要逻辑:
(1)获取 statefulset 所在命名空间下的所有pod;
(2)定义过滤出属于 statefulset 对象的pod的函数,即isMemberOf函数(根据pod的名称与statefulset名称匹配来过滤属于statefulset的pod);
(3)调用cm.ClaimPods,过滤出属于该statefulset对象的pod,且若有孤儿pod的label与 statefulset 的selector能匹配的则进行关联,若已关联的pod的label不再与statefulset的selector匹配,则更新解除它们的关联关系。
// getPodsForStatefulSet returns the Pods that a given StatefulSet should manage.
// It also reconciles ControllerRef by adopting/orphaning.
//
// NOTE: Returned Pods are pointers to objects from the cache.
// If you need to modify one, you need to copy it first.
func (ssc *StatefulSetController) getPodsForStatefulSet(ctx context.Context, set *apps.StatefulSet, selector labels.Selector) ([]*v1.Pod, error) {
// List all pods to include the pods that don't match the selector anymore but
// has a ControllerRef pointing to this StatefulSet.
pods, err := ssc.podLister.Pods(set.Namespace).List(labels.Everything())
if err != nil {
return nil, err
}
filter := func(pod *v1.Pod) bool {
// Only claim if it matches our StatefulSet name. Otherwise release/ignore.
return isMemberOf(set, pod)
}
cm := controller.NewPodControllerRefManager(ssc.podControl, set, selector, controllerKind, ssc.canAdoptFunc(ctx, set))
return cm.ClaimPods(ctx, pods, filter)
}
2.2 ssc.syncStatefulSet
ssc.syncStatefulSet方法可以说是statefulset controller的核心处理逻辑所在了,主要看到ssc.control.UpdateStatefulSet
方法。
// syncStatefulSet syncs a tuple of (statefulset, []*v1.Pod).
func (ssc *StatefulSetController) syncStatefulSet(ctx context.Context, set *apps.StatefulSet, pods []*v1.Pod) error {
logger := klog.FromContext(ctx)
logger.V(4).Info("Syncing StatefulSet with pods", "statefulSet", klog.KObj(set), "pods", len(pods))
var status *apps.StatefulSetStatus
var err error
status, err = ssc.control.UpdateStatefulSet(ctx, set, pods)
if err != nil {
return err
}
logger.V(4).Info("Successfully synced StatefulSet", "statefulSet", klog.KObj(set))
// One more sync to handle the clock skew. This is also helping in requeuing right after status update
if set.Spec.MinReadySeconds > 0 && status != nil && status.AvailableReplicas != *set.Spec.Replicas {
ssc.enqueueSSAfter(set, time.Duration(set.Spec.MinReadySeconds)*time.Second)
}
return nil
}
ssc.control.UpdateStatefulSet方法主要逻辑:
(1)获取statefulset的所有ControllerRevision并根据版本新老顺序排序;
(2) 执行 performUpdate方法
(2.1)调用ssc.getStatefulSetRevisions,获取现存最新的statefulset版本以及计算出一个新的版本;
(2.2)调用ssc.updateStatefulSet,完成statefulset对象对于pod的创建、删除、更新、扩缩容等操作; (2.3)调用ssc.updateStatefulSetStatus,更新statefulset对象的status状态;
(3)调用ssc.truncateHistory,根据statefulset对象配置的历史版本数量限制,按之前的排序顺序清理掉没有pod的statefulset历史版本。
// UpdateStatefulSet executes the core logic loop for a stateful set, applying the predictable and
// consistent monotonic update strategy by default - scale up proceeds in ordinal order, no new pod
// is created while any pod is unhealthy, and pods are terminated in descending order. The burst
// strategy allows these constraints to be relaxed - pods will be created and deleted eagerly and
// in no particular order. Clients using the burst strategy should be careful to ensure they
// understand the consistency implications of having unpredictable numbers of pods available.
func (ssc *defaultStatefulSetControl) UpdateStatefulSet(ctx context.Context, set *apps.StatefulSet, pods []*v1.Pod) (*apps.StatefulSetStatus, error) {
set = set.DeepCopy() // set is modified when a new revision is created in performUpdate. Make a copy now to avoid mutation errors.
// list all revisions and sort them
revisions, err := ssc.ListRevisions(set)
if err != nil {
return nil, err
}
history.SortControllerRevisions(revisions)
currentRevision, updateRevision, status, err := ssc.performUpdate(ctx, set, pods, revisions)
if err != nil {
errs := []error{err}
if agg, ok := err.(utilerrors.Aggregate); ok {
errs = agg.Errors()
}
return nil, utilerrors.NewAggregate(append(errs, ssc.truncateHistory(set, pods, revisions, currentRevision, updateRevision)))
}
// maintain the set's revision history limit
return status, ssc.truncateHistory(set, pods, revisions, currentRevision, updateRevision)
}
func (ssc *defaultStatefulSetControl) performUpdate(
ctx context.Context, set *apps.StatefulSet, pods []*v1.Pod, revisions []*apps.ControllerRevision) (*apps.ControllerRevision, *apps.ControllerRevision, *apps.StatefulSetStatus, error) {
var currentStatus *apps.StatefulSetStatus
logger := klog.FromContext(ctx)
// get the current, and update revisions
currentRevision, updateRevision, collisionCount, err := ssc.getStatefulSetRevisions(set, revisions)
if err != nil {
return currentRevision, updateRevision, currentStatus, err
}
// perform the main update function and get the status
currentStatus, err = ssc.updateStatefulSet(ctx, set, currentRevision, updateRevision, collisionCount, pods)
if err != nil && currentStatus == nil {
return currentRevision, updateRevision, nil, err
}
// make sure to update the latest status even if there is an error with non-nil currentStatus
statusErr := ssc.updateStatefulSetStatus(ctx, set, currentStatus)
if statusErr == nil {
logger.V(4).Info("Updated status", "statefulSet", klog.KObj(set),
"replicas", currentStatus.Replicas,
"readyReplicas", currentStatus.ReadyReplicas,
"currentReplicas", currentStatus.CurrentReplicas,
"updatedReplicas", currentStatus.UpdatedReplicas)
}
switch {
case err != nil && statusErr != nil:
logger.Error(statusErr, "Could not update status", "statefulSet", klog.KObj(set))
return currentRevision, updateRevision, currentStatus, err
case err != nil:
return currentRevision, updateRevision, currentStatus, err
case statusErr != nil:
return currentRevision, updateRevision, currentStatus, statusErr
}
logger.V(4).Info("StatefulSet revisions", "statefulSet", klog.KObj(set),
"currentRevision", currentStatus.CurrentRevision,
"updateRevision", currentStatus.UpdateRevision)
return currentRevision, updateRevision, currentStatus, nil
}
2.2.1 ssc.updateStatefulSet
updateStatefulSet方法是statefulset对象调谐操作中的核心方法,完成statefulset对象对于pod的创建、删除、更新、扩缩容等操作。此方法代码比较长,跟随我的节奏慢慢分析。
主要逻辑:
(1)看到第一个for循环,将statefulset的所有pod按ord(ord即为pod name中的序号)的值分到replicas和condemned两个数组中,序号小于statefulset对象期望副本数值的放到replicas数组(因为序号从0开始,所以是小于期望副本数值),大于等于期望副本数值的放到condemned数组,replicas数组代表正常的可用的pod列表,condemned数组中的是需要被删除的pod列表;在遍历pod时,同时根据pod的状态计算statefulset对象的status值;
(2)第二个for循环,当序号小于statefulset期望副本数值的pod未创建出来时,则根据statefulset对象中的pod模板,构建出相应序号值的pod对象(此时还并没有向apiserver发起创建pod的请求,只是构建好pod结构体);
(3)第三个和第四个for循环,遍历replicas和condemned两个数组,找到非healthy状态的最小序号的pod记录下来,并记录序号;
(4)当statefulset对象的DeletionTimestamp不为nil时,直接返回前面计算出来的statefulset的新status值,不再进行方法后续的逻辑处理;
(5)获取monotonic的值,当statefulset.Spec.PodManagementPolicy的值为Parallel时,monotonic的值为false,否则为true(Parallel代表statefulset controller可以并行的处理同一statefulset的pod,串行则代表在启动和终止下一个pod之前需要等待前一个pod变成ready状态或pod对象被删除掉);
(6)第五个for循环,遍历replicas数组,处理statefulset的pod,主要是做pod的创建(包括根据statefulset.Spec.VolumeClaimTemplates中定义的pod所需的pvc的创建):
(6.1)当pod处于fail状态(pod.Status.Phase的值为Failed)时,调用apiserver删除该pod(pod对应的pvc在这里不会做删除操作)并给replicas数组构建相应序号的新的pod结构体(用于下一步中重新创建该序号的pod);
(6.2)如果相应序号的pod未创建时,调用apiserver创建该序号的pod(包括创建pvc),且当monotonic为true时(statefulset没有配置Parallel),直接return,结束updateStatefulSet方法的执行;
(6.3)剩下的逻辑就是当没有配置Parallel时,将串行处理pod,在启动和终止下一个pod之前需要等待前一个pod变成ready状态或pod对象被删除掉,不再展开分析;
(7)第六个for循环,逆序(pod序号从大到小)遍历condemned数组,处理statefulset的pod,主要是做多余pod的删除,删除逻辑也受Parallel影响,不展开分析。
(8)判断statefulset的更新策略,若为OnDelete,则直接return(使用了该更新策略,则需要人工删除pod后才会重建相应序号的pod);
(9)获取滚动更新配置中的Partition值,当statefulset进行滚动更新时,小于等于该序号的pod将不会被更新;
(10)第七个for循环,主要是处理更新策略为RollingUpdate的statefulset对象的更新。statefulset的滚动更新,从序号大到小的顺序对pod进行更新,先删除后创建,且需等待当前序号的pod再次创建完成且状态为Ready时才能进行下一个pod的更新处理。
// updateStatefulSet performs the update function for a StatefulSet. This method creates, updates, and deletes Pods in
// the set in order to conform the system to the target state for the set. The target state always contains
// set.Spec.Replicas Pods with a Ready Condition. If the UpdateStrategy.Type for the set is
// RollingUpdateStatefulSetStrategyType then all Pods in the set must be at set.Status.CurrentRevision.
// If the UpdateStrategy.Type for the set is OnDeleteStatefulSetStrategyType, the target state implies nothing about
// the revisions of Pods in the set. If the UpdateStrategy.Type for the set is PartitionStatefulSetStrategyType, then
// all Pods with ordinal less than UpdateStrategy.Partition.Ordinal must be at Status.CurrentRevision and all other
// Pods must be at Status.UpdateRevision. If the returned error is nil, the returned StatefulSetStatus is valid and the
// update must be recorded. If the error is not nil, the method should be retried until successful.
func (ssc *defaultStatefulSetControl) updateStatefulSet(
ctx context.Context,
set *apps.StatefulSet,
currentRevision *apps.ControllerRevision,
updateRevision *apps.ControllerRevision,
collisionCount int32,
pods []*v1.Pod) (*apps.StatefulSetStatus, error) {
logger := klog.FromContext(ctx)
// get the current and update revisions of the set.
currentSet, err := ApplyRevision(set, currentRevision)
if err != nil {
return nil, err
}
updateSet, err := ApplyRevision(set, updateRevision)
if err != nil {
return nil, err
}
// set the generation, and revisions in the returned status
status := apps.StatefulSetStatus{}
status.ObservedGeneration = set.Generation
status.CurrentRevision = currentRevision.Name
status.UpdateRevision = updateRevision.Name
status.CollisionCount = new(int32)
*status.CollisionCount = collisionCount
updateStatus(&status, set.Spec.MinReadySeconds, currentRevision, updateRevision, pods)
replicaCount := int(*set.Spec.Replicas)
// slice that will contain all Pods such that getStartOrdinal(set) <= getOrdinal(pod) <= getEndOrdinal(set)
replicas := make([]*v1.Pod, replicaCount)
// slice that will contain all Pods such that getOrdinal(pod) < getStartOrdinal(set) OR getOrdinal(pod) > getEndOrdinal(set)
condemned := make([]*v1.Pod, 0, len(pods))
unhealthy := 0
var firstUnhealthyPod *v1.Pod
// 第一个for循环,将statefulset的pod分到replicas和condemned两个数组中,其中condemned数组中的pod代表需要被删除的
// First we partition pods into two lists valid replicas and condemned Pods
for _, pod := range pods {
if podInOrdinalRange(pod, set) {
// if the ordinal of the pod is within the range of the current number of replicas,
// insert it at the indirection of its ordinal
replicas[getOrdinal(pod)-getStartOrdinal(set)] = pod
} else if getOrdinal(pod) >= 0 {
// if the ordinal is valid, but not within the range add it to the condemned list
condemned = append(condemned, pod)
}
// If the ordinal could not be parsed (ord < 0), ignore the Pod.
}
// 第二个for循环,当序号小于statefulset期望副本数值的pod未创建出来时,则根据statefulset对象中的pod模板,构建出相应序号值的pod对象(此时还并没有向apiserver发起创建pod的请求,只是构建好pod结构体)
// for any empty indices in the sequence [0,set.Spec.Replicas) create a new Pod at the correct revision
for ord := getStartOrdinal(set); ord <= getEndOrdinal(set); ord++ {
replicaIdx := ord - getStartOrdinal(set)
if replicas[replicaIdx] == nil {
replicas[replicaIdx] = newVersionedStatefulSetPod(
currentSet,
updateSet,
currentRevision.Name,
updateRevision.Name, ord)
}
}
// sort the condemned Pods by their ordinals
sort.Sort(descendingOrdinal(condemned))
// 第三个和第四个for循环,遍历replicas和condemned两个数组,找到非healthy状态的最小序号的pod记录下来,并记录序号
// find the first unhealthy Pod
for i := range replicas {
if !isHealthy(replicas[i]) {
unhealthy++
if firstUnhealthyPod == nil {
firstUnhealthyPod = replicas[i]
}
}
}
// or the first unhealthy condemned Pod (condemned are sorted in descending order for ease of use)
for i := len(condemned) - 1; i >= 0; i-- {
if !isHealthy(condemned[i]) {
unhealthy++
if firstUnhealthyPod == nil {
firstUnhealthyPod = condemned[i]
}
}
}
if unhealthy > 0 {
logger.V(4).Info("StatefulSet has unhealthy Pods", "statefulSet", klog.KObj(set), "unhealthyReplicas", unhealthy, "pod", klog.KObj(firstUnhealthyPod))
}
// 当statefulset对象的DeletionTimestamp不为nil时,直接返回前面计算出来的statefulset的新status值,不再进行方法后续的逻辑处理
// If the StatefulSet is being deleted, don't do anything other than updating
// status.
if set.DeletionTimestamp != nil {
return &status, nil
}
// 获取monotonic的值,当statefulset.Spec.PodManagementPolicy的值为Parallel时,monotonic的值为false,否则为true
monotonic := !allowsBurst(set)
// First, process each living replica. Exit if we run into an error or something blocking in monotonic mode.
processReplicaFn := func(i int) (bool, error) {
return ssc.processReplica(ctx, set, updateSet, monotonic, replicas, i)
}
// 第五个for循环,遍历replicas数组,处理statefulset的pod,主要是做pod的创建
if shouldExit, err := runForAll(replicas, processReplicaFn, monotonic); shouldExit || err != nil {
updateStatus(&status, set.Spec.MinReadySeconds, currentRevision, updateRevision, replicas, condemned)
return &status, err
}
// Fix pod claims for condemned pods, if necessary.
if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetAutoDeletePVC) {
fixPodClaim := func(i int) (bool, error) {
if matchPolicy, err := ssc.podControl.ClaimsMatchRetentionPolicy(ctx, updateSet, condemned[i]); err != nil {
return true, err
} else if !matchPolicy {
if err := ssc.podControl.UpdatePodClaimForRetentionPolicy(ctx, updateSet, condemned[i]); err != nil {
return true, err
}
}
return false, nil
}
if shouldExit, err := runForAll(condemned, fixPodClaim, monotonic); shouldExit || err != nil {
updateStatus(&status, set.Spec.MinReadySeconds, currentRevision, updateRevision, replicas, condemned)
return &status, err
}
}
// 第六个for循环,逆序(pod序号从大到小)遍历condemned数组,处理statefulset的pod,主要是做多余pod的删除
// At this point, in monotonic mode all of the current Replicas are Running, Ready and Available,
// and we can consider termination.
// We will wait for all predecessors to be Running and Ready prior to attempting a deletion.
// We will terminate Pods in a monotonically decreasing order.
// Note that we do not resurrect Pods in this interval. Also note that scaling will take precedence over
// updates.
processCondemnedFn := func(i int) (bool, error) {
return ssc.processCondemned(ctx, set, firstUnhealthyPod, monotonic, condemned, i)
}
if shouldExit, err := runForAll(condemned, processCondemnedFn, monotonic); shouldExit || err != nil {
updateStatus(&status, set.Spec.MinReadySeconds, currentRevision, updateRevision, replicas, condemned)
return &status, err
}
updateStatus(&status, set.Spec.MinReadySeconds, currentRevision, updateRevision, replicas, condemned)
// 判断statefulset的更新策略,若为OnDelete,则直接return(使用了该更新策略,则需要人工删除pod后才会重建相应序号的pod)
// for the OnDelete strategy we short circuit. Pods will be updated when they are manually deleted.
if set.Spec.UpdateStrategy.Type == apps.OnDeleteStatefulSetStrategyType {
return &status, nil
}
if utilfeature.DefaultFeatureGate.Enabled(features.MaxUnavailableStatefulSet) {
return updateStatefulSetAfterInvariantEstablished(ctx,
ssc,
set,
replicas,
updateRevision,
status,
)
}
// 获取滚动更新配置中的Partition值,当statefulset进行滚动更新时,小于等于该序号的pod将不会被更新
// we compute the minimum ordinal of the target sequence for a destructive update based on the strategy.
updateMin := 0
if set.Spec.UpdateStrategy.RollingUpdate != nil {
updateMin = int(*set.Spec.UpdateStrategy.RollingUpdate.Partition)
}
// 第七个for循环,主要是处理更新策略为RollingUpdate的statefulset对象的更新
// we terminate the Pod with the largest ordinal that does not match the update revision.
for target := len(replicas) - 1; target >= updateMin; target-- {
// delete the Pod if it is not already terminating and does not match the update revision.
if getPodRevision(replicas[target]) != updateRevision.Name && !isTerminating(replicas[target]) {
logger.V(2).Info("Pod of StatefulSet is terminating for update",
"statefulSet", klog.KObj(set), "pod", klog.KObj(replicas[target]))
if err := ssc.podControl.DeleteStatefulPod(set, replicas[target]); err != nil {
if !errors.IsNotFound(err) {
return &status, err
}
}
status.CurrentReplicas--
return &status, err
}
// wait for unhealthy Pods on update
if !isHealthy(replicas[target]) {
logger.V(4).Info("StatefulSet is waiting for Pod to update",
"statefulSet", klog.KObj(set), "pod", klog.KObj(replicas[target]))
return &status, nil
}
}
return &status, nil
}
结合上面对该方法的分析,来总结下在此方法中都有哪些步骤涉及了statefulset对象对于pod的创建、删除、扩缩容、更新操作: 1.创建:主要是(6)第五个for循环; 2.删除:主要是(7)第六个for循环; 3.扩缩容: (1)~(7); 4.更新:主要是(8)(9)与(10)第七个for循环,其中(8)为OnDelete更新策略的处理,(9)(10)为滚动更新策略的处理。
statefulset更新策略
(1)OnDelete:使用 OnDelete 更新策略时,在更新 statefulset pod模板后,只有当你手动删除老的 statefulset pods 之后,新的 statefulset Pod 才会被自动创建。
(2)RollingUpdate:使用 RollingUpdate 更新策略时,在更新 statefulset pod模板后, 老的 statefulset pods 将被删除,并且将根据滚动更新配置自动创建新的 statefulset pods。滚动更新期间,每个序号的statefulset pod最多只能有一个,且滚动更新下一个pod之前,需等待前一个pod更新完成并处于ready状态。与statefulset pod按0 - n的顺序创建不同,滚动更新时Pod按逆序的方式(即n - 0)删除并创建。
statefulset的滚动升级中还有一个Partition配置,在设置partition后,滚动更新过程中,statefulset的Pod中序号大于或等于partition的Pod会进行滚动升级,而其余的Pod保持不变,不会进行滚动更新。
statefulset pod的命名规则、pod创建与删除
如果创建一个名称为web、replicas为3的statefulset对象,则其pod名称分别为web-0、web-1、web-2。
statefulset pod的创建按0 - n的顺序创建,且在创建下一个pod之前,需要等待前一个pod创建完成并处于ready状态。
同样拿上面的例子来说明,在web statefulset创建后,3 个 Pod 将按顺序创建 web-0,web-1,web-2。在 web-0 处于 ready 状态之前,web-1 将不会被创建,同样当 web-1 处于ready状态之前 web-2也不会被创建。如果在 web-1 ready后,web-2 创建之前, web-0 不处于ready状态了,这个时候 web-2 将不会被创建,直到 web-0 再次回到ready状态。
statefulset滚动更新或缩容过程中pod的删除按n - 0的顺序删除,且在删除下一个pod之前,需要等待前一个pod删除完成。
另外,当statefulset.Spec.VolumeClaimTemplates中定义了pod所需的pvc时,statefulset controller在创建pod时,会同时创建对应的pvc出来,但删除pod时,不会做对应pvc的删除操作,这些pvc需要人工额外做删除操作。
想要原文可以加作者v:mkjnnm