【kubernetes/k8s源码分析】kubelet源码分析之volume manager源码分析

最新推荐文章于 2024-05-10 17:04:18 发布
最新推荐文章于 2024-05-10 17:04:18 发布
阅读量6.2k 收藏 5
点赞数 5
分类专栏： # kubelet
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/zhonglinzhang/article/details/82800287
版权

      kubernetes version: v1.12.1

      kubelet调用VolumeManager，为pods准备存储设备，存储设备就绪会挂载存储设备到pod所在的节点上，并在容器启动的时候挂载在容器指定的目录中；同时，删除卸载不在使用的存储；

      kubernetes采用Volume Plugins来实现存储卷的挂载等操作

     这图谁画的咋这么好呢，借来用用！！！

 

        kubelet会调用VolumeManager，为pods准备存储设备，存储设备就绪会挂载存储设备到pod所在的节点上，并在容器启动的时候挂载在容器指定的目录中；同时，删除卸载不在使用的存储

 

VolumeManager接口

  • 运行在kubelet 里让存储Ready的部件，主要是mount/unmount（attach/detach可选）
  • pod调度到这个node上后才会有卷的相应操作，所以它的触发端是kubelet（严格讲是kubelet里的pod manager），根据Pod Manager里pod spec里申明的存储来触发卷的挂载操作
  • Kubelet会监听到调度到该节点上的pod声明，会把pod缓存到Pod Manager中，VolumeManager通过Pod Manager获取PV/PVC的状态，并进行分析出具体的attach/detach、mount/umount, 操作然后调用plugin进行相应的业务处理
// VolumeManager runs a set of asynchronous loops that figure out which volumes
// need to be attached/mounted/unmounted/detached based on the pods scheduled on
// this node and makes it so.
type VolumeManager interface {

	Run(sourcesReady config.SourcesReady, stopCh <-chan struct{})

	WaitForAttachAndMount(pod *v1.Pod) error

	GetMountedVolumesForPod(podName types.UniquePodName) container.VolumeMap

	GetExtraSupplementalGroupsForPod(pod *v1.Pod) []int64

	GetVolumesInUse() []v1.UniqueVolumeName

	ReconcilerStatesHasBeenSynced() bool

	VolumeIsAttached(volumeName v1.UniqueVolumeName) bool

	MarkVolumesAsReportedInUse(volumesReportedAsInUse []v1.UniqueVolumeName)
}

volumeManager结构体

   路径：pkg/kubelet/volumemanager/volume_manager.go

  • volumeManager结构体实现了VolumeManager接口，主要有两个需要注意：
  • desiredStateOfWorld：预期状态，volume需要被attach，哪些pods引用这个volume
  • actualStateOfWorld：实际状态，volume已经被atttach哪个node，哪个pod mount volume
/ volumeManager implements the VolumeManager interface
type volumeManager struct {
	// kubeClient is the kube API client used by DesiredStateOfWorldPopulator to
	// communicate with the API server to fetch PV and PVC objects
	kubeClient clientset.Interface

	// volumePluginMgr is the volume plugin manager used to access volume
	// plugins. It must be pre-initialized.
	volumePluginMgr *volume.VolumePluginMgr

	// desiredStateOfWorld is a data structure containing the desired state of
	// the world according to the volume manager: i.e. what volumes should be
	// attached and which pods are referencing the volumes).
	// The data structure is populated by the desired state of the world
	// populator using the kubelet pod manager.
	desiredStateOfWorld cache.DesiredStateOfWorld

	// actualStateOfWorld is a data structure containing the actual state of
	// the world according to the manager: i.e. which volumes are attached to
	// this node and what pods the volumes are mounted to.
	// The data structure is populated upon successful completion of attach,
	// detach, mount, and unmount actions triggered by the reconciler.
	actualStateOfWorld cache.ActualStateOfWorld

	// operationExecutor is used to start asynchronous attach, detach, mount,
	// and unmount operations.
	operationExecutor operationexecutor.OperationExecutor

	// reconciler runs an asynchronous periodic loop to reconcile the
	// desiredStateOfWorld with the actualStateOfWorld by triggering attach,
	// detach, mount, and unmount operations using the operationExecutor.
	reconciler reconciler.Reconciler

	// desiredStateOfWorldPopulator runs an asynchronous periodic loop to
	// populate the desiredStateOfWorld using the kubelet PodManager.
	desiredStateOfWorldPopulator populator.DesiredStateOfWorldPopulator
}

 

1. volumeManager初始化

    路径： pkg/kubelet/kubelet.go文件中NewMainKubelet函数

    初始化VolumeManager结构体，结构体VolumeManager如上所示：

	// setup volumeManager
	klet.volumeManager = volumemanager.NewVolumeManager(
		kubeCfg.EnableControllerAttachDetach,
		nodeName,
		klet.podManager,
		klet.statusManager,
		klet.kubeClient,
		klet.volumePluginMgr,
		klet.containerRuntime,
		kubeDeps.Mounter,
		klet.getPodsDir(),
		kubeDeps.Recorder,
		experimentalCheckNodeCapabilitiesBeforeMount,
		keepTerminatedPodVolumes)

   1.1 启动VolumeManager的run方法

  •        goroutine启动descriedStateOfWorldPopulator的Run函数，从apiserver同步到的pod信息，来更新DesiredStateOfWorld
  •        goroutine启动reconciler的Run函数 ，预期状态和实际状态的协调者，负责调整实际状态至预期状态
func (vm *volumeManager) Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) {
	defer runtime.HandleCrash()

	go vm.desiredStateOfWorldPopulator.Run(sourcesReady, stopCh)
	glog.V(2).Infof("The desired_state_of_world populator starts")

	glog.Infof("Starting Kubelet Volume Manager")
	go vm.reconciler.Run(stopCh)

	metrics.Register(vm.actualStateOfWorld, vm.desiredStateOfWorld, vm.volumePluginMgr)

	<-stopCh
	glog.Infof("Shutting down Kubelet Volume Manager")
}

  1.2 desiredStateOfWorldPopulator Run函数

     从apiserver同步到的pod信息，来更新DesiredStateOfWorld，主要函数为populatorLoop函数第2章节讲解

func (dswp *desiredStateOfWorldPopulator) Run(sourcesReady config.SourcesReady, stopCh <-chan struct{}) {
	// Wait for the completion of a loop that started after sources are all ready, then set hasAddedPods accordingly
	glog.Infof("Desired state populator starts to run")
	wait.PollUntil(dswp.loopSleepDuration, func() (bool, error) {
		done := sourcesReady.AllReady()
		dswp.populatorLoop()
		return done, nil
	}, stopCh)
	dswp.hasAddedPodsLock.Lock()
	dswp.hasAddedPods = true
	dswp.hasAddedPodsLock.Unlock()
	wait.Until(dswp.populatorLoop, dswp.loopSleepDuration, stopCh)
}

  1.3 reconciler Run函数

     预期状态和实际状态的协调者，负责调整实际状态至预期状态，reconcile函数第4章节讲解

func (rc *reconciler) Run(stopCh <-chan struct{}) {
	wait.Until(rc.reconciliationLoopFunc(), rc.loopSleepDuration, stopCh)
}

func (rc *reconciler) reconciliationLoopFunc() func() {
	return func() {
		rc.reconcile()

		// Sync the state with the reality once after all existing pods are added to the desired state from all sources.
		// Otherwise, the reconstruct process may clean up pods' volumes that are still in use because
		// desired state of world does not contain a complete list of pods.
		if rc.populatorHasAddedPods() && !rc.StatesHasBeenSynced() {
			glog.Infof("Reconciler: start to sync state")
			rc.sync()
		}
	}
}

 

2. populatorLoop函数

  findAndAddNewPods函数主要是调用podManager获取所有pods，调用processPodVolumes更新desiredStateOfWorld

func (dswp *desiredStateOfWorldPopulator) populatorLoop() {
	dswp.findAndAddNewPods()

	// findAndRemoveDeletedPods() calls out to the container runtime to
	// determine if the containers for a given pod are terminated. This is
	// an expensive operation, therefore we limit the rate that
	// findAndRemoveDeletedPods() is called independently of the main
	// populator loop.
	if time.Since(dswp.timeOfLastGetPodStatus) < dswp.getPodStatusRetryDuration {
		glog.V(5).Infof(
			"Skipping findAndRemoveDeletedPods(). Not permitted until %v (getPodStatusRetryDuration %v).",
			dswp.timeOfLastGetPodStatus.Add(dswp.getPodStatusRetryDuration),
			dswp.getPodStatusRetryDuration)

		return
	}

	dswp.findAndRemoveDeletedPods()
}

  2.1 findAndAddNewPods函数

  •     调用podManager获取所有的pods
  •     调用processPodVolumes去更新desiredStateOfWorld，第3章节讲解
// Iterate through all pods and add to desired state of world if they don't
// exist but should
func (dswp *desiredStateOfWorldPopulator) findAndAddNewPods() {
	// Map unique pod name to outer volume name to MountedVolume.
	mountedVolumesForPod := make(map[volumetypes.UniquePodName]map[string]cache.MountedVolume)
	if utilfeature.DefaultFeatureGate.Enabled(features.ExpandInUsePersistentVolumes) {
		for _, mountedVolume := range dswp.actualStateOfWorld.GetMountedVolumes() {
			mountedVolumes, exist := mountedVolumesForPod[mountedVolume.PodName]
			if !exist {
				mountedVolumes = make(map[string]cache.MountedVolume)
				mountedVolumesForPod[mountedVolume.PodName] = mountedVolumes
			}
			mountedVolumes[mountedVolume.OuterVolumeSpecName] = mountedVolume
		}
	}

	processedVolumesForFSResize := sets.NewString()
	for _, pod := range dswp.podManager.GetPods() {
		if dswp.isPodTerminated(pod) {
			// Do not (re)add volumes for terminated pods
			continue
		}
		dswp.processPodVolumes(pod, mountedVolumesForPod, processedVolumesForFSResize)
	}
}

 

3. processPodVolumes函数

    更新desiredStateOfWorld

  3.1 如果先前在processedPods map中，表示无需处理

	uniquePodName := util.GetUniquePodName(pod)
	if dswp.podPreviouslyProcessed(uniquePodName) {
		return
	}

   3.2 对pod下所有container的volumeDevices与volumeMounts加入map中 

   volumeDevices    <[]Object>
     volumeDevices is the list of block devices to be used by the container.
     This is an alpha feature and may change in the future.

   volumeMounts    <[]Object>
     Pod volumes to mount into the container's filesystem. Cannot be updated.

func (dswp *desiredStateOfWorldPopulator) makeVolumeMap(containers []v1.Container) (map[string]bool, map[string]bool) {
	volumeDevicesMap := make(map[string]bool)
	volumeMountsMap := make(map[string]bool)

	for _, container := range containers {
		if container.VolumeMounts != nil {
			for _, mount := range container.VolumeMounts {
				volumeMountsMap[mount.Name] = true
			}
		}
		// TODO: remove feature gate check after no longer needed
		if utilfeature.DefaultFeatureGate.Enabled(features.BlockVolume) &&
			container.VolumeDevices != nil {
			for _, device := range container.VolumeDevices {
				volumeDevicesMap[device.Name] = true
			}
		}
	}

	return volumeMountsMap, volumeDevicesMap
}

  3.3 AddPodToVolume函数

    3.3.1 调用FindPluginBySpec函数根据volume.spec找到volume plugin

	volumePlugin, err := dsw.volumePluginMgr.FindPluginBySpec(volumeSpec)
	if err != nil || volumePlugin == nil {
		return "", fmt.Errorf(
			"failed to get Plugin from volumeSpec for volume %q err=%v",
			volumeSpec.Name(),
			err)
	}

    3.3.2 isAttachableVolume函数，检查插件是否需要attach

     不是所有的插件都需要实现AttachableVolumePlugin接口

	// The unique volume name used depends on whether the volume is attachable
	// or not.
	attachable := dsw.isAttachableVolume(volumeSpec)
	if attachable {
		// For attachable volumes, use the unique volume name as reported by
		// the plugin.
		volumeName, err =
			util.GetUniqueVolumeNameFromSpec(volumePlugin, volumeSpec)
		if err != nil {
			return "", fmt.Errorf(
				"failed to GetUniqueVolumeNameFromSpec for volumeSpec %q using volume plugin %q err=%v",
				volumeSpec.Name(),
				volumePlugin.GetPluginName(),
				err)
		}
	} else {
		// For non-attachable volumes, generate a unique name based on the pod
		// namespace and name and the name of the volume within the pod.
		volumeName = util.GetUniqueVolumeNameForNonAttachableVolume(podName, volumePlugin, volumeSpec)
	}

    3.3.3 记录volume与pod之间的关系

	if _, volumeExists := dsw.volumesToMount[volumeName]; !volumeExists {
		dsw.volumesToMount[volumeName] = volumeToMount{
			volumeName:              volumeName,
			podsToMount:             make(map[types.UniquePodName]podToMount),
			pluginIsAttachable:      attachable,
			pluginIsDeviceMountable: deviceMountable,
			volumeGidValue:          volumeGidValue,
			reportedInUse:           false,
		}
	}

	// Create new podToMount object. If it already exists, it is refreshed with
	// updated values (this is required for volumes that require remounting on
	// pod update, like Downward API volumes).
	dsw.volumesToMount[volumeName].podsToMount[podName] = podToMount{
		podName:             podName,
		pod:                 pod,
		volumeSpec:          volumeSpec,
		outerVolumeSpecName: outerVolumeSpecName,
	}

     3.3.4 对pod name标记为已处理，actual_state_of_world标记重新挂载

	// some of the volume additions may have failed, should not mark this pod as fully processed
	if allVolumesAdded {
		dswp.markPodProcessed(uniquePodName)
		// New pod has been synced. Re-mount all volumes that need it
		// (e.g. DownwardAPI)
		dswp.actualStateOfWorld.MarkRemountRequired(uniquePodName)
	}

 

4. reconcile函数

  路径 pkg/kubelet/volumemanager/reconciler/reconciler.go

func (rc *reconciler) reconcile() {
	// Unmounts are triggered before mounts so that a volume that was
	// referenced by a pod that was deleted and is now referenced by another
	// pod is unmounted from the first pod before being mounted to the new
	// pod.

  4.1 对于实际已经挂载的与预期不一样的需要unmount

      UnmountVolume函数中处理，volume分为filesystem与block，在desiredStateOfWorld不包括actualStateOfWorld的情况需要unmount

	// Ensure volumes that should be unmounted are unmounted.
	for _, mountedVolume := range rc.actualStateOfWorld.GetMountedVolumes() {
		if !rc.desiredStateOfWorld.PodExistsInVolume(mountedVolume.PodName, mountedVolume.VolumeName) {
			// Volume is mounted, unmount it
			glog.V(5).Infof(mountedVolume.GenerateMsgDetailed("Starting operationExecutor.UnmountVolume", ""))
			err := rc.operationExecutor.UnmountVolume(
				mountedVolume.MountedVolume, rc.actualStateOfWorld, rc.kubeletPodsDir)
			if err != nil &&
				!nestedpendingoperations.IsAlreadyExists(err) &&
				!exponentialbackoff.IsExponentialBackoff(err) {
				// Ignore nestedpendingoperations.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
				// Log all other errors.
				glog.Errorf(mountedVolume.GenerateErrorDetailed(fmt.Sprintf("operationExecutor.UnmountVolume failed (controllerAttachDetachEnabled %v)", rc.controllerAttachDetachEnabled), err).Error())
			}
			if err == nil {
				glog.Infof(mountedVolume.GenerateMsgDetailed("operationExecutor.UnmountVolume started", ""))
			}
		}
	}

  4.2 从desiredStateOfWorld中获取需要mount的volomes

    调用PodExistesInVolume与实际podName进行对比

    attachdVolumes不存在volumeName则为volumeNotAttachedError

    mountedPods中存在podName，podObj.remountRequired为 newRemountRequiredError, podObj.fsResizeRequired为newFsResizeRequiredError

// Ensure volumes that should be attached/mounted are attached/mounted.
	for _, volumeToMount := range rc.desiredStateOfWorld.GetVolumesToMount() {
		volMounted, devicePath, err := rc.actualStateOfWorld.PodExistsInVolume(volumeToMount.PodName, volumeToMount.VolumeName)
		volumeToMount.DevicePath = devicePath

  4.3 err为volumeNotAttachedError的情况

    如果volume没有attach，actualStateOfWorld结构体中attachedVolumes没有volumeName，在则调用AttachVolume操作

       if cache.IsVolumeNotAttachedError(err) {
			if rc.controllerAttachDetachEnabled || !volumeToMount.PluginIsAttachable {
				// Volume is not attached (or doesn't implement attacher), kubelet attach is disabled, wait
				// for controller to finish attaching volume.
				glog.V(5).Infof(volumeToMount.GenerateMsgDetailed("Starting operationExecutor.VerifyControllerAttachedVolume", ""))
				err := rc.operationExecutor.VerifyControllerAttachedVolume(
					volumeToMount.VolumeToMount,
					rc.nodeName,
					rc.actualStateOfWorld)
				if err != nil &&
					!nestedpendingoperations.IsAlreadyExists(err) &&
					!exponentialbackoff.IsExponentialBackoff(err) {
					// Ignore nestedpendingoperations.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
					// Log all other errors.
					glog.Errorf(volumeToMount.GenerateErrorDetailed(fmt.Sprintf("operationExecutor.VerifyControllerAttachedVolume failed (controllerAttachDetachEnabled %v)", rc.controllerAttachDetachEnabled), err).Error())
				}
				if err == nil {
					glog.Infof(volumeToMount.GenerateMsgDetailed("operationExecutor.VerifyControllerAttachedVolume started", ""))
				}
			} else {
				// Volume is not attached to node, kubelet attach is enabled, volume implements an attacher,
				// so attach it
				volumeToAttach := operationexecutor.VolumeToAttach{
					VolumeName: volumeToMount.VolumeName,
					VolumeSpec: volumeToMount.VolumeSpec,
					NodeName:   rc.nodeName,
				}
				glog.V(5).Infof(volumeToAttach.GenerateMsgDetailed("Starting operationExecutor.AttachVolume", ""))
				err := rc.operationExecutor.AttachVolume(volumeToAttach, rc.actualStateOfWorld)
				if err != nil &&
					!nestedpendingoperations.IsAlreadyExists(err) &&
					!exponentialbackoff.IsExponentialBackoff(err) {
					// Ignore nestedpendingoperations.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
					// Log all other errors.
					glog.Errorf(volumeToMount.GenerateErrorDetailed(fmt.Sprintf("operationExecutor.AttachVolume failed (controllerAttachDetachEnabled %v)", rc.controllerAttachDetachEnabled), err).Error())
				}
				if err == nil {
					glog.Infof(volumeToMount.GenerateMsgDetailed("operationExecutor.AttachVolume started", ""))
				}
			}
		}

  4.4 volume没有被mount，error为remountRequiredError情况

    调用MountVolume函数操作，这个分为filesystem和block

else if !volMounted || cache.IsRemountRequiredError(err) {
			// Volume is not mounted, or is already mounted, but requires remounting
			remountingLogStr := ""
			isRemount := cache.IsRemountRequiredError(err)
			if isRemount {
				remountingLogStr = "Volume is already mounted to pod, but remount was requested."
			}
			glog.V(4).Infof(volumeToMount.GenerateMsgDetailed("Starting operationExecutor.MountVolume", remountingLogStr))
			err := rc.operationExecutor.MountVolume(
				rc.waitForAttachTimeout,
				volumeToMount.VolumeToMount,
				rc.actualStateOfWorld,
				isRemount)
			if err != nil &&
				!nestedpendingoperations.IsAlreadyExists(err) &&
				!exponentialbackoff.IsExponentialBackoff(err) {
				// Ignore nestedpendingoperations.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
				// Log all other errors.
				glog.Errorf(volumeToMount.GenerateErrorDetailed(fmt.Sprintf("operationExecutor.MountVolume failed (controllerAttachDetachEnabled %v)", rc.controllerAttachDetachEnabled), err).Error())
			}
			if err == nil {
				if remountingLogStr == "" {
					glog.V(1).Infof(volumeToMount.GenerateMsgDetailed("operationExecutor.MountVolume started", remountingLogStr))
				} else {
					glog.V(5).Infof(volumeToMount.GenerateMsgDetailed("operationExecutor.MountVolume started", remountingLogStr))
				}
			}
		}

  4.5 error为fsResizeRequiredError情况

    调用ExpandVolumeFSWithoutUnmounting函数

            else if cache.IsFSResizeRequiredError(err) &&
			utilfeature.DefaultFeatureGate.Enabled(features.ExpandInUsePersistentVolumes) {
			glog.V(4).Infof(volumeToMount.GenerateMsgDetailed("Starting operationExecutor.ExpandVolumeFSWithoutUnmounting", ""))
			err := rc.operationExecutor.ExpandVolumeFSWithoutUnmounting(
				volumeToMount.VolumeToMount,
				rc.actualStateOfWorld)
			if err != nil &&
				!nestedpendingoperations.IsAlreadyExists(err) &&
				!exponentialbackoff.IsExponentialBackoff(err) {
				// Ignore nestedpendingoperations.IsAlreadyExists and exponentialbackoff.IsExponentialBackoff errors, they are expected.
				// Log all other errors.
				glog.Errorf(volumeToMount.GenerateErrorDetailed("operationExecutor.ExpandVolumeFSWithoutUnmounting failed", err).Error())
			}
			if err == nil {
				glog.V(4).Infof(volumeToMount.GenerateMsgDetailed("operationExecutor.ExpandVolumeFSWithoutUnmounting started", ""))
			}
		}

 

5. operationExecutor

    NewVolumeManager函數中初始化operationExecutor的函數爲NewOperationExecutor

		operationExecutor: operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator(
			kubeClient,
			volumePluginMgr,
			recorder,
			checkNodeCapabilitiesBeforeMount,
			volumepathhandler.NewBlockVolumePathHandler())),


// NewOperationExecutor returns a new instance of OperationExecutor.
func NewOperationExecutor(
	operationGenerator OperationGenerator) OperationExecutor {

	return &operationExecutor{
		pendingOperations: nestedpendingoperations.NewNestedPendingOperations(
			true /* exponentialBackOffOnError */),
		operationGenerator: operationGenerator,
	}
}

    5.1 MountVolume函数

     路径 pkg/volume/util/operationexecutor/operation_executor.go

     根据--feature-gates中BlockVolume=true|false (ALPHA - default=false)，执行函数GenerateMountVolumeFunc

func (oe *operationExecutor) MountVolume(
	waitForAttachTimeout time.Duration,
	volumeToMount VolumeToMount,
	actualStateOfWorld ActualStateOfWorldMounterUpdater,
	isRemount bool) error {
	fsVolume, err := util.CheckVolumeModeFilesystem(volumeToMount.VolumeSpec)
	if err != nil {
		return err
	}
	var generatedOperations volumetypes.GeneratedOperations
	if fsVolume {
		// Filesystem volume case
		// Mount/remount a volume when a volume is attached
		generatedOperations, err = oe.operationGenerator.GenerateMountVolumeFunc(
			waitForAttachTimeout, volumeToMount, actualStateOfWorld, isRemount)

	} else {
		// Block volume case
		// Creates a map to device if a volume is attached
		generatedOperations, err = oe.operationGenerator.GenerateMapVolumeFunc(
			waitForAttachTimeout, volumeToMount, actualStateOfWorld)
	}
	if err != nil {
		return err
	}
	// Avoid executing mount/map from multiple pods referencing the
	// same volume in parallel
	podName := nestedpendingoperations.EmptyUniquePodName

	// TODO: remove this -- not necessary
	if !volumeToMount.PluginIsAttachable && !volumeToMount.PluginIsDeviceMountable {
		// volume plugins which are Non-attachable and Non-deviceMountable can execute mount for multiple pods
		// referencing the same volume in parallel
		podName = util.GetUniquePodName(volumeToMount.Pod)
	}

	// TODO mount_device
	return oe.pendingOperations.Run(
		volumeToMount.VolumeName, podName, generatedOperations)
}

    5.1.1 GenerateMountVolumeFunc函数

    路径pkg/volume/util/opeationexecutor/operation_generator.go

// Generates the MountVolume function needed to perform the mount of a volume plugin
func (og *operationGenerator) GenerateMountVolumeFunc(
	waitForAttachTimeout time.Duration,
	volumeToMount VolumeToMount,
	actualStateOfWorld ActualStateOfWorldMounterUpdater,
	isRemount bool) (volumetypes.GeneratedOperations, error)

    5.1.1.1 FindPluginBySpec函数遍历所有的plugin判断volumeSpec符合哪种plugin

	// Get mounter plugin
	volumePlugin, err :=
		og.volumePluginMgr.FindPluginBySpec(volumeToMount.VolumeSpec)
	if err != nil || volumePlugin == nil {
		return volumetypes.GeneratedOperations{}, volumeToMount.GenerateErrorDetailed("MountVolume.FindPluginBySpec failed", err)
	}

    5.1.1.2 NewMouter根据插件调用

      比如hostPath或者nfs等（https://blog.csdn.net/zhonglinzhang/article/details/86534780分析插件） 

	volumeMounter, newMounterErr := volumePlugin.NewMounter(
		volumeToMount.VolumeSpec,
		volumeToMount.Pod,
		volume.VolumeOptions{})
	if newMounterErr != nil {
		eventErr, detailedErr := volumeToMount.GenerateError("MountVolume.NewMounter initialization failed", newMounterErr)
		og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FailedMountVolume, eventErr.Error())
		return volumetypes.GeneratedOperations{}, detailedErr
	}

    5.1.1.3 定义mountVolumeFunc函数

    执行plugin的SetUp方法，以及更新actual state of world

	mountVolumeFunc := func() (error, error) {
		devicePath := volumeToMount.DevicePath
		if volumeAttacher != nil {
			
		}

		if volumeDeviceMounter != nil {
			
		}


		// Execute mount
		mountErr := volumeMounter.SetUp(fsGroup)


		// Update actual state of world
		markVolMountedErr := actualStateOfWorld.MarkVolumeAsMounted(
			volumeToMount.PodName,
			volumeToMount.Pod.UID,
			volumeToMount.VolumeName,
			volumeMounter,
			nil,
			volumeToMount.OuterVolumeSpecName,
			volumeToMount.VolumeGidValue,
			volumeToMount.VolumeSpec)

		return nil, nil

    5.1.1.4 返回GeneratedOperations结构，包括方法字段

	return volumetypes.GeneratedOperations{
		OperationFunc:     mountVolumeFunc,
		EventRecorderFunc: eventRecorderFunc,
		CompleteFunc:      util.OperationCompleteHook(volumePlugin.GetPluginName(), "volume_mount"),
	}, nil

    5.1.2 pendingOperations

    根据pendingOperations: nestedpendingoperations.NewNestedPendingOperations，nestedPendingOperations实现了NestedPendingOperations接口，包括Run方法

    路径 pkg/volume/util/nestedpendingoperations/nestedpendingoperations.go

   5.1.2.1 pendingOperations Run函数

   异步处理主要是调用5.1.1.3中定义的mountVolumeFunc函数

func (grm *nestedPendingOperations) Run(
	volumeName v1.UniqueVolumeName,
	podName types.UniquePodName,
	generatedOperations types.GeneratedOperations) error {
	grm.lock.Lock()
	defer grm.lock.Unlock()
	opExists, previousOpIndex := grm.isOperationExists(volumeName, podName)

	go func() (eventErr, detailedErr error) {
		// Handle unhandled panics (very unlikely)
		defer k8sRuntime.HandleCrash()
		// Handle completion of and error, if any, from operationFunc()
		defer grm.operationComplete(volumeName, podName, &detailedErr)
		if generatedOperations.CompleteFunc != nil {
			defer generatedOperations.CompleteFunc(&detailedErr)
		}
		if generatedOperations.EventRecorderFunc != nil {
			defer generatedOperations.EventRecorderFunc(&eventErr)
		}
		// Handle panic, if any, from operationFunc()
		defer k8sRuntime.RecoverFromPanic(&detailedErr)
		return generatedOperations.OperationFunc()
	}()

	return nil
}

     

6. SyncPod 函数

   同步 pod 时，等待 pod attach 和 mount 完成

// Volume manager will not mount volumes for terminated pods
if !kl.podIsTerminated(pod) {
	// Wait for volumes to attach/mount
	if err := kl.volumeManager.WaitForAttachAndMount(pod); err != nil {
		kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedMountVolume, "Unable to mount volumes for pod %q: %v", format.Pod(pod), err)
		klog.Errorf("Unable to mount volumes for pod %q: %v; skipping pod", format.Pod(pod), err)
		return err
	}
}

    6.1 WaitForAttachAndMount 函数

      获取 pod 所有的 volume 

func (vm *volumeManager) WaitForAttachAndMount(pod *v1.Pod) error {
	if pod == nil {
		return nil
	}

	expectedVolumes := getExpectedVolumes(pod)
	if len(expectedVolumes) == 0 {
		// No volumes to verify
		return nil
	}

    6.2 verifyVolumesMountedFunc 函数

      没有被 mount 的volume 数量为0，表示成功完成挂载

// verifyVolumesMountedFunc returns a method that returns true when all expected
// volumes are mounted.
func (vm *volumeManager) verifyVolumesMountedFunc(podName types.UniquePodName, expectedVolumes []string) wait.ConditionFunc {
	return func() (done bool, err error) {
		return len(vm.getUnmountedVolumes(podName, expectedVolumes)) == 0, nil
	}
}

 

张忠琳
关注
  • 5
    点赞
  • 踩
  • 5
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
【kubernetes/k8s源码分析】kubernetes volume plugin源码分析
zhonglinzhang
01-18 3300
plugin调用函数： 文章https://blog.csdn.net/zhonglinzhang/article/details/82800287分析volume manager调用 reconcile -> MountVolume -> NewMouter根据插件调用 -> 执行plugin的SetUp volumePlugin接口 存储提...
kubelet之volume manager源码分析
良凯尔的博客
06-06 950
kubernetes ceph-csi分析目录导航 基于tag v1.17.4 https://github.com/kubernetes/kubernetes/releases/tag/v1.17.4 概述 volume manager存在于kubelet中，主要是管理卷的attach/detach（与AD controller作用相同，通过kubelet启动参数控制哪个组件来做该操作，后续会详细介绍）、mount/umount等操作。 简介 容器的存储挂载分为两大步： （1）attach； （2）mou
K8s源码分析（一）-K8s调度框架及调度器初始化介绍
最新发布
slipegg的博客
05-10 822
对K8s调度框架进行了介绍，还介绍了Cobra，并分析了K8s调度器的初始化代码
Kubernetes volumes简介
weixin_34294649的博客
04-07 483
2019独角兽企业重金招聘Python工程师标准>>> ...
【kubernetes/k8s源码分析】kubelet pod 挂载volume源码分析
zhonglinzhang
05-07 3917
1. NewVolumeManager 初始化volumeManager volumePluginMgr 使用volume 插件 实例化 volumeManager，实例化desiredStateOfWorldPopulator 这个reconciler是主要调整，attach detach mount unmount操作的，这个主要讲解 ...
kubernetes volume 源码详解
xinghun_4的专栏
05-24 1303
kubernetes volume 相关源码详解
linuxfoundation kubernetes/k8s CKS考试实验手册
01-28
LFS260-labs_V2021-01-07.pdf
k8s-auto-kubernetes/k8s一键安装
12-15
kubernetes/k8s自动安装程序，版本对应:v1.18.2，linux环境 使用kubeadm安装，改程序若环境不符合要求，是不能一键安装的，需按照程序指示分布安装，该程序是为了搭建测试环境时，简化繁琐的配置时所用，不能用作...
基于Kubernetes/K8S构建Jenkins持续集成平台
06-21
工作流程：手动/自动构建 -> Jenkins 调度 K8S API?-＞动态生成 Jenkins Slave pod -＞ Slave pod 拉取 Git 代码／编译／打包镜像 -＞推送到镜像仓库?Harbor -＞?Slave 工作完成，Pod 自动销毁 -＞部署到测试或生产 ...
Kubernetes/K8S CKS安全专家认证实践（2023年新课，基于k8s 1.26版本）
05-29
分享一套cks课程——Kubernetes/K8S CKS安全专家认证实践（2023年新课，基于k8s 1.26版本）
Kubernetes/K8S CKS安全专家认证实践视频课程
03-28
分享一套课程——《Kubernetes/K8S CKS安全专家认证实践》2023年新课，基于k8s 1.26版本 帮助学员掌握CKS考试所必备的安全相关技能，同时提供考题原题分析，帮助学员轻松拿下CKS认证考试
k8s关于pod的metadata、spec.containers、spec.volumes的属性介绍（yaml格式）
树下一少年的博客
11-12 1671
一.metadata常用属性 二.spec.containers子属性介绍 explain pod.spec.containers给出的参考 1.command示例 2.env和envFrom示例 3.ports部分详解 4.resources部分详解 5.startupProbe格式 6.terminationMessagePath和terminationMessagePolicy格式 7.volumeDevices格式 8.volumeMounts格式 三.spec.volumes子属性介绍
kubelet源码解析-创建与删除POD(下)
ThinkMo的专栏
01-10 1708
kubeGenericRuntimeManager kubeGenericRuntimeManager为kubelet提供Runtime接口, 管理pods与containers生命周期, 调用remote runtime api完成操作；Kubelet与CRI交互如下图 SyncPod 接上文Pod处理走到调用kubeGenericRuntimeManager.SyncPod pkg/kubelet/kuberuntime/kuberuntime_manager.go:661 func (m *kube
deletepod = restart pod?
喝醉酒的小白
04-18 1155
如果容器使用的资源超过了预设的阈值，Kubernetes将认为Pod处于异常状态，并尝试重新启动该Pod中的容器。需要注意的是，定期检查Pod状态的时间间隔并不是唯一的检测异常的方式，还有其他的方式，如前面提到的容器存活探针、容器就绪探针和资源利用率等。这些方式可以更快地检测到Pod异常，并进行相应的处理。总之，Kubernetes控制平面会通过定期检查、容器存活探针、容器就绪探针和资源利用率等方式来检测到Pod异常，并尝试自动修复或重新启动异常Pod，以确保应用程序的高可用性和可靠性。
k8s源码分析--kubelet中pod处理流程(续)
程序源234的专栏
09-10 2940
源码为k8s v1.3.0稳定版本 (7) updatepod起的协程在何时退出 再kubelet中有两个过程涉及到两个处理函数   一个是： HandlePodRemoves-》deletePod-》kl.podWorkers.ForgetWorker(pod.UID)-》removeWorker   另外一个是： HandlePodCleanups -》Forget
【kubernetes/k8s概念】k8s 坑问题汇总
热门推荐
zhonglinzhang
12-26 3万+
1.Pod始终处于Pending状态 如果Pod保持在Pending的状态，意味着无法被正常的调度到节点上。由于某种系统资源无法满足Pod运行的需求 系统没有足够的资源：已经用尽了集群中所有的CPU或内存资源。需要清理一些不在需要的Pod，调整它们所需的资源量，或者向集群中增加新的节点。 用户指定了hostPort：通过hostPort用户能够将服务暴露到指定的主机端口上，会限......
kubelet分析-pvc扩容源码分析
良凯尔的博客
06-12 900
pvc扩容分析。pvc存储扩容分析。存储的扩容分为controller端操作与node端操作两大步骤，controller端操作由external-resizer来调用ceph完成，而node端操作由kubelet来完成，下面来分析下kubelet中有关存储扩容的相关代码。...
Kubernetes K8S之存储Volume详解
踏歌行的专栏
10-19 2594
K8S之存储Volume概述与说明，并详解常用Volume示例
kubernetes/k8s运维架构师实战集训营
07-19
kubernetes/k8s运维架构师实战集训营是一种培训计划，旨在培养和提升对于kubernetes和k8s运维架构的理解和实践经验。该集训营通过结合实际案例和实操，深入探讨kubernetes的使用和管理，帮助学员掌握k8s的核心概念、架构原理以及常见的运维策略和技巧。 在实战集训营中，学员将进行一系列的讲座和实践项目，包括搭建kubernetes集群、配置和管理容器、应用监控和日志管理、故障排除和性能调优等内容。通过这些实际操作，学员将能够深入了解k8s的运行机制，并掌握如何在生产环境中部署和管理kubernetes集群。 此外，该集训营还将引导学员探索最佳实践，包括如何设计高可用的架构、如何构建自动化的部署流水线，以及如何进行容器网络和存储的管理等。通过这些实际案例的学习，学员将能够应对复杂的场景，并提供有效的解决方案。 总的来说，kubernetes/k8s运维架构师实战集训营为学员提供了一个全方位的学习和实践平台，帮助他们成为在kubernetes生态系统中具备丰富经验和技能的运维架构师。这将为他们的职业发展打下坚实的基础，并为企业提供具备kubernetes运维能力的人才。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值