kubernetes 的管理之:存储

     容器磁盘上的文件的生命周期是短暂的，这就使得在容器中运行重要应用时会出现一些问题。首先，当容器崩溃时，kubelet 会重启它，但是容器中的文件将丢失——容器以干净的状态（镜像最初的状态）重新启动。其次，在 Pod 中同时运行多个容器时，这些容器之间通常需要共享文件。Kubernetes 中的 Volume 抽象就很好的解决了这些问题。在原docker环境中也有存储卷的概念，但docker环境的存储卷调度在宿主机上的目录，当docker重新创建的时候存储卷还会挂载统一宿主机上，但我们知道Kubernetes是分布式集群，当我们销毁一个pod的时候，可能pod会在其他节点上启动，但其他节点宿主机上并没有这个目录，这样就不会挂载到原来的数据了。因此K8s提供了持久化存储。

     如果想要搞清楚k8s的存储，就要先理解几个概念PV，PVC，StorageClass。下图具体描绘了这几个图的关系。

一、StorageClass

    StorageClass作为对存储资源的抽象定义，对用户设置的PVC申请屏蔽后端的存储细节，一方面减少了用户对于存储资源细节的关注，另一方面减轻了管理员手工管理PV的工作，由系统自动完成PV的创建和绑定，实现了动态的资源供应。
    StorageClass定义了包括主要名称、后端存储的提供者和后端存储的相关参数设置。

可以通过kubectl explain pods.spec.volumes来查看支持的存储卷：

[root@master ~]# kubectl explain pods.spec.volumes
KIND:     Pod
VERSION:  v1

RESOURCE: volumes <[]Object>

DESCRIPTION:
     List of volumes that can be mounted by containers belonging to the pod.
     More info: https://kubernetes.io/docs/concepts/storage/volumes

     Volume represents a named volume in a pod that may be accessed by any
     container in the pod.

FIELDS:
   awsElasticBlockStore <Object>
     AWSElasticBlockStore represents an AWS Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore

   azureDisk    <Object>
     AzureDisk represents an Azure Data Disk mount on the host and bind mount to
     the pod.

   azureFile    <Object>
     AzureFile represents an Azure File Service mount on the host and bind mount
     to the pod.

   cephfs       <Object>
     CephFS represents a Ceph FS mount on the host that shares a pod's lifetime

   cinder       <Object>
     Cinder represents a cinder volume attached and mounted on kubelets host
     machine More info:
     https://releases.k8s.io/HEAD/examples/mysql-cinder-pd/README.md

   configMap    <Object>
     ConfigMap represents a configMap that should populate this volume

   downwardAPI  <Object>
     DownwardAPI represents downward API about the pod that should populate this
     volume

   emptyDir     <Object>
     EmptyDir represents a temporary directory that shares a pod's lifetime.
     More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir

   fc   <Object>
     FC represents a Fibre Channel resource that is attached to a kubelet's host
     machine and then exposed to the pod.

   flexVolume   <Object>
     FlexVolume represents a generic volume resource that is
     provisioned/attached using an exec based plugin.

   flocker      <Object>
     Flocker represents a Flocker volume attached to a kubelet's host machine.
     This depends on the Flocker control service being running

   gcePersistentDisk    <Object>
     GCEPersistentDisk represents a GCE Disk resource that is attached to a
     kubelet's host machine and then exposed to the pod. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk

   gitRepo      <Object>
     GitRepo represents a git repository at a particular revision. DEPRECATED:
     GitRepo is deprecated. To provision a container with a git repo, mount an
     EmptyDir into an InitContainer that clones the repo using git, then mount
     the EmptyDir into the Pod's container.

   glusterfs    <Object>
     Glusterfs represents a Glusterfs mount on the host that shares a pod's
     lifetime. More info:
     https://releases.k8s.io/HEAD/examples/volumes/glusterfs/README.md

   hostPath     <Object>
     HostPath represents a pre-existing file or directory on the host machine
     that is directly exposed to the container. This is generally used for
     system agents or other privileged things that are allowed to see the host
     machine. Most containers will NOT need this. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#hostpath

   iscsi        <Object>
     ISCSI represents an ISCSI Disk resource that is attached to a kubelet's
     host machine and then exposed to the pod. More info:
     https://releases.k8s.io/HEAD/examples/volumes/iscsi/README.md

   name <string> -required-
     Volume's name. Must be a DNS_LABEL and unique within the pod. More info:
     https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

   nfs  <Object>
     NFS represents an NFS mount on the host that shares a pod's lifetime More
     info: https://kubernetes.io/docs/concepts/storage/volumes#nfs

   persistentVolumeClaim        <Object>
     PersistentVolumeClaimVolumeSource represents a reference to a
     PersistentVolumeClaim in the same namespace. More info:
     https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

   photonPersistentDisk <Object>
     PhotonPersistentDisk represents a PhotonController persistent disk attached
     and mounted on kubelets host machine

   portworxVolume       <Object>
     PortworxVolume represents a portworx volume attached and mounted on
     kubelets host machine

   projected    <Object>
     Items for all in one resources secrets, configmaps, and downward API

   quobyte      <Object>
     Quobyte represents a Quobyte mount on the host that shares a pod's lifetime

   rbd  <Object>
     RBD represents a Rados Block Device mount on the host that shares a pod's
     lifetime. More info:
     https://releases.k8s.io/HEAD/examples/volumes/rbd/README.md

   scaleIO      <Object>
     ScaleIO represents a ScaleIO persistent volume attached and mounted on
     Kubernetes nodes.

   secret       <Object>
     Secret represents a secret that should populate this volume. More info:
     https://kubernetes.io/docs/concepts/storage/volumes#secret

   storageos    <Object>
     StorageOS represents a StorageOS volume attached and mounted on Kubernetes
     nodes.

   vsphereVolume        <Object>
     VsphereVolume represents a vSphere volume attached and mounted on kubelets
     host machine

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
  - debug
volumeBindingMode: Immediate

二、PV

    PV作为存储资源，主要包含存储能力、访问模式、存储类型、回收策略、
后端存储类型等相关参数设置。

• 存储能力：
  表述存储设备具备的能力，既存储空间的设置（storage=xx）。
• 存储卷模式：
  包括Filesystem（文件系统）和Block（块设备），默认值为Filesystem。
• 访问模式：
  ReadWriteOnce（RWO）：读写权限，并且只能被单个Node挂载、
  ReadOnlyMany（ROX）：只读权限，允许被多个Node挂载。
  ReadWriteMany（RWX）：读写权限，允许被多个Node挂载。
• 存储类别
  通过storageClassName参数指定。
• 回收策略：
  保留（retain）：保留数据，手工处理。
  回收空间（recycle）：简单清除文件操作。
  删除（delete）：与PV相连的后端存储完成Volume的删除操作。
• 挂载参数：
  mountOptions字段进行设置。
• 几点亲和性：
  用来限制某些Node可以访问Volume，nodeAffinity字段进行设置。公有云提供的存储卷，都由公有云自动完成节点亲和性设置。

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0003
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /tmp
    server: 172.17.0.2

三、PVC

    PVC作为用户对存储资源的需求申请，主要包括存储空间请求、访问模式、PV选择条件和存储类别等信息的设置。

• 资源请求（Resource）：request.storage的设置，即存储空间大小。
• 访问模式（Access Mode）：同PV访问模式设置相同。
• 存储卷模式（Volume Mode）：用户描述希望使用的PV存储卷模式，包括文件系统和块设备。
• PV选择条件（Selector）：即Label Selector的设置，用于进行PV的筛选。
• 存储类别（Class）：通过storageClassName字段设置需要的后端存储类别。此字段也可以不设置或者为空，这样PVC会选择系统未设定Class的PV与之绑定。
• 未启动 DefaultStorageClass：等效于storageClassName值设置为空。
• 启动DefaultStorageClass：设置之后，系统将自动为PVC创建一个PV，并将他们绑定。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 8Gi
  storageClassName: slow
  selector:
    matchLabels:
      release: "stable"
    matchExpressions:
      - {key: environment, operator: In, values: [dev]}

注意：PVC和PV都受限与Namespace，PVC在选择PV时受到Namespace的限制，只有相同Namespace中的PV才能与PVC绑定。

四、供应模式

• 静态模式：集群管理员手工创建许多PV，在定义PV时需要将后端存储的特性进行设置。
• 动态模式：集群管理员无须手工创建PV，而是通过StorageClass的设置对后端存储进行描述，标记为某种类型。此时要求PVC对存储类型进行声明，系统将自动完成PV的创建及与PVC的绑定。PVC可以声明Class为“”，说明该PVC禁止使用动态模式。

五、client-go

1. PV 接口

client-go/kubernetes/type/core/v1/persistentvolume.go

// PersistentVolumeInterface has methods to work with PersistentVolume resources.
type PersistentVolumeInterface interface {
	Create(*v1.PersistentVolume) (*v1.PersistentVolume, error)
	Update(*v1.PersistentVolume) (*v1.PersistentVolume, error)
	UpdateStatus(*v1.PersistentVolume) (*v1.PersistentVolume, error)
	Delete(name string, options *metav1.DeleteOptions) error
	DeleteCollection(options *metav1.DeleteOptions, listOptions metav1.ListOptions) error
	Get(name string, options metav1.GetOptions) (*v1.PersistentVolume, error)
	List(opts metav1.ListOptions) (*v1.PersistentVolumeList, error)
	Watch(opts metav1.ListOptions) (watch.Interface, error)
	Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.PersistentVolume, err error)
	PersistentVolumeExpansion
}

2. PVC 接口
   client-go/kubernetes/type/core/v1/persistentvolumeclaim.go

// PersistentVolumeClaimInterface has methods to work with PersistentVolumeClaim resources.
type PersistentVolumeClaimInterface interface {
	Create(*v1.PersistentVolumeClaim) (*v1.PersistentVolumeClaim, error)
	Update(*v1.PersistentVolumeClaim) (*v1.PersistentVolumeClaim, error)
	UpdateStatus(*v1.PersistentVolumeClaim) (*v1.PersistentVolumeClaim, error)
	Delete(name string, options *metav1.DeleteOptions) error
	DeleteCollection(options *metav1.DeleteOptions, listOptions metav1.ListOptions) error
	Get(name string, options metav1.GetOptions) (*v1.PersistentVolumeClaim, error)
	List(opts metav1.ListOptions) (*v1.PersistentVolumeClaimList, error)
	Watch(opts metav1.ListOptions) (watch.Interface, error)
	Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.PersistentVolumeClaim, err error)
	PersistentVolumeClaimExpansion
}

参考：
Kubernets官方文档.
Kubernetes权威指南